Scenario · 34B parameters · Full fine-tune
34B full fine-tune on 1 billion tokens: cost estimate
34 billion parameters, 1 billion tokens, full fine-tune. The lowest estimate across 11 GPU entries is $168 on Vultr with MI355X, 2.7 days of GPU time at 35% MFU. An estimate, not a quote.
Step by step
The arithmetic
- 1. Training FLOPs. 6 × 34 × 10^9 parameters × 1 billion tokens = 2.04 × 10^20 FLOPs. About 2 x parameters x tokens for the forward pass plus about 4 x for the backward pass, because the backward pass computes gradients for activations and for every weight.
- 2. Useful throughput per GPU. Dense BF16 TFLOPS × 10^12 × 35% MFU. The table gives it for each GPU.
- 3. GPU-hours. FLOPs ÷ throughput ÷ 3,600. For L40S (362.05 TFLOPS): 2.04 × 10^20 ÷ 1.27 × 10^14 ÷ 3,600 = 447 GPU-hours.
- 4. Wall-clock. GPU-hours ÷ 8 GPUs, assuming the work spreads evenly.
- 5. Cost. GPU-hours × each provider's lowest listed on-demand price per GPU-hour for that GPU.
Result
Estimated cost by GPU
| GPU | Dense BF16 TFLOPS | GPU-hours | 8 GPUs | Providers | Lowest | Median | Highest |
|---|---|---|---|---|---|---|---|
| MI355X | 2500 | 65 | 8.1 h | 2 | $168Vultr | $362 | $557Oracle Cloud |
| MI300X | 1307.4 | 124 | 15 h | 4 | $229Vultr | $374 | $743Oracle Cloud |
| MI325X | 1307.4 | 124 | 15 h | 2 | $248Vultr | $359 | $471DigitalOcean (Paperspace) |
| H100 SXMpreliminary | 1000 | 162 | 20 h | 6 | $478Thunder Compute | $513 | $824Verda |
| GH200 | 990 | 164 | 20 h | 2 | $325Vultr | $694 | $1,063CoreWeave |
| H100 PCIepreliminary | 800 | 202 | 25 h | 2 | $506Hyperstack | $545 | $585RunPod |
| L40S | 362.05 | 447 | 2.3 days | 14 | $246Runcrate | $686 | $1,565Oracle Cloud |
| A100 80GB | 312 | 519 | 2.7 days | 12 | $566Thunder Compute | $983 | $1,780Amazon Web Services |
| L40 | 181.05 | 894 | 4.7 days | 6 | $706Thunder Compute | $832 | $1,127Runcrate |
| A40 | 149.7 | 1,082 | 5.6 days | 3 | $638RunPod | $1,852 | $2,877Runcrate |
| A10 | 125 | 1,295 | 6.7 days | 4 | $1,295OVHcloud | $2,493 | $4,145Microsoft Azure |
Prices are each provider's lowest listed on-demand rate per GPU-hour, read 12 November 2026. Only GPUs whose maker prints a dense BF16 figure appear; the coverage page says which are out and why.
Sensitivity
What if MFU were different? (L40S)
| MFU assumed | GPU-hours | 8 GPUs | Lowest cost |
|---|---|---|---|
| 20% | 783 | 4.1 days | $430 |
| 35% (default) | 447 | 2.3 days | $246 |
| 50% | 313 | 39 h | $172 |
The 20% and 50% rows are what-if inputs chosen to show sensitivity, not measurements. Lowest cost uses Runcrate at $0.55 per GPU-hour.
Try it
Change the assumptions
Estimate, step by step
- 1. Training FLOPs = 6 × 34 billion parameters × 1000 million tokens = 2.04 × 10^20
- 2. Useful FLOP/s per GPU = 362.05 TFLOPS × 35% MFU = 1.27 × 10^14
- 3. GPU-hours = step 1 ÷ step 2 ÷ 3,600 = 447.2
- 4. Wall-clock on 8 GPUs = step 3 ÷ 8 = 2.3 days, assuming the work spreads evenly
- 5. Cost = step 3 × each provider's lowest listed price per GPU-hour
| Provider | Billing | $/GPU-hr | Estimated cost |
|---|---|---|---|
| Runcrateours, live | per second | $0.55 | $246 |
| CloudRift | per second | $0.69 | $309 |
| Massed Compute | billing not stated | $0.97 | $434 |
| RunPod | per second | $1.09 | $487 |
| Koyeb | per second | $1.20 | $537 |
| Civo | per hour | $1.29 | $577 |
| Crusoe | per hour | $1.50 | $671 |
| DigitalOcean (Paperspace) | per second | $1.57 | $702 |
| Vultr | per hour | $1.67 | $747 |
| OVHcloud | per hour | $1.80 | $805 |
| Amazon Web Services | per second | $1.86 | $832 |
| Verda | billing not stated | $2.00 | $893 |
| CoreWeave | billing not stated | $2.25 | $1,006 |
| Oracle Cloud | billing not stated | $3.50 | $1,565 |
Range $246 to $1,565, median $686 across 14 providers. An estimate, not a quote: real runs differ in utilisation, memory fit, communication and restarts, and providers that bill per hour round up.
Questions
34B full fine-tune on 1 billion tokens: common questions
How many FLOPs does this take?
About 2.04 × 10^20: 6 × 34 billion parameters × 1 billion tokens. About 2 x parameters x tokens for the forward pass plus about 4 x for the backward pass, because the backward pass computes gradients for activations and for every weight.
What does it cost at the low end?
$168, on Vultr with MI355X at $2.59 per GPU-hour, for 65 GPU-hours.
Is this a quote?
No. It is an estimate from a stated formula and published list prices. Real runs differ in utilisation, memory fit and restarts, and the MFU is an assumption.
Why are some GPUs missing?
A GPU needs a dense BF16 figure printed by its maker. Those that print only a sparse figure are excluded rather than halved.
Related
- 34B LoRA fine-tune on 100 million tokensSame size, other method
- 34B LoRA fine-tune on 1 billion tokensSame size, other method
- 34B full fine-tune on 100 million tokensSame size, other token count
- 7B full fine-tune on 1 billion tokensSame method and tokens
- 13B full fine-tune on 1 billion tokensSame method and tokens
Disclosure
Fine-Tune Cost Lab is published by the Runcrate team, who also run Runcrate, an AI cloud. Runcrate is listed in the table, marked "ours, live". It is sorted by price like every other row, with no pinned position. Other providers' prices are read from their own public pricing pages. We have no paid placement.