Scenario · 13B parameters · Full fine-tune
13B full fine-tune on 100 million tokens: cost estimate
13 billion parameters, 100 million tokens, full fine-tune. The lowest estimate across 11 GPU entries is $6.41 on Vultr with MI355X, 2.5 h of GPU time at 35% MFU. An estimate, not a quote.
Step by step
The arithmetic
- 1. Training FLOPs. 6 × 13 × 10^9 parameters × 100 million tokens = 7.80 × 10^18 FLOPs. About 2 x parameters x tokens for the forward pass plus about 4 x for the backward pass, because the backward pass computes gradients for activations and for every weight.
- 2. Useful throughput per GPU. Dense BF16 TFLOPS × 10^12 × 35% MFU. The table gives it for each GPU.
- 3. GPU-hours. FLOPs ÷ throughput ÷ 3,600. For L40S (362.05 TFLOPS): 7.80 × 10^18 ÷ 1.27 × 10^14 ÷ 3,600 = 17 GPU-hours.
- 4. Wall-clock. GPU-hours ÷ 8 GPUs, assuming the work spreads evenly.
- 5. Cost. GPU-hours × each provider's lowest listed on-demand price per GPU-hour for that GPU.
Result
Estimated cost by GPU
| GPU | Dense BF16 TFLOPS | GPU-hours | 8 GPUs | Providers | Lowest | Median | Highest |
|---|---|---|---|---|---|---|---|
| MI355X | 2500 | 2.48 | 19 min | 2 | $6.41Vultr | $14 | $21Oracle Cloud |
| MI300X | 1307.4 | 4.73 | 36 min | 4 | $8.76Vultr | $14 | $28Oracle Cloud |
| MI325X | 1307.4 | 4.73 | 36 min | 2 | $9.47Vultr | $14 | $18DigitalOcean (Paperspace) |
| H100 SXMpreliminary | 1000 | 6.19 | 46 min | 6 | $18Thunder Compute | $20 | $31Verda |
| GH200 | 990 | 6.25 | 47 min | 2 | $12Vultr | $27 | $41CoreWeave |
| H100 PCIepreliminary | 800 | 7.74 | 58 min | 2 | $19Hyperstack | $21 | $22RunPod |
| L40S | 362.05 | 17 | 2.1 h | 14 | $9.40Runcrate | $26 | $60Oracle Cloud |
| A100 80GB | 312 | 20 | 2.5 h | 12 | $22Thunder Compute | $38 | $68Amazon Web Services |
| L40 | 181.05 | 34 | 4.3 h | 6 | $27Thunder Compute | $32 | $43Runcrate |
| A40 | 149.7 | 41 | 5.2 h | 3 | $24RunPod | $71 | $110Runcrate |
| A10 | 125 | 50 | 6.2 h | 4 | $50OVHcloud | $95 | $158Microsoft Azure |
Prices are each provider's lowest listed on-demand rate per GPU-hour, read 12 November 2026. Only GPUs whose maker prints a dense BF16 figure appear; the coverage page says which are out and why.
Sensitivity
What if MFU were different? (L40S)
| MFU assumed | GPU-hours | 8 GPUs | Lowest cost |
|---|---|---|---|
| 20% | 30 | 3.7 h | $16 |
| 35% (default) | 17 | 2.1 h | $9.40 |
| 50% | 12 | 1.5 h | $6.58 |
The 20% and 50% rows are what-if inputs chosen to show sensitivity, not measurements. Lowest cost uses Runcrate at $0.55 per GPU-hour.
Try it
Change the assumptions
Estimate, step by step
- 1. Training FLOPs = 6 × 13 billion parameters × 100 million tokens = 7.80 × 10^18
- 2. Useful FLOP/s per GPU = 362.05 TFLOPS × 35% MFU = 1.27 × 10^14
- 3. GPU-hours = step 1 ÷ step 2 ÷ 3,600 = 17.1
- 4. Wall-clock on 8 GPUs = step 3 ÷ 8 = 2.1 h, assuming the work spreads evenly
- 5. Cost = step 3 × each provider's lowest listed price per GPU-hour
| Provider | Billing | $/GPU-hr | Estimated cost |
|---|---|---|---|
| Runcrateours, live | per second | $0.55 | $9.40 |
| CloudRift | per second | $0.69 | $12 |
| Massed Compute | billing not stated | $0.97 | $17 |
| RunPod | per second | $1.09 | $19 |
| Koyeb | per second | $1.20 | $21 |
| Civo | per hour | $1.29 | $22 |
| Crusoe | per hour | $1.50 | $26 |
| DigitalOcean (Paperspace) | per second | $1.57 | $27 |
| Vultr | per hour | $1.67 | $29 |
| OVHcloud | per hour | $1.80 | $31 |
| Amazon Web Services | per second | $1.86 | $32 |
| Verda | billing not stated | $2.00 | $34 |
| CoreWeave | billing not stated | $2.25 | $38 |
| Oracle Cloud | billing not stated | $3.50 | $60 |
Range $9.40 to $60, median $26 across 14 providers. An estimate, not a quote: real runs differ in utilisation, memory fit, communication and restarts, and providers that bill per hour round up.
Questions
13B full fine-tune on 100 million tokens: common questions
How many FLOPs does this take?
About 7.80 × 10^18: 6 × 13 billion parameters × 100 million tokens. About 2 x parameters x tokens for the forward pass plus about 4 x for the backward pass, because the backward pass computes gradients for activations and for every weight.
What does it cost at the low end?
$6.41, on Vultr with MI355X at $2.59 per GPU-hour, for 2.5 GPU-hours.
Is this a quote?
No. It is an estimate from a stated formula and published list prices. Real runs differ in utilisation, memory fit and restarts, and the MFU is an assumption.
Why are some GPUs missing?
A GPU needs a dense BF16 figure printed by its maker. Those that print only a sparse figure are excluded rather than halved.
Related
- 13B LoRA fine-tune on 100 million tokensSame size, other method
- 13B LoRA fine-tune on 1 billion tokensSame size, other method
- 13B full fine-tune on 1 billion tokensSame size, other token count
- 7B full fine-tune on 100 million tokensSame method and tokens
- 34B full fine-tune on 100 million tokensSame method and tokens
Disclosure
Fine-Tune Cost Lab is published by the Runcrate team, who also run Runcrate, an AI cloud. Runcrate is listed in the table, marked "ours, live". It is sorted by price like every other row, with no pinned position. Other providers' prices are read from their own public pricing pages. We have no paid placement.