Scenario · 13B parameters · Full fine-tune
13B full fine-tune on 1 billion tokens: cost estimate
13 billion parameters, 1 billion tokens, full fine-tune. The lowest estimate across 11 GPU entries is $64 on Vultr with MI355X, 25 h of GPU time at 35% MFU. An estimate, not a quote.
Step by step
The arithmetic
- 1. Training FLOPs. 6 × 13 × 10^9 parameters × 1 billion tokens = 7.80 × 10^19 FLOPs. About 2 x parameters x tokens for the forward pass plus about 4 x for the backward pass, because the backward pass computes gradients for activations and for every weight.
- 2. Useful throughput per GPU. Dense BF16 TFLOPS × 10^12 × 35% MFU. The table gives it for each GPU.
- 3. GPU-hours. FLOPs ÷ throughput ÷ 3,600. For L40S (362.05 TFLOPS): 7.80 × 10^19 ÷ 1.27 × 10^14 ÷ 3,600 = 171 GPU-hours.
- 4. Wall-clock. GPU-hours ÷ 8 GPUs, assuming the work spreads evenly.
- 5. Cost. GPU-hours × each provider's lowest listed on-demand price per GPU-hour for that GPU.
Result
Estimated cost by GPU
| GPU | Dense BF16 TFLOPS | GPU-hours | 8 GPUs | Providers | Lowest | Median | Highest |
|---|---|---|---|---|---|---|---|
| MI355X | 2500 | 25 | 3.1 h | 2 | $64Vultr | $139 | $213Oracle Cloud |
| MI300X | 1307.4 | 47 | 5.9 h | 4 | $88Vultr | $143 | $284Oracle Cloud |
| MI325X | 1307.4 | 47 | 5.9 h | 2 | $95Vultr | $137 | $180DigitalOcean (Paperspace) |
| H100 SXMpreliminary | 1000 | 62 | 7.7 h | 6 | $183Thunder Compute | $196 | $315Verda |
| GH200 | 990 | 63 | 7.8 h | 2 | $124Vultr | $265 | $406CoreWeave |
| H100 PCIepreliminary | 800 | 77 | 9.7 h | 2 | $193Hyperstack | $209 | $224RunPod |
| L40S | 362.05 | 171 | 21 h | 14 | $94Runcrate | $262 | $598Oracle Cloud |
| A100 80GB | 312 | 198 | 25 h | 12 | $216Thunder Compute | $376 | $681Amazon Web Services |
| L40 | 181.05 | 342 | 43 h | 6 | $270Thunder Compute | $318 | $431Runcrate |
| A40 | 149.7 | 414 | 2.2 days | 3 | $244RunPod | $708 | $1,100Runcrate |
| A10 | 125 | 495 | 2.6 days | 4 | $495OVHcloud | $953 | $1,585Microsoft Azure |
Prices are each provider's lowest listed on-demand rate per GPU-hour, read 12 November 2026. Only GPUs whose maker prints a dense BF16 figure appear; the coverage page says which are out and why.
Sensitivity
What if MFU were different? (L40S)
| MFU assumed | GPU-hours | 8 GPUs | Lowest cost |
|---|---|---|---|
| 20% | 299 | 37 h | $165 |
| 35% (default) | 171 | 21 h | $94 |
| 50% | 120 | 15 h | $66 |
The 20% and 50% rows are what-if inputs chosen to show sensitivity, not measurements. Lowest cost uses Runcrate at $0.55 per GPU-hour.
Try it
Change the assumptions
Estimate, step by step
- 1. Training FLOPs = 6 × 13 billion parameters × 1000 million tokens = 7.80 × 10^19
- 2. Useful FLOP/s per GPU = 362.05 TFLOPS × 35% MFU = 1.27 × 10^14
- 3. GPU-hours = step 1 ÷ step 2 ÷ 3,600 = 171
- 4. Wall-clock on 8 GPUs = step 3 ÷ 8 = 21 h, assuming the work spreads evenly
- 5. Cost = step 3 × each provider's lowest listed price per GPU-hour
| Provider | Billing | $/GPU-hr | Estimated cost |
|---|---|---|---|
| Runcrateours, live | per second | $0.55 | $94 |
| CloudRift | per second | $0.69 | $118 |
| Massed Compute | billing not stated | $0.97 | $166 |
| RunPod | per second | $1.09 | $186 |
| Koyeb | per second | $1.20 | $205 |
| Civo | per hour | $1.29 | $221 |
| Crusoe | per hour | $1.50 | $256 |
| DigitalOcean (Paperspace) | per second | $1.57 | $268 |
| Vultr | per hour | $1.67 | $286 |
| OVHcloud | per hour | $1.80 | $308 |
| Amazon Web Services | per second | $1.86 | $318 |
| Verda | billing not stated | $2.00 | $342 |
| CoreWeave | billing not stated | $2.25 | $385 |
| Oracle Cloud | billing not stated | $3.50 | $598 |
Range $94 to $598, median $262 across 14 providers. An estimate, not a quote: real runs differ in utilisation, memory fit, communication and restarts, and providers that bill per hour round up.
Questions
13B full fine-tune on 1 billion tokens: common questions
How many FLOPs does this take?
About 7.80 × 10^19: 6 × 13 billion parameters × 1 billion tokens. About 2 x parameters x tokens for the forward pass plus about 4 x for the backward pass, because the backward pass computes gradients for activations and for every weight.
What does it cost at the low end?
$64, on Vultr with MI355X at $2.59 per GPU-hour, for 25 GPU-hours.
Is this a quote?
No. It is an estimate from a stated formula and published list prices. Real runs differ in utilisation, memory fit and restarts, and the MFU is an assumption.
Why are some GPUs missing?
A GPU needs a dense BF16 figure printed by its maker. Those that print only a sparse figure are excluded rather than halved.
Related
- 13B LoRA fine-tune on 100 million tokensSame size, other method
- 13B LoRA fine-tune on 1 billion tokensSame size, other method
- 13B full fine-tune on 100 million tokensSame size, other token count
- 7B full fine-tune on 1 billion tokensSame method and tokens
- 34B full fine-tune on 1 billion tokensSame method and tokens
Disclosure
Fine-Tune Cost Lab is published by the Runcrate team, who also run Runcrate, an AI cloud. Runcrate is listed in the table, marked "ours, live". It is sorted by price like every other row, with no pinned position. Other providers' prices are read from their own public pricing pages. We have no paid placement.