Scenario · 13B parameters · LoRA or QLoRA
13B LoRA fine-tune on 1 billion tokens: cost estimate
13 billion parameters, 1 billion tokens, lora or qlora. The lowest estimate across 11 GPU entries is $43 on Vultr with MI355X, 17 h of GPU time at 35% MFU. An estimate, not a quote.
Step by step
The arithmetic
- 1. Training FLOPs. 4 × 13 × 10^9 parameters × 1 billion tokens = 5.20 × 10^19 FLOPs. The forward pass costs about 2 x parameters x tokens and the backward pass about 2 x, because gradients for the frozen base weights are not computed. The small adapter matrices and QLoRA dequantisation are not counted.
- 2. Useful throughput per GPU. Dense BF16 TFLOPS × 10^12 × 35% MFU. The table gives it for each GPU.
- 3. GPU-hours. FLOPs ÷ throughput ÷ 3,600. For L40S (362.05 TFLOPS): 5.20 × 10^19 ÷ 1.27 × 10^14 ÷ 3,600 = 114 GPU-hours.
- 4. Wall-clock. GPU-hours ÷ 8 GPUs, assuming the work spreads evenly.
- 5. Cost. GPU-hours × each provider's lowest listed on-demand price per GPU-hour for that GPU.
Result
Estimated cost by GPU
| GPU | Dense BF16 TFLOPS | GPU-hours | 8 GPUs | Providers | Lowest | Median | Highest |
|---|---|---|---|---|---|---|---|
| MI355X | 2500 | 17 | 2.1 h | 2 | $43Vultr | $92 | $142Oracle Cloud |
| MI300X | 1307.4 | 32 | 3.9 h | 4 | $58Vultr | $95 | $189Oracle Cloud |
| MI325X | 1307.4 | 32 | 3.9 h | 2 | $63Vultr | $92 | $120DigitalOcean (Paperspace) |
| H100 SXMpreliminary | 1000 | 41 | 5.2 h | 6 | $122Thunder Compute | $131 | $210Verda |
| GH200 | 990 | 42 | 5.2 h | 2 | $83Vultr | $177 | $271CoreWeave |
| H100 PCIepreliminary | 800 | 52 | 6.4 h | 2 | $129Hyperstack | $139 | $149RunPod |
| L40S | 362.05 | 114 | 14 h | 14 | $63Runcrate | $175 | $399Oracle Cloud |
| A100 80GB | 312 | 132 | 17 h | 12 | $144Thunder Compute | $251 | $454Amazon Web Services |
| L40 | 181.05 | 228 | 28 h | 6 | $180Thunder Compute | $212 | $287Runcrate |
| A40 | 149.7 | 276 | 34 h | 3 | $163RunPod | $472 | $733Runcrate |
| A10 | 125 | 330 | 41 h | 4 | $330OVHcloud | $636 | $1,057Microsoft Azure |
Prices are each provider's lowest listed on-demand rate per GPU-hour, read 12 November 2026. Only GPUs whose maker prints a dense BF16 figure appear; the coverage page says which are out and why.
Sensitivity
What if MFU were different? (L40S)
| MFU assumed | GPU-hours | 8 GPUs | Lowest cost |
|---|---|---|---|
| 20% | 199 | 25 h | $110 |
| 35% (default) | 114 | 14 h | $63 |
| 50% | 80 | 10.0 h | $44 |
The 20% and 50% rows are what-if inputs chosen to show sensitivity, not measurements. Lowest cost uses Runcrate at $0.55 per GPU-hour.
Try it
Change the assumptions
Estimate, step by step
- 1. Training FLOPs = 4 × 13 billion parameters × 1000 million tokens = 5.20 × 10^19
- 2. Useful FLOP/s per GPU = 362.05 TFLOPS × 35% MFU = 1.27 × 10^14
- 3. GPU-hours = step 1 ÷ step 2 ÷ 3,600 = 114
- 4. Wall-clock on 8 GPUs = step 3 ÷ 8 = 14 h, assuming the work spreads evenly
- 5. Cost = step 3 × each provider's lowest listed price per GPU-hour
| Provider | Billing | $/GPU-hr | Estimated cost |
|---|---|---|---|
| Runcrateours, live | per second | $0.55 | $63 |
| CloudRift | per second | $0.69 | $79 |
| Massed Compute | billing not stated | $0.97 | $111 |
| RunPod | per second | $1.09 | $124 |
| Koyeb | per second | $1.20 | $137 |
| Civo | per hour | $1.29 | $147 |
| Crusoe | per hour | $1.50 | $171 |
| DigitalOcean (Paperspace) | per second | $1.57 | $179 |
| Vultr | per hour | $1.67 | $190 |
| OVHcloud | per hour | $1.80 | $205 |
| Amazon Web Services | per second | $1.86 | $212 |
| Verda | billing not stated | $2.00 | $228 |
| CoreWeave | billing not stated | $2.25 | $256 |
| Oracle Cloud | billing not stated | $3.50 | $399 |
Range $63 to $399, median $175 across 14 providers. An estimate, not a quote: real runs differ in utilisation, memory fit, communication and restarts, and providers that bill per hour round up.
Questions
13B LoRA fine-tune on 1 billion tokens: common questions
How many FLOPs does this take?
About 5.20 × 10^19: 4 × 13 billion parameters × 1 billion tokens. The forward pass costs about 2 x parameters x tokens and the backward pass about 2 x, because gradients for the frozen base weights are not computed. The small adapter matrices and QLoRA dequantisation are not counted.
What does it cost at the low end?
$43, on Vultr with MI355X at $2.59 per GPU-hour, for 17 GPU-hours.
Is this a quote?
No. It is an estimate from a stated formula and published list prices. Real runs differ in utilisation, memory fit and restarts, and the MFU is an assumption.
Why are some GPUs missing?
A GPU needs a dense BF16 figure printed by its maker. Those that print only a sparse figure are excluded rather than halved.
Related
- 13B LoRA fine-tune on 100 million tokensSame size, other token count
- 13B full fine-tune on 100 million tokensSame size, other method
- 13B full fine-tune on 1 billion tokensSame size, other method
- 7B LoRA fine-tune on 1 billion tokensSame method and tokens
- 34B LoRA fine-tune on 1 billion tokensSame method and tokens
Disclosure
Fine-Tune Cost Lab is published by the Runcrate team, who also run Runcrate, an AI cloud. Runcrate is listed in the table, marked "ours, live". It is sorted by price like every other row, with no pinned position. Other providers' prices are read from their own public pricing pages. We have no paid placement.