LoRA against full
What the method changes in the arithmetic
Full fine-tuning uses 6 FLOPs per parameter per token and LoRA 4, so the same job takes 1.5 times as many FLOPs when fully trained. Here is each size side by side, on the L40S.
Side by side
One billion tokens, four sizes
| Size | LoRA FLOPs | Full FLOPs | LoRA GPU-hours | Full GPU-hours | LoRA lowest | Full lowest |
|---|---|---|---|---|---|---|
| 7B | 2.80 × 10^19 | 4.20 × 10^19 | 61 | 92 | $34 | $51 |
| 13B | 5.20 × 10^19 | 7.80 × 10^19 | 114 | 171 | $63 | $94 |
| 34B | 1.36 × 10^20 | 2.04 × 10^20 | 298 | 447 | $164 | $246 |
| 70B | 2.80 × 10^20 | 4.20 × 10^20 | 614 | 921 | $338 | $506 |
On the L40S (362.05 TFLOPS dense BF16, 14 providers) at 35% MFU. Costs use the lowest provider. The ratio between the columns is the ratio of the factors, because everything else is the same.
Why the factors differ
Forward, backward and the frozen weights
About 2 x parameters x tokens for the forward pass plus about 4 x for the backward pass, because the backward pass computes gradients for activations and for every weight.
The forward pass costs about 2 x parameters x tokens and the backward pass about 2 x, because gradients for the frozen base weights are not computed. The small adapter matrices and QLoRA dequantisation are not counted.
These factors say nothing about memory. LoRA also changes how much GPU memory a job needs, which decides how many GPUs you need and sometimes which GPU you can use at all. This site does not model memory, so a GPU that appears in a table may not hold your job.
Questions
LoRA against full: common questions
Is LoRA always cheaper?
In FLOPs, by the ratio of the factors: 4 against 6. Whether the job is cheaper in practice depends on memory, utilisation and the quality you need, none of which this site models.
Does QLoRA cost the same as LoRA here?
Yes. We use one factor for both and do not count the dequantisation work QLoRA adds, so its real cost is somewhat higher than shown.
Can I change the factors?
Yes. Both are editable in the calculator.
Disclosure
Fine-Tune Cost Lab is published by the Runcrate team, who also run Runcrate, an AI cloud. Runcrate is listed in the table, marked "ours, live". It is sorted by price like every other row, with no pinned position. Other providers' prices are read from their own public pricing pages. We have no paid placement.