The formula, and everything it assumes

The formula

Four lines

Assumptions

What the estimate takes as given

1. Full fine-tune factor

   6. About 2 x parameters x tokens for the forward pass plus about 4 x for the backward pass, because the backward pass computes gradients for activations and for every weight.

2. LoRA or QLoRA factor

   4. The forward pass costs about 2 x parameters x tokens and the backward pass about 2 x, because gradients for the frozen base weights are not computed. The small adapter matrices and QLoRA dequantisation are not counted.

3. MFU

   35% by default. The share of the GPU’s peak dense BF16 throughput that training achieves. Real values depend on the software, sequence length, batch size and hardware. This is an assumption and not a measurement of any GPU or provider.

4. Peak throughput

   The maker’s printed dense BF16 tensor figure for the GPU. Dense means without structured sparsity. Where a maker prints only a sparse figure the GPU is left out.

5. Scaling

   Work spreads evenly, so wall-clock time is GPU-hours divided by the GPU count. The scenario pages use 8 GPUs. Communication losses are not modelled, and cost does not depend on the count.

6. Price

   Each provider’s lowest published on-demand price per GPU-hour for the GPU, provider-set prices only, read from its pricing page. Billing granularity is not applied, so providers that bill per hour could cost a little more.

Left out

What is not modelled

• Whether the model, optimiser state and activations fit in GPU memory. A job that does not fit needs more GPUs or a different method.
• Activation checkpointing, which recomputes work and raises FLOPs. Raise the factor to model it.
• Mixed or lower precision such as FP8. Only dense BF16 is used.
• Data preparation, evaluation runs, failed or restarted jobs, storage, data transfer and tax.
• Any measured benchmark. We did not run or read one, so none is shown.

Questions

Assumptions: common questions