Fine-tuning compute: FLOPs, GPU-hours and what a training run really costs

The finding#

A full fine-tune of a 7B model on 60 million tokens — say 20,000 examples of 1,000 tokens for three epochs — takes about 2.6 H100-hours at 40% model FLOPs utilisation: roughly $5.3–$11 at $2–$4 per GPU-hour. Even a 72B model costs $50–$100 for the same data.

That should change how you budget. If compute for one run is a rounding error, project cost is driven by how many runs you need and by work that isn’t training at all: data, evaluation and engineering time. The levers that matter are the ones that reduce wasted experiments.

FLOPs per training token#

For a dense transformer with $N$ parameters, a forward pass costs about $2N$ floating-point operations per token (one multiply and one add per weight). The backward pass computes two things — gradients with respect to activations and gradients with respect to weights — each costing about the same as the forward pass. That gives the familiar estimate from Kaplan et al.:

LoRA changes the backward pass. The base weights are frozen, so their weight gradients are never computed; activation gradients still flow through every layer. Adapter weight gradients are negligible for small ranks:

So LoRA saves about a third of the compute, not the 99% its parameter count suggests. Activation checkpointing adds one recomputed forward pass, $+2N$, in both cases. Attention adds a term that grows with sequence length $s$: roughly $6Lsh$ per token for training, small next to $6N$ at typical fine-tuning lengths.

where $D$ is training tokens, $P_{\text{peak}}$ is the accelerator’s dense BF16 throughput (989 TFLOPS for an H100 SXM) and MFU is the fraction you actually achieve. Well-tuned training reaches 35–55%; small batches, short sequences and LoRA kernels often land at 15–30%.

Cost per run, 0.5B to 72B#

60 million training tokens, sequence length 2,048, activation checkpointing on, H100 SXM at 40% MFU, priced at $2–$4 per GPU-hour:

Memory, not FLOPs, usually decides how many GPUs a run occupies: full fine-tuning a 7B model needs ~114 GB of optimizer and weight state (see our memory model), so it runs on at least two GPUs even though one could do the arithmetic in under three hours.

Sensitivity to utilisation#

MFU is the most uncertain input. For the 7B full fine-tune above:

As a sanity check we computed the throughput our own laptop-scale experiment actually achieved. The LoRA rank experiment trained a 0.5B model on an Apple M4 Max GPU shared with other workloads. Full fine-tuning sustained about 0.7 TFLOP/s, LoRA r=16 about 0.6 TFLOP/s — a small fraction of a data-centre GPU, which is why the same job that took minutes there takes seconds on an H100.

Where the money actually goes#

If a run costs tens of dollars, why do fine-tuning projects spend thousands on compute? Four multipliers:

1. The number of runs#

A typical project runs a base-model bake-off (3–5 models), learning-rate and rank sweeps, two or three data iterations and ablations. Thirty to eighty training runs is normal. Multiply the table by 50.

2. Reinforcement learning is mostly generation#

GRPO and PPO sample several completions per prompt every step. Generation costs $2N$ FLOPs per token but is memory-bandwidth bound, so it typically runs at single-digit MFU. An RL step can cost ten times an SFT step on the same number of tokens. See GRPO explained.

3. Evaluation is generation too#

Scoring every checkpoint on a 2,000-example held-out set with 300-token outputs is 600k generated tokens per checkpoint, plus judge-model calls. Over a project, evaluation compute rivals training compute.

4. Idle and minimum billing#

Reserved multi-GPU nodes bill while you analyse results. The cheapest compute strategy is usually a small node for exploration and short bursts on large nodes for final runs.

From MFU to tokens per second#

Inverting the cost equation gives the training throughput to expect, which is the number to check against a real run in its first minutes:

For a 7B model with full fine-tuning and checkpointing ($\approx 8N$ FLOPs per token) at 40% MFU on an H100, that is about 6,490 tokens per second per GPU. If your logged throughput is half that, you are at 20% MFU and something — data loading, padding, small micro-batches, communication — deserves attention before you scale up.

# Measure achieved MFU from your training logs
N          = 7.62e9          # parameters
flops_tok  = 8 * N           # full FT with activation checkpointing (6N + 2N)
tokens_s   = 3.1e3           # logged training tokens/s per GPU
peak       = 989e12          # H100 SXM dense BF16
print(f"MFU = {tokens_s * flops_tok / peak:.1%}")   # -> 19.1%

Worked example: compute budget for a project#

A realistic 8B project, assuming 25% MFU for training jobs and 5% for generation-heavy work (evaluation and RL sampling are memory-bandwidth bound):

At $2–$4 per GPU-hour that is roughly $54–$108 — consistent with the $500–$5,000 range for SFT+DPO projects in our cost guide, with RL pushing toward the top. Note how evaluation and RL, both generation-heavy, account for 21% of the total despite being a minority of the “training”.

Assumptions and limitations#

The $6N$ rule ignores embedding lookups and some elementwise operations and slightly overstates FLOPs for models with large tied embeddings. MFU varies with kernel quality, sequence packing and interconnect. GPU prices are assumption ranges, not quotes; spot and reserved pricing can sit well outside them. Mixture-of-experts models need $N_{\text{active}}$ instead of $N$ for FLOPs but full $N$ for memory.

References#

1. Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
2. NVIDIA. H100 Tensor Core GPU datasheet (BF16 dense tensor throughput, SXM form factor: 989 TFLOPS).
3. Korthikanti, V. et al. (2023). Reducing Activation Recomputation in Large Transformer Models. MLSys 2023. arXiv:2205.05198.
4. Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.