How much VRAM do I need to fine-tune a 7B model?

About 117 GB for full fine-tuning with AdamW (two or more 80 GB GPUs with sharding), about 18 GB for LoRA and about 7.3 GB for QLoRA, at 4,096-token sequences with activation checkpointing.

Can I fine-tune a 70B model on one GPU?

With QLoRA, yes: our model gives about 45 GB for a 72B model at 4,096 tokens, which fits on a single 80 GB GPU. Full fine-tuning needs over a terabyte of optimizer and weight state.

Why does full fine-tuning need 16 bytes per parameter?

Mixed-precision AdamW stores bf16 weights (2 bytes), bf16 gradients (2), an fp32 master copy of the weights (4) and two fp32 Adam moments (8).

GPU Memory for Fine-Tuning LLMs: Full vs LoRA vs QLoRA (7B to 72B)

The answer, first#

For a sequence length of 4,096 tokens, micro-batch 1 and full activation checkpointing, our model gives these training-memory requirements (weights, gradients, optimizer state and activations; no framework overhead):

Model	Params	Full fine-tune	LoRA r=16	QLoRA r=16
Qwen2.5-0.5B	0.49B	9.8 GB	3.5 GB	2.9 GB
Qwen2.5-1.5B	1.54B	26 GB	5.8 GB	3.7 GB
Qwen2.5-3B	3.09B	49 GB	9.1 GB	4.8 GB
Mistral-7B-v0.1	7.24B	109 GB	16 GB	5.6 GB
Qwen2.5-7B	7.62B	117 GB	18 GB	7.3 GB
Qwen2.5-14B	14.77B	224 GB	33 GB	12 GB
Qwen2.5-32B	32.76B	493 GB	68 GB	23 GB
Qwen2.5-72B	72.71B	1,091 GB	146 GB	45 GB

Three things stand out. Full fine-tuning of a 7B model needs about 117 GB, which means at least two 80 GB GPUs with sharded optimizer state. LoRA brings the same model to 18 GB, a single 24 GB card. And QLoRA puts a 72B model at 45 GB, inside one 80 GB GPU. Add 1–3 GB for the CUDA context, allocator fragmentation and framework buffers before you buy.

Training memory versus model size for the Qwen2.5 family (log–log), including activations at 4,096 tokens with checkpointing. Dashed lines mark 24 GB and 80 GB GPUs. Source: research/e2_memory/memory.py.

Counting parameters from a config file#

Every number above starts from the model’s config.json. For a decoder with hidden size $h$ , $L$ layers, MLP width $I$ , $n_{h}$ query heads, $n_{k v}$ key/value heads (grouped-query attention), head dimension $d = h / n_{h}$ and vocabulary $V$ :

N_{layer} = q, o, k, v 2 h n_{h} d + 2 h n_{k v} d + SwiGLU MLP 3 h I + norms 2 h (+ biases)

N = V h (1 + [untied head]) + L N_{layer} + h

The SwiGLU MLP has three matrices (gate, up, down), which is why the MLP term is $3 h I$ rather than the $2 h I$ of older GPT-style models; at 7B it is roughly two-thirds of all parameters. Grouped-query attention shrinks k and v by $n_{h} / n_{k v}$ (7× for Qwen2.5-7B). Embedding tying matters for small models: in Qwen2.5-0.5B the 136M-parameter embedding is shared with the output head; untying it would add 28% to the model.

Our counter reproduces the published sizes: 0.49B for Qwen2.5-0.5B, 7.62B for 7B, 72.71B for 72B and 7.24B for Mistral-7B, which is our check that the formula and the configs agree.

What each byte is for#

Training with mixed-precision AdamW keeps four things per trainable parameter. Following the accounting in the ZeRO paper:

Tensor	Precision	Bytes / param
Weights (used in forward/backward)	bf16	2
Gradients	bf16	2
Master copy of weights	fp32	4
Adam first moment m	fp32	4
Adam second moment v	fp32	4
Total, full fine-tuning		16

Mem_{full} \approx 16 N bytes

LoRA freezes the base weights, so they need only their bf16 copy (2 bytes) and no gradients or optimizer state. Each adapted $d_{in} \times d_{out}$ matrix gains $r (d_{in} + d_{out})$ trainable parameters, which carry the full 16 bytes:

Mem_{LoRA} \approx 2 N + 16 N_{LoRA}, N_{LoRA} = L W \in targets \sum r (d_{in}^{W} + d_{out}^{W})

QLoRA stores the frozen base in 4-bit NormalFloat with double-quantized scaling constants, about 4.127 bits (0.516 bytes) per parameter according to the QLoRA paper’s accounting:

Mem_{QLoRA} \approx 0.516 N + 16 N_{LoRA}

Two refinements change these figures in practice. An 8-bit optimizer stores Adam moments in one byte each, cutting the 12-byte optimizer term to roughly 6. ZeRO stage 3 or FSDP shards the 16 bytes across $G$ GPUs, so each holds $16 N / G$ . Full fine-tuning of 72B — 1,083 GB of states — therefore needs at least 14 GPUs of 80 GB before activations; in practice two 8-GPU nodes.

How big is a LoRA adapter?#

Adapter size is linear in rank. For Qwen2.5-7B with adapters on all seven linear projections in every layer:

Rank r	Trainable params (all linear)	% of model	Trainable params (q, v only)
1	2.5M	0.03%	0.32M
2	5.0M	0.07%	0.63M
4	10.1M	0.13%	1.26M
8	20.2M	0.27%	2.52M
16	40.4M	0.53%	5.05M
32	80.7M	1.06%	10.09M
64	161.5M	2.12%	20.19M
128	323.0M	4.24%	40.37M
256	645.9M	8.48%	80.74M

Even at rank 256 the adapter is under a tenth of the model, so the optimizer state is never the problem with LoRA: the frozen base weights and the activations are. That is why rank is a quality decision rather than a memory decision. We measured how much rank actually buys on a real task in our LoRA rank experiment.

Activations: the term that grows with sequence length#

Weights scale with the model; activations scale with the batch. Korthikanti et al. derive the activation memory of one transformer layer, in bytes, for sequence length $s$ , micro-batch $b$ , hidden size $h$ and $a$ attention heads, with 16-bit activations:

A_{layer} = s b h (34 + \frac{5 a s}{h})

The $5 a s / h$ term is the attention score matrix, quadratic in sequence length. Memory-efficient attention kernels such as FlashAttention never materialize it, leaving $34 s bh$ . Full activation checkpointing stores only each layer’s input, $2 s bh$ , and recomputes the rest during the backward pass at the cost of one extra forward pass (see our compute model).

The constant 34 was derived for GPT-style layers; SwiGLU MLPs and grouped-query attention move it somewhat, so treat the no-checkpointing column as an estimate. One term is easy to forget: the output logits, $4 s bV$ bytes in fp32. With a 152k vocabulary and 4,096 tokens that alone is 2.3 GB per sequence — for Qwen2.5-0.5B it is most of the activation budget. Chunked cross-entropy implementations avoid materializing it.

Model	Activations, checkpointed	Activations, not checkpointed
Qwen2.5-0.5B	2.5 GB	5.1 GB
Qwen2.5-1.5B	2.6 GB	7.9 GB
Qwen2.5-3B	2.9 GB	12 GB
Mistral-7B-v0.1	1.5 GB	17 GB
Qwen2.5-7B	3.1 GB	15 GB
Qwen2.5-14B	4.2 GB	34 GB
Qwen2.5-32B	4.8 GB	45 GB
Qwen2.5-72B	7.3 GB	87 GB

Which GPU do you need?#

Mapping the totals onto common cards, with ~2 GB of headroom for framework overhead:

GPU memory	Full fine-tuning	LoRA (bf16 base)	QLoRA
24 GB	up to ~1B	up to ~7B	up to ~14B (32B with short sequences)
48 GB	up to ~2B	up to ~14B	up to ~32B
80 GB	up to ~3B	up to ~32B (tight)	up to ~72B
8 × 80 GB, sharded	up to ~32B	72B	—

Our default for enterprise work: LoRA on bf16 weights when it fits, because QLoRA trades memory for slower steps (de-quantization on every forward pass) and a small quality cost from training against a quantized base. Full fine-tuning when the evaluation shows a gap LoRA cannot close — usually for larger distribution shifts.

Worked example: sizing a 14B QLoRA job#

A client wants to fine-tune Qwen2.5-14B on contracts averaging 8,000 tokens, on a single 48 GB GPU.

Frozen base in NF4: $0.516 \times 14.77 B$ ≈ 7.1 GB.
Adapters, rank 16, all linear layers: 68.8M parameters × 16 bytes ≈ 1.0 GB.
Activations at 8,192 tokens with checkpointing: roughly double the 4,096-token figure, ≈ 8.4 GB, of which logits are 4.6 GB.
Overhead: 2–3 GB.

Total ≈ 19 GB: it fits on 48 GB with room for a micro-batch of 2. Chunked cross-entropy would recover most of the logits term; switching to bf16 LoRA (39 GB) would need an 80 GB card.

Six ways to cut training memory#

Technique	What it reduces	Typical saving	Cost
Activation checkpointing	activations	≈ 17× per layer (34 → 2 sbh)	+33% compute
8-bit Adam	optimizer state	12 → ~6 bytes/param	negligible quality impact in most reports
ZeRO-3 / FSDP sharding	weights, grads, optimizer	÷ number of GPUs	communication
Chunked cross-entropy	logits	4sbV → small	minor compute
Gradient accumulation	activations (via smaller micro-batch)	linear in micro-batch	more steps, same tokens
CPU offload of optimizer	optimizer state on GPU	up to 12 bytes/param	slow steps

Sequence packing (concatenating short examples to fill the context) does not lower peak memory, but it removes padding waste, which raises useful tokens per step at the same memory.

Limitations#

This is an analytical model, not a profiler trace. It excludes CUDA context and allocator fragmentation, assumes bf16 compute, ignores MoE routing buffers, and uses the GPT-derived activation constant for all architectures. Microsoft, PyTorch and Hugging Face tooling also allocate temporary buffers during optimizer steps and gradient all-reduce. Treat the tables as the floor you design against, then confirm with one short profiling run on target hardware.

Method and reproducibility

Model configurations downloaded from the official Hugging Face repositories (Qwen2.5 0.5B–72B, Mistral-7B-v0.1). Script: research/e2_memory/memory.py; output: research/results/e2_memory.json. Run with python memory.py; no GPU needed.

References#

Rajbhandari, S. et al. (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC20. arXiv:1910.02054.
Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. arXiv:2305.14314.
Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
Korthikanti, V. et al. (2023). Reducing Activation Recomputation in Large Transformer Models. MLSys 2023. arXiv:2205.05198.
Dao, T. et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. arXiv:2205.14135.
Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115. Model configurations from the official Hugging Face repositories.

How much GPU memory does it take to fine-tune an LLM?