Is LoRA worse than full fine-tuning?

On hard, broad domain adaptation such as code and mathematics, published results show LoRA under-performing full fine-tuning. On narrow tasks like extraction and classification the gap is usually small, and LoRA forgets less.

Does QLoRA reduce quality?

The QLoRA paper reports matching 16-bit fine-tuning on its benchmarks. In practice we use it when memory requires it and verify on the task evaluation after merging and quantizing for serving.

What LoRA rank should I use?

Rank 8–16 is a sound default for narrow tasks. Increase it if inputs in production will differ from training data; in our experiment higher ranks generalized better to unseen formats.

LoRA vs QLoRA vs Full Fine-Tuning: Memory, Speed, Quality Compared

At a glance#

	Full fine-tuning	LoRA	QLoRA
Weights trained	all	low-rank adapters (0.3–2%)	low-rank adapters
Base weights stored as	bf16 + fp32 master	bf16, frozen	4-bit NF4, frozen
Training memory, 7B	117 GB	18 GB	7.3 GB
Training memory, 72B	1,091 GB	146 GB	45 GB
FLOPs per token	≈ 6N	≈ 4N	≈ 4N + de-quantization
Step speed	baseline	similar or faster	slower (de-quantization)
Forgetting	most	less	less
Serving	one model per task	merge, or many adapters on one base	merge into bf16 or serve quantized

Memory from our memory model at 4,096 tokens with activation checkpointing; FLOPs from our compute model.

How each one works#

Full fine-tuning updates every weight $W \leftarrow W - η \nabla_{W} L$ , carrying 16 bytes of weight, gradient and AdamW state per parameter.

LoRA freezes $W_{0}$ and learns a rank- $r$ update for each targeted matrix:

W = W_{0} + \frac{α}{r} B A, A \in R^{r \times d_{in}}, B \in R^{d_{out} \times r}, B initialised to 0

so training starts exactly at the base model. Trainable parameters per matrix are $r (d_{in} + d_{out})$ instead of $d_{in} d_{out}$ : for a 3,584 × 18,944 MLP matrix in Qwen2.5-7B at $r = 16$ , that is 360k instead of 68M.

QLoRA is LoRA on a base model stored in 4-bit NormalFloat, de-quantized on the fly to bf16 for each matrix multiply. Gradients flow through the frozen quantized weights into the bf16 adapters.

What the evidence says about quality#

Biderman et al. compared LoRA and full fine-tuning on code and mathematics at scale and found LoRA substantially under-performs full fine-tuning in that setting, while forgetting less of the base model’s other abilities. They also found that full fine-tuning learns weight perturbations with ranks 10–100× higher than typical LoRA configurations — a plausible reason for the gap on hard, broad domain adaptation.

Narrow enterprise tasks sit at the other end of the spectrum. In our structured-extraction experiment on a 0.5B model, LoRA r=16 reached 100.0% in-distribution and 97.0% on unseen formats, against 100.0% and 91.3% for full fine-tuning, while training 1.8% of the parameters. Rank mattered for unseen formats, not for familiar ones.

QLoRA’s authors report matching 16-bit fine-tuning quality on their benchmarks. In our client work we treat QLoRA as a memory tool: when a model fits for LoRA in bf16, we use bf16.

A decision rule#

Start with LoRA in bf16, adapters on all linear layers, rank 16, as long as it fits your GPUs (a 7B model needs one 24 GB card, 14B about 33 GB).
Use QLoRA when the model you need doesn’t fit otherwise — typically 32B–72B on a single 48–80 GB GPU — and accept slower steps.
Move to full fine-tuning when the evaluation shows a gap LoRA can’t close at higher ranks, especially for broad domain shifts (new language, code, heavy domain vocabulary), and you have the multi-GPU memory budget.
Validate on held-out formats, not only held-out examples, before choosing rank.

Serving considerations#

A LoRA adapter can be merged into the base weights ( $W = W_{0} + \frac{α}{r} B A$ ) for zero inference overhead, or kept separate so one base model serves many adapters; serving engines such as vLLM support batching requests for different adapters together. That makes LoRA attractive when you will run many task-specific models: one 7B base plus twenty adapters of ~40 MB each, instead of twenty 15 GB models.

For QLoRA-trained adapters, merge into a bf16 copy of the base (not the 4-bit one) and re-quantize for serving if needed, then re-run the evaluation: adapters were trained against de-quantized weights, and small mismatches can cost a point.

References#

Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.
Biderman, D. et al. (2024). LoRA Learns Less and Forgets Less. TMLR. arXiv:2405.09673.
Rajbhandari, S. et al. (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054.

LoRA vs QLoRA vs full fine-tuning: how to choose