LoRA rank vs full fine-tuning: an experiment on structured extraction

Summary#

We trained Qwen2.5-0.5B-Instruct to turn free-text insurance claim notes into a five-field JSON record, comparing zero-shot prompting, LoRA at ranks 1, 4, 16 and 64, and full fine-tuning, with 800 training examples. The headline numbers (exact match on all five fields):

On the in-distribution test set, every fine-tuned configuration from rank 4 upward reached 100%. The difference appeared on the out-of-distribution set — note formats the model never saw in training — where exact match rose from 85.0% at rank 1 to 97.0% at rank 16 and 98.0% at rank 64, against 91.3% for full fine-tuning.

The task and data#

Each example is a claim note in one of five templates (first-notice-of-loss notes, call summaries, agent notes) with a policy number, a date of loss in one of four formats, a loss described in everyday language (“found water coming through the kitchen ceiling”), an amount in one of nine currency formats, and distractors: phone numbers, reference numbers and call dates that must not be confused with the date of loss. The target is:

{"policy_number": "HM-341804", "date_of_loss": "2023-06-15",
 "loss_type": "storm", "amount": 893.0, "currency": "EUR"}

Loss type is one of eight categories that must be inferred from descriptions such as “had a chip-pan fire in the kitchen” or “accidentally damaged a neighbour’s car with a ladder” (liability, not collision). The unseen-formats test set uses three templates that never appear in training: a customer email in the first person, a pipe-delimited system record, and a voicemail transcript. It measures whether the model learned the task or the templates.

Data is generated deterministically by data.py: 800 training examples, 300 in-distribution and 300 unseen-format test examples, with disjoint random seeds.

Training setup#

LoRA was implemented by hand (no adapter library) so the parameter accounting is explicit. Each adapted linear layer $W_0\in {\mathbb{R}}^{d_{\text{out}}\times d_{\text{in}}}$ computes

with adapters on all seven linear projections (q, k, v, o, gate, up, down) in all 24 layers, $\alpha =2r$, learning rate $2\times 10^{-4}$. Full fine-tuning used $2\times 10^{-5}$. Both: AdamW, no weight decay, 10% linear warm-up then linear decay, batch 16, two epochs (100 steps), gradient clipping at 1.0, fp32, loss on the JSON target only. Decoding was greedy with an 80-token limit. Hardware: an Apple M4 Max GPU via PyTorch MPS, shared with other workloads, so wall-clock times are indicative only.

Where the gains came from#

Field-level accuracy on unseen formats shows which skills each configuration learned:

The base model already copied policy numbers and amounts almost perfectly; prompting was enough for those. It failed on loss type (39%), currency (59%) and date of loss (81%): the fields that require mapping a convention (“pounds” → GBP, “had roof tiles torn off in the storm” → storm, choosing the loss date rather than the call date). Those conventions are exactly what fine-tuning teaches. Rank 1 learned them on familiar templates but transferred less of the currency and date logic to new formats.

Why full fine-tuning generalized worse here#

Full fine-tuning matched LoRA on familiar templates (100%) but dropped to 91.3% on unseen formats, 5.7 points below rank 16. Its errors concentrated in two fields: date of loss (93.0%) and amount (98.0%). Updating all 494M parameters on 800 examples from five templates let the model fit those templates more tightly — where the date sits, how the amount is phrased — and that specificity did not transfer.

Is the gap real? Treating the two runs as independent samples of 300 items, the standard error of the difference is about 1.9 points, so 5.7 points is roughly 3.0 standard errors — unlikely to be test-set noise. It is still one seed and one learning rate per method, and run-to-run variance in fine-tuning can be several points; we would not generalize from it without repeats. What it does show is that “more trainable parameters” is not automatically “better” when training data is narrow, which is consistent with the finding of Biderman et al. that LoRA acts as a regularizer and forgets less.

How much data was needed?#

We repeated LoRA r=16 with 100 and 200 training examples (four epochs, so the number of optimizer steps is comparable):

With only 100 examples the model reached 97.0% in-distribution and 91.0% on unseen formats. The pattern matches what we see in client work: a few hundred clean examples teach the format and most conventions; robustness to unfamiliar inputs is what the extra data buys.

What it cost#

LoRA r=1: 12.3 min; LoRA r=4: 7.3 min; LoRA r=16: 17.4 min; LoRA r=64: 8.4 min; Full fine-tuning: 22.9 min of training on the shared laptop GPU. Trainable parameters ranged from 0.55M (rank 1, 0.11% of the model) to 494M for full fine-tuning. Our compute model converts these runs to achieved throughput, and our memory model explains why the same recipe on a 7B model needs a 24 GB GPU with LoRA but two 80 GB GPUs for full fine-tuning.

Interpretation#

Three conclusions, with the caveats below:

1. Fine-tuning is the difference between unusable and production-grade here. Zero-shot exact match was 18% even though every field-level score except three was high. All-fields-correct is what downstream systems need.
2. Rank is a robustness knob more than an accuracy knob. On inputs like the training data, rank 4 was enough. On new formats, higher rank kept paying. If your production inputs drift — new channels, new templates, new partners — don’t pick rank from an in-distribution validation set.
3. Full fine-tuning did not beat LoRA r=16 on unseen formats. Full fine-tuning reached 91.3% on unseen formats versus 97.0% for LoRA r=16, training 56× as many parameters. Biderman et al. report that LoRA under-performs full fine-tuning on harder, larger-scale domain adaptation (code and math) while forgetting less; a narrow extraction task sits at the other end of that spectrum.

Reproducing it#

cd research && uv venv .venv -p 3.12 && uv pip install -p .venv/bin/python torch transformers numpy
cd e1_lora_rank
python data.py                  # prints sample examples
../.venv/bin/python run.py --mode base
../.venv/bin/python run.py --mode lora --rank 16
../.venv/bin/python run.py --mode full
./sweep.sh                      # all eight runs, ~2 h on a laptop GPU

Each run writes results/<name>.json with trainable parameter count, wall-clock time, a loss curve and field-level metrics on both test sets. The page you are reading is generated from those files at build time.

What we would test next#

• Seeds: three seeds per configuration to put error bars on every cell.
• Learning-rate sweep for full fine-tuning: a lower rate or one epoch may close the generalization gap.
• Rank-stabilized scaling ($\alpha /\sqrt{r}$) to separate the effect of rank from the effect of update magnitude.
• A 7B model on real, de-identified claim notes, where base-model zero-shot performance is much higher and the headroom smaller.

Limitations#

One seed per configuration, so differences of a point or two are within run-to-run noise (300 test items give a 95% interval of roughly ±2–3 points near 95% accuracy). One small model, one synthetic task, one learning rate per method; a sweep could change the LoRA–full comparison. $\alpha =2r$ scaling means higher ranks also have larger effective update magnitude; rank-stabilized scaling ($\alpha /\sqrt{r}$) is an alternative. Synthetic templates are cleaner than real claim notes. We present this as a well-instrumented case study, not a general law.

References#

1. Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
2. Biderman, D. et al. (2024). LoRA Learns Less and Forgets Less. TMLR. arXiv:2405.09673.
3. Kalajdzievski, D. (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arXiv:2312.03732.
4. Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115. Model configurations from the official Hugging Face repositories.