LoRA (Low-Rank Adaptation): definition, math and practical numbers

Definition#

LoRA (Low-Rank Adaptation, Hu et al. 2021) is a parameter-efficient fine-tuning method. Instead of updating a weight matrix $W_0\in {\mathbb{R}}^{d_{\text{out}}\times d_{\text{in}}}$, it learns two small matrices whose product is the update:

$B$ starts at zero so training begins at the base model; $r$ (the rank, typically 4–64) and $\alpha$ (a scaling factor) are hyperparameters. Each adapted matrix gains $r(d_{\text{in}}+d_{\text{out}})$ trainable parameters.

In numbers#

For Qwen2.5-7B with rank 16 on all linear layers, LoRA trains 40.4M parameters (0.53% of the model), and weight-plus-optimizer memory falls from about 113 GB to 15 GB (memory model). Compute per token falls only by about a third, from $6N$ to $4N$, because activation gradients still flow through every layer.

Choosing the rank#

In our experiment, rank 4 already reached 100% on familiar inputs; on unseen formats accuracy rose from 85% at rank 1 to 97% at rank 16. Rank buys robustness more than in-distribution accuracy.

Related: QLoRA, LoRA vs QLoRA vs full fine-tuning.