Definition#
QLoRA (Dettmers et al. 2023) combines LoRA with a quantized frozen base model. Three ingredients make it work: the 4-bit NormalFloat (NF4) data type, designed for normally distributed weights; double quantization, which quantizes the per-block scaling constants themselves, bringing their overhead to about 0.127 bits per parameter; and paged optimizers that move optimizer state to CPU memory during spikes.
During training, each 4-bit weight block is de-quantized to bf16 for the matrix multiply; gradients flow through the frozen weights into bf16 adapters. Memory per parameter of the frozen base falls from 2 bytes to about 0.52:
Trade-offs#
Steps are slower than bf16 LoRA because of de-quantization, and adapters are trained against a quantized approximation of the base. Merge adapters into a bf16 copy of the base before serving and re-run the evaluation. For a 7B model QLoRA needs about 7.3 GB including activations, against 17.9 GB for LoRA (memory model).