LoRA and QLoRA math: rank, alpha, memory, and what actually trains

Fine-tuning every weight of a 7B model needs over 100 GB of GPU memory. LoRA trains well under 1% of the parameters instead and gets surprisingly close. QLoRA goes further and keeps the frozen model in 4 bits. Here’s the arithmetic of both, and where the shortcut costs you.

The idea

A fine-tuned weight matrix is the original plus a change: W' = W + ΔW. LoRA (Hu et al., 2021) bets that the change has low rank, and trains it as a product of two thin matrices:

W' = W + (α / r) · B · A
W: d × k (frozen)    B: d × r    A: r × k    r ≪ min(d, k)

A starts random and B starts at zero, so the model begins exactly where it was. Only A and B receive gradients.

How many parameters that is

One 4096 × 4096 projection has 16.8M weights. A rank-16 adapter on it has 16 × (4096 + 4096) = 131,072 parameters, 0.78% of the matrix.

Across a whole Llama-2-7B-shaped model (32 layers, hidden size 4096, MLP width 11008), with adapters on all seven linear layers in every block at rank 16:

q, k, v, o:   4 × 16 × (4096 + 4096)  = 524,288
gate, up:     2 × 16 × (4096 + 11008) = 483,328
down:         1 × 16 × (11008 + 4096) = 241,664
per layer                             = 1,249,280
× 32 layers                           ≈ 40.0M   (0.59% of 6.74B)

Forty million trainable parameters out of 6.74 billion.

Why memory collapses

Full fine-tuning with Adam in mixed precision costs about 16 bytes per parameter:

• 2 bytes for bf16 weights and 2 for gradients;
• 4 for the fp32 master copy;
• 4 + 4 for Adam’s two moment estimates.

For 6.74B parameters that’s about 108 GB, before activations.

With LoRA, the frozen base only needs its weights: 13.5 GB in bf16. The 16 bytes per parameter apply only to the 40M adapter parameters, about 0.64 GB. Activations come on top, and that’s where gradient checkpointing earns its keep.

QLoRA (Dettmers et al., 2023) keeps the frozen base in 4-bit NF4, a data type with levels placed for normally distributed weights. It also quantises the quantisation constants themselves (double quantisation), for about 4.127 bits per parameter in total. The base is dequantised on the fly in the forward pass, and gradients flow through it into the bf16 adapters:

Base model	bf16 frozen	NF4 frozen (QLoRA)
7B	13.5 GB	3.5 GB
27B	54 GB	13.9 GB

That’s why a 27B model can be adapted on one 24 GB card at all: a 14 GB base, small adapters, and the rest for activations at modest batch sizes and sequence lengths.

What alpha does

The update is scaled by α / r. Raise the rank with α fixed and each rank direction contributes less, so the learning rate effectively changes with rank. Many setups set α to r or 2r and leave it. rsLoRA (Kalajdzievski, 2023) argues for α / √r, which keeps updates stable as rank grows. Whichever you use, if you change the rank, re-check the learning rate.

Where to put adapters

The original paper adapted only the attention projections. QLoRA found that adapting every linear layer, MLP included, was needed to match full fine-tuning quality. It costs more parameters, but at these sizes they’re cheap. I default to all linear layers.

What LoRA is good and bad at

LoRA is good at teaching behaviour: a format, a tone, tool-call discipline, a reasoning style, a domain’s conventions. That’s what my lora repo is for: a small, effective way to fine-tune a reasoning model.

It’s weaker at adding large amounts of new knowledge. Biderman et al. (2024) found LoRA learns less than full fine-tuning on continued pre-training for code and maths, and also forgets less of what the base model knew. That’s the trade. If the job is “know a new domain deeply”, you need full fine-tuning or a lot more rank. If the job is “behave like this, reliably”, LoRA is usually the better deal.

Rules I use

1. Adapt all linear layers at rank 16–64 before reaching for higher rank.
2. Treat rank and learning rate as a pair. Change one, re-tune the other.
3. Use QLoRA when the base doesn’t fit in bf16. The NF4 base costs a little quality; not training at all costs more.
4. Merge for serving. W + (α/r)·BA folds back into one matrix, so inference pays nothing extra. Then re-quantise the merged weights for deployment.

Sources: Hu et al., LoRA (2021). Dettmers et al., QLoRA (2023). Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (2023). Biderman et al., LoRA Learns Less and Forgets Less (2024). Parameter counts are computed for a Llama-2-7B-shaped model.