Fine-tune a giant model by training two thin matrices instead of one huge one. Set the rank and watch trainable parameters collapse by orders of magnitude while the update still does its job.
The Decomposition
h = Wx + BAx
W stays frozen. Only A and B receive gradients.
Parameter Comparison
Full fine-tune (d × k)0
LoRA r×(d+k)0
What a Rank-r Update Can Express
A small ΔW = BA built with the current rank. Higher rank produces visibly richer structure; rank 1 can only produce one repeating pattern scaled up and down.
LoRA: Fine-Tuning Without Touching the Model
Fine-tuning a 7B model the traditional way means updating all 7 billion weights — and storing optimizer state for each, which typically needs four to six times the model's own size in GPU memory. LoRA (Low-Rank Adaptation) sidesteps this almost entirely, and it has become the default way to adapt large models.
The Insight: Updates Are Low-Rank
Fine-tuning changes a weight matrix from W to W + ΔW. The observation behind LoRA is that although W is enormously expressive, the change needed to specialise a model for one task has very low intrinsic rank — it does not need the full space.
So instead of learning a full d × k matrix, LoRA factorises the update into two thin matrices:
ΔW = B · A where B is d×r and A is r×k, with r &lll; min(d, k)
The original weights are frozen. Only A and B are trained.
Why the Savings Are So Extreme
A full update on a 4096×4096 layer is 16.7 million parameters. The LoRA version at rank 8 is 8 × (4096 + 4096) = 65,536 — about 0.4%. The saving comes from replacing a product of dimensions with a sum of them.
The memory win is even larger than the parameter count suggests, because optimizers like Adam store two extra values per trainable parameter. Freezing 99.6% of the weights removes that overhead almost completely — which is what lets a 7B model be fine-tuned on a single consumer GPU.
Choosing the Rank
Rank is the one hyperparameter that matters, and it is a capacity dial:
r = 1–4 — extremely cheap; enough for style or tone adjustments.
r = 8–16 — the common default. Handles most instruction-tuning and domain adaptation.
r = 64–128 — for teaching genuinely new capabilities, approaching full fine-tuning behaviour at a fraction of the cost.
Higher is not automatically better: past the task's intrinsic rank you are adding parameters that mostly learn noise. Note also that A is initialised randomly while B is initialised to zero, so BA = 0 at the start and training begins from exactly the pretrained model.
The Practical Superpower: Swappable Adapters
Because the base model is untouched, a LoRA adapter is a small standalone file — often just a few megabytes. One copy of a 70B model in memory can serve many specialisations by swapping adapters, and they can be merged into the weights before deployment (W' = W + BA) so inference costs nothing extra.
Combine LoRA with a 4-bit quantized base and you get QLoRA, the recipe that made fine-tuning very large models possible on a single GPU.
Interactive Exploration Guide
Start at r = 8 on a 4096×4096 layer and read the trainable fraction — well under 1%.
Drag the rank to 128. Parameters climb steeply but remain a small slice of the full matrix. Even a large rank is cheap.
Widen d and k while holding rank fixed. Full fine-tuning grows quadratically; LoRA grows only linearly. The bigger the model, the better LoRA looks.
Watch the ΔW preview at r = 1, then raise the rank. At rank 1 every row is a scaled copy of a single pattern; higher ranks can express genuinely independent structure.
Switch to 70B and compare the adapter file size to the full model. That gap is why model hubs host thousands of adapters instead of thousands of full checkpoints.
Key Takeaway
LoRA does not make the model smaller — it makes the update smaller. By constraining the change to a low-rank subspace, it converts fine-tuning from an infrastructure project into something that runs on one GPU and ships as a few-megabyte file, with no inference penalty once merged.
Cheat sheet
LoRA in LLMs
Fine-tune a giant model by training two thin matrices instead of one huge one. Set the rank and watch trainable parameters collapse by orders of magnitude while the update still does its job.