Fine-tune a giant model by training two thin matrices instead of one huge one. Set the rank and watch trainable parameters collapse by orders of magnitude while the update still does its job.
The Decomposition
h = Wx + BAx
W stays frozen. Only A and B receive gradients.
Parameter Comparison
Full fine-tune (d × k)0
LoRA r×(d+k)0
What a Rank-r Update Can Express
A small ΔW = BA built with the current rank. Higher rank produces visibly richer structure; rank 1 can only produce one repeating pattern scaled up and down.
Overview
The Insight: Updates Are Low-Rank
Fine-tuning changes a weight matrix from W to W + ΔW. The observation behind LoRA is that though W is enormously expressive, the change needed to specialise a model for one task has very low intrinsic rank. It does not need the full space.
So instead of learning a full d × k matrix, LoRA factorises the update into two thin matrices:
ΔW = B · A where B is d×r and A is r×k, with r &lll; min(d, k)
The first weights are frozen. Only A and B are trained. If the frozen matrix W is d-by-d and the rank r is 16, A is d-by-16 and B is 16-by-d. So the trainable parameter count drops from d-squared to 2dr. A reduction of roughly 10,000x for a 7B model's attention weights. At inference time the product BA is added right to W. This gives a single merged matrix with zero extra latency. Multiple LoRA adapters can coexist: each task gets its own A and B, and switching tasks means swapping two small matrices while the base model stays in memory once. This is how services like Hugging Face PEFT serve hundreds of fine-tuned variants from one base checkpoint. The rank r is the main hyperparameter: too low and the adapter cannot represent the task's deviation from the base model; too high and it overfits or loses the speed edge. In practice, r between 4 and 64 covers most tasks. A is set up with a random Gaussian and B with zeros. So the adapter starts as a no-op and learns the delta. This page freezes a weight matrix, trains a low-rank adapter beside it, and shows how the merged output matches the full fine-tune within a controlled tolerance.
Fine-tune a giant model by training two thin matrices instead of one huge one.
Fine-tuning a 7B model the traditional way means updating all 7 billion weights — and storing optimizer state for each, which typically needs four to six times the model's own size in GPU memory. LoRA (Low-Rank Adaptation) sidesteps this almost entirely, and it has become the default way to adapt large models.
The Insight: Updates Are Low-Rank
Fine-tuning changes a weight matrix from W to W + ΔW. The observation behind LoRA is that though W is enormously expressive, the change needed to specialise a model for one task has very low intrinsic rank. It does not need the full space.
So instead of learning a full d × k matrix, LoRA factorises the update into two thin matrices:
ΔW = B · A where B is d×r and A is r×k, with r &lll; min(d, k)
The first weights are frozen. Only A and B are trained. If the frozen matrix W is d-by-d and the rank r is 16, A is d-by-16 and B is 16-by-d. So the trainable parameter count drops from d-squared to 2dr. A reduction of roughly 10,000x for a 7B model's attention weights. At inference time the product BA is added right to W. This gives a single merged matrix with zero extra latency. Multiple LoRA adapters can coexist: each task gets its own A and B, and switching tasks means swapping two small matrices while the base model stays in memory once. This is how services like Hugging Face PEFT serve hundreds of fine-tuned variants from one base checkpoint. The rank r is the main hyperparameter: too low and the adapter cannot represent the task's deviation from the base model; too high and it overfits or loses the speed edge. In practice, r between 4 and 64 covers most tasks. A is set up with a random Gaussian and B with zeros. So the adapter starts as a no-op and learns the delta. This page freezes a weight matrix, trains a low-rank adapter beside it, and shows how the merged output matches the full fine-tune within a controlled tolerance.
Why the Savings Are So Extreme
A full update on a 4096×4096 layer is 16.7 million parameters. The LoRA version at rank 8 is 8 × (4096 + 4096) = 65,536 — about 0.4%. The saving comes from replacing a product of dimensions with a sum of them.
The memory win is even larger than the parameter count suggests, because optimizers like Adam store two extra values per trainable parameter. Freezing 99.6% of the weights removes that cost almost fully —. This is what lets a 7B model be fine-tuned on a single consumer GPU.
Choosing the Rank
Rank is the one hyperparameter that matters, and it is a capacity dial:
r = 1–4 — very cheap; enough for style or tone adjustments.
r = 8–16 — the common default. Handles most instruction-tuning and domain adaptation.
r = 64–128 — for teaching truly new powers, approaching full fine-tuning behaviour at a fraction of the cost.
Higher is not by itself better: past the task's intrinsic rank you are adding parameters that mostly learn noise. Note also that A is set up randomly while B is set up to zero. So BA = 0 at the start and training begins from exactly the pretrained model.
The Practical Superpower: Swappable Adapters
Because the base model is untouched, a LoRA adapter is a small standalone file — often just a few megabytes. One copy of a 70B model in memory can serve many specialisations by swapping adapters. And they can be merged into the weights before deployment (W' = W + BA) so inference costs nothing extra.
Combine LoRA with a 4-bit quantized base and you get QLoRA, the recipe that made fine-tuning very large models doable on a single GPU.
Training a small update instead of the whole model
Fine-tuning a 7-billion-parameter model often means updating all 7 billion weights. With Adam, that needs the weights, their gradients, and two optimiser moments. Roughly 4× the model in memory, so about 112GB in 32-bit. Beyond most hardware.
LoRA changes what is trained. The pretrained weights are frozen, and for each target matrix a small pair of matrices is trained alongside it:
h = W₀x + BAx where A is r×d and B is d×r
W₀ never changes. A and B are the learned update. And their product has the same shape as W₀ while holding far fewer parameters,. This is because r — the rank — is small.
For a 4096×4096 weight matrix with r = 8:
Parameters
Full matrix
16,777,216
A (8×4096) + B (4096×8)
65,536
Ratio
0.4%
Across a whole model, LoRA often trains 0.1–1% of the parameters. A few million instead of billions.
Why a low-rank update is enough
The empirical claim behind the method is that the update needed to adapt a pretrained model to a task has low intrinsic rank. The base model already knows the language; adaptation is a comparatively small, structured change.
That is an observation rather than a theorem. And it has held up well: LoRA fine-tunes often match full fine-tuning on task metrics, sometimes within noise, sometimes better. This is because there is less capacity to overfit.
Two details make it practical:
A is set up randomly and B to zero, so BA = 0 at the start and the model begins exactly as the pretrained one. Training then moves it slowly rather than disrupting it on the first step.
The adapter can be merged. After training, W₀ + BA can be computed once and stored as a single matrix. So inference has no extra cost. Or the adapter can be kept separate and swapped — which is what allows one base model to serve many tasks.
The parameters that matter
Parameter
Meaning
Typical
r
Rank of the update
8–64
alpha
Scaling applied to BA
Often 2×r
target_modules
Which matrices get adapters
Attention projections, often the FFN too
dropout
Dropout on the adapter
0.05–0.1
Rank is the capacity dial. 8 is enough for style and format adaptation; 32–64 for teaching truly new behaviour or a specialised domain. Larger is not by itself better — it adds capacity to overfit a small dataset.
Which modules matters more than people expect. Adapting only the query and value projections was the first recipe; adapting all attention projections plus the feed-forward layers often works better, at more parameters.
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters() # ~0.5% of total
Training two thin matrices instead of one fat one
LoRA freezes the model and learns a low-rank correction beside it. The whole method is one factorisation, and the arithmetic explains both why it saves so much memory and what it cannot represent.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(5)
D_IN, D_OUT = 512, 512
print("ONE WEIGHT MATRIX from a transformer, %d x %d = %s parameters."
% (D_IN, D_OUT, "{:,}".format(D_IN * D_OUT)))
print("fine-tuning normally means learning a change dW of the same shape.")
print()
print("THE OBSERVATION LoRA RESTS ON: dW is usually close to LOW RANK.")
print("a full-rank %dx%d matrix needs %s numbers. a rank-r one needs only"
% (D_IN, D_OUT, "{:,}".format(D_IN * D_OUT)))
print("r*(%d + %d), because you can store it as two thin matrices:" % (D_IN, D_OUT))
print(" dW = B @ A with A: r x %d and B: %d x r" % (D_IN, D_OUT))
print()
print("%-8s %16s %16s %14s %12s"
% ("rank r", "A params", "B params", "total", "vs full"))
full = D_IN * D_OUT
for r in (1, 2, 4, 8, 16, 64, 512):
tot = r * D_IN + r * D_OUT
print("%-8d %16s %16s %14s %11.2f%%"
% (r, "{:,}".format(r * D_IN), "{:,}".format(r * D_OUT),
"{:,}".format(tot), 100.0 * tot / full))
print(" at r = 8 you are training %.2f%% of the parameters."
% (100.0 * (8 * (D_IN + D_OUT)) / full))
print(" the two forms cost exactly the same at r = %d, and above that"
% (full // (D_IN + D_OUT)))
print(" the factorisation is LARGER than the matrix it factorises. that")
print(" break-even is why r is always small -- there is no version of")
print(" this idea that saves anything at high rank.")
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
print("DOES IT ACTUALLY WORK? build a low-rank change and try to recover")
print("it. first a genuinely low-rank target:")
TRUE_R = 4
Bt = rng.normal(0, 1, (D_OUT, TRUE_R))
At = rng.normal(0, 1, (TRUE_R, D_IN))
dW_true = Bt @ At / np.sqrt(D_IN)
def best_rank_r(M, r):
U, S, Vt = np.linalg.svd(M, full_matrices=False)
return (U[:, :r] * S[:r]) @ Vt[:r], S
print("%-10s %20s %16s"
% ("rank used", "relative Frobenius err", "1 - err"))
_, S_true = best_rank_r(dW_true, 1)
for r in (1, 2, 4, 8, 16):
approx, S = best_rank_r(dW_true, r)
err = np.linalg.norm(dW_true - approx) / np.linalg.norm(dW_true)
print("%-10d %20.6f %15.2f%%" % (r, err, 100 * (1 - err)))
print(" the error hits 0 at exactly r = %d, the true rank, and stays" % TRUE_R)
print(" there. a rank-%d approximation of a rank-%d matrix is EXACT --"
% (TRUE_R, TRUE_R))
print(" not close, exact -- and every rank above it is wasted capacity.")
print()
Output
The following block covers the left steps.
example_03.pyNumPy
import numpy as np
print("NOW A TARGET THAT IS NOT LOW RANK -- a full-rank random change:")
dW_full = rng.normal(0, 1, (D_OUT, D_IN)) / np.sqrt(D_IN)
print("%-10s %20s %16s"
% ("rank used", "relative Frobenius err", "1 - err"))
for r in (1, 2, 4, 8, 16, 64, 256):
approx, S = best_rank_r(dW_full, r)
err = np.linalg.norm(dW_full - approx) / np.linalg.norm(dW_full)
print("%-10d %20.6f %15.2f%%" % (r, err, 100 * (1 - err)))
print(" rank 8 gets a random full-rank change %.1f%% of the way there."
% (100 * (1 - np.linalg.norm(dW_full - best_rank_r(dW_full, 8)[0])
/ np.linalg.norm(dW_full))))
print(" LoRA is not")
print(" a general-purpose compression -- it is a BET that the update you")
print(" need has low intrinsic rank, and that bet is empirical. it holds")
print(" for adapting a pretrained model to a task, and does not hold for")
print(" training from scratch.")
print()
print("LOOK AT THE SINGULAR VALUES to see why the bet usually pays:")
_, S_low = best_rank_r(dW_true, 1)
_, S_rand = best_rank_r(dW_full, 1)
print("%-10s %18s %18s" % ("index", "low-rank target", "random target"))
for i in (0, 1, 3, 4, 8, 32, 128):
print("%-10d %18.6f %18.6f" % (i, S_low[i], S_rand[i]))
print(" the low-rank target's singular values fall off a cliff after")
print(" index %d. the random one decays slowly and never reaches zero."
% (TRUE_R - 1))
print(" 'is this matrix low rank' is exactly the question 'do its")
print(" singular values collapse', and for real fine-tuning updates they")
print(" largely do.")
print()
Output
The next block goes on with the left demonstrations.
example_04.pyNumPy
import numpy as np
print("THE INITIALISATION, which is not an arbitrary choice. LoRA sets")
print("A to random and B to ZERO, so that at step 0:")
A0 = rng.normal(0, 0.02, (8, D_IN))
B0 = np.zeros((D_OUT, 8))
print(" dW = B @ A = %s, max |dW| = %.1e"
% ("all zeros" if np.abs(B0 @ A0).max() == 0 else "nonzero",
float(np.abs(B0 @ A0).max())))
print(" the adapted model is EXACTLY the base model before training")
print(" starts. no warm-up, no degradation, no risk of the random")
print(" initialisation damaging the pretrained weights.")
print(" both to zero would not work: the gradient of B @ A with respect")
print(" to both would be zero as well, and nothing would ever move.")
print(" one of them has to be nonzero to carry gradient, and the other")
print(" has to be zero to keep the product zero.")
print()
print("AND THE SCALING FACTOR. LoRA applies alpha/r, not just dW:")
print(" W_effective = W + (alpha / r) * B @ A")
print("%-10s %10s %14s %s" % ("r", "alpha", "alpha/r", "effect"))
for r, alpha in ((4, 8), (8, 8), (8, 16), (16, 16), (16, 32), (64, 16)):
print("%-10d %10d %14.4f %s"
% (r, alpha, alpha / float(r),
"amplified" if alpha > r else
"neutral" if alpha == r else "damped"))
print(" the point of dividing by r is that it lets you change r without")
print(" retuning the learning rate. without it, doubling r roughly")
print(" doubles the size of the update at the same learning rate, and")
print(" every rank experiment needs its own sweep.")
Output
Building on the same setup, the next block shows the left cases.
example_05.pyNumPy
print(" alpha = r is the neutral setting and alpha = 2r is the common")
print(" default -- which is simply 'apply the update at double strength'.")
print()
print("WHAT YOU ACTUALLY SAVE, for a 7-billion-parameter model:")
BASE = 7e9
print("%-34s %16s %s" % ("", "trainable params", "optimiser memory"))
print("%-34s %16s %s"
% ("full fine-tune", "{:,.0f}".format(BASE), "~%.0f GB (Adam, fp32)"
% (BASE * 12 / 1e9)))
lora_p = 0.0015 * BASE
print("%-34s %16s %s"
% ("LoRA r=8 on attention only", "{:,.0f}".format(lora_p),
"~%.2f GB" % (lora_p * 12 / 1e9)))
print(" Adam keeps two moments per trainable parameter plus the")
print(" parameter itself, which is where the 12 bytes comes from. that")
print(" is the memory that actually decides whether a fine-tune fits on")
print(" your GPU, and LoRA cuts it by %.0fx."
% (BASE / lora_p))
print(" the frozen base weights still have to be held for the forward")
print(" pass -- LoRA does not shrink the model, only what you train.")
print()
print("AND THE PROPERTY THAT MADE IT UBIQUITOUS: because dW = B @ A is")
print("just a matrix, you can ADD it into W when you are done.")
print(" W_merged = W + (alpha/r) * B @ A")
print(" the merged model has the base model's exact shape and speed --")
print(" zero inference overhead, unlike an adapter layer you have to")
print(" run separately. and because the adapter is a few megabytes, you")
print(" can keep dozens of them for one base model and swap them per")
print(" request, which is how a single served model offers many")
print(" fine-tunes at once.")
Output
Things to try
Start at r = 8 on a 4096×4096 layer and read the trainable fraction. Well under 1%.
Drag the rank to 128. Parameters climb steeply but remain a small slice of the full matrix. Even a large rank is cheap.
Widen d and k while holding rank fixed. Full fine-tuning grows quadratically. LoRA grows only linearly. The bigger the model, the better LoRA looks.
Watch the ΔW preview at r = 1. Then raise the rank. At rank 1 every row is a scaled copy of a single pattern; higher ranks can express truly free structure.
Switch to 70B and compare the adapter file size to the full model. That gap is why model hubs host thousands of adapters instead of thousands of full checkpoints.
Worth recalling
LoRA does not make the model smaller — it makes the update smaller. By constraining the change to a low-rank subspace, it converts fine-tuning from an infrastructure project into something that runs on one GPU and ships as a few-megabyte file, with no inference cost once merged.
QLoRA: fine-tuning on one consumer GPU
LoRA reduces the trainable parameters. The frozen base model still has to be held in memory, and 7 billion parameters in 16-bit is 14GB before anything else.
QLoRA quantises the frozen base to 4-bit — about 3.5GB — while training the LoRA adapters in higher precision. Gradients flow through the quantised weights without updating them.
That combination is what put fine-tuning a 7B model within reach of a single 24GB card. And a 70B model within reach of two.
Approach
Memory for a 7B model
Full fine-tuning (Adam, fp32)
~110GB
Full fine-tuning (mixed precision)
~60GB
LoRA (fp16 base)
~16GB
QLoRA (4-bit base)
~6GB
The quality cost of the quantisation is small — measurably present, and often acceptable given what it enables.
Adapters as deployable artefacts
Because a LoRA adapter is a few megabytes rather than gigabytes, it changes how fine-tuned models are shipped.
One base, many adapters. Serve a single base model in memory and load a small adapter per customer, per task or per tone. Serving frameworks support switching adapters per request, so a single deployment handles many fine-tunes.
Cheap distribution. A 20MB adapter can be versioned in git and downloaded in seconds. A 14GB full fine-tune cannot.
Cheap testing. Training a few adapters and comparing them is affordable, where a few full fine-tunes are not.
The trade: an unmerged adapter adds a small amount of inference latency. Since BAx is computed alongside W₀x. Merging removes it and gives up the ability to swap.
When LoRA is not the answer
You need new knowledge, not new behaviour. Fine-tuning of any kind is a poor way to add facts. Use retrieval.
The domain is very far from pretraining. Adapting an English model to a low-resource language or to a truly novel modality may need more capacity than a low-rank update gives. Think of full fine-tuning or continued pretraining.
You have very little data. Below a few hundred cases, few-shot prompting is often better than any fine-tuning, and much cheaper to iterate.
Absolute maximum quality with unlimited compute. Full fine-tuning still edges ahead on some tasks.
Questions people ask
What rank should I use? 8–16 for style and format; 32–64 for large behaviour change. Tune it on your evaluation set.
What is alpha for? It scales the adapter's contribution by alpha/r. Setting it to twice the rank is a common convention that keeps the effective scale stable as rank changes.
Which modules should I target? All attention projections plus the feed-forward layers is a strong default. Query and value only is cheaper and often slightly worse.
Can I combine a few adapters? Yes — adapters can be summed or weighted. This sometimes composes their behaviours and sometimes interferes.
Does merging change quality? No, if precision matches. Merging into a quantised base can lose a little.
Is LoRA as good as full fine-tuning? On most task metrics, in effect yes. Sometimes better, because there is less capacity to overfit.
Recap in one screen
Freeze the pretrained weights; train a small low-rank pair BA added to selected matrices.
Often 0.1–1% of parameters are trained, with quality close to full fine-tuning.
B starts at zero so training begins from the pretrained model exactly.
QLoRA quantises the frozen base to 4-bit. This puts 7B fine-tuning on a single consumer GPU.
Adapters are megabytes: one base model can serve many, and merging removes the inference cost.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Insight: Updates Are Low-Rank”?
What does this module say about “Why the Savings Are So Extreme”?
What does this module say about “The Practical Superpower: Swappable Adapters”?
Cheat sheet
LoRA in LLMs
Freeze the base model and train two thin matrices whose product approximates the full weight update — 10,000x fewer parameters.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.