Shrink a model by storing its weights in fewer bits. Watch a real weight distribution snap onto a coarse grid, measure the error you introduce, and see how many gigabytes you save.
Overview
The Core Idea: A Coarser Ruler
A 32-bit float can express billions of distinct values. An 8-bit integer can express 256, and a 4-bit integer just 16. Quantization finds the range your weights actually occupy and divides it into that many evenly spaced levels:
Every weight is then snapped to its nearest level. The difference between where a weight was and where it landed is the quantization error — unavoidable, and the entire subject in one sentence.
Weights Snapping to the Grid
originalquantized
Grey dots are the original FP32 weights; green dots are where they land after rounding. Horizontal lines are the representable levels.
Per-Weight Detail
Quantization: Trading Precision for Memory
A 70B-parameter model in full 32-bit precision needs about 280 GB just to hold its weights — far beyond any single GPU. Quantization stores those same weights in fewer bits, and it is the single technique most responsible for large models running on ordinary hardware.
Symmetric vs Asymmetric
Symmetric centres the grid on zero, using a range of ±max(|w|). Simpler and faster, since zero maps exactly to zero — but if your weights are lopsided, half the levels are wasted on values that never occur.
Asymmetric fits the grid to the actual min and max, adding a zero-point offset. It uses every level, at the cost of slightly more arithmetic at inference.
Toggle between them with the outlier enabled and watch the step size and RMS error change.
Outliers Are the Real Enemy
Quantization error does not grow smoothly with bit-width — it is dominated by range. A single weight far from the others stretches the min-max span, and since the step size is that span divided by a fixed number of levels, every other weight gets a coarser grid.
Tick "Inject an outlier" and watch the step size jump while the ordinary weights suddenly land further from home. This is not a toy effect: emergent outlier features in large transformers are precisely why naive INT8 broke down, and why methods like LLM.int8(), GPTQ and AWQ exist — they isolate or protect the outliers instead of letting them ruin the scale for everyone.
Storing weights in fewer bits
A model's weights are numbers, and how precisely each is stored is a choice. Quantisation stores them with fewer bits.
Precision
Bits
7B model size
FP32
32
28GB
FP16 / BF16
16
14GB
INT8
8
7GB
INT4
4
3.5GB
The arithmetic is what makes this matter: a 70B model in 16-bit needs 140GB and does not fit on a single GPU. At 4-bit it is 35GB and fits on one. Quantisation is frequently the difference between a model being deployable and not.
Speed also improves, and for a reason worth understanding: generation is memory-bandwidth-bound, not compute-bound. Every token requires reading all the weights, so halving their size roughly halves the time spent waiting for memory.
The trade, measured
Quality loss is real and, at the sizes people use, modest.
Precision
Typical quality impact
FP16 / BF16
None — the standard baseline
INT8
Very small, often unmeasurable
INT4
Small but present; more visible on reasoning tasks
INT3 and below
Noticeable degradation
Two patterns hold across evaluations. Larger models tolerate quantisation better — a 70B model at 4-bit generally beats a 13B model at 16-bit, and it is the better use of the same memory. And degradation appears first on tasks with narrow margins: multi-step reasoning and code, rather than fluent prose.
The practical guidance follows: prefer a larger model quantised over a smaller model at full precision.
How it works, and why outliers matter
The basic operation maps a range of floats onto a small set of integers:
q = round((w − zero_point) / scale)
Applied per tensor, this fails on transformers for a specific reason: their weight and activation distributions contain outliers, values far larger than the rest. One outlier in a tensor forces the scale to cover its range, and every other value collapses into a handful of levels.
The methods that work all address this:
Group-wise quantisation. Use a separate scale per block of 64 or 128 values rather than per tensor, so an outlier only damages its own group. This is the foundation of most 4-bit formats.
GPTQ quantises layer by layer, adjusting the remaining weights to compensate for the error introduced — a one-shot method needing a small calibration set.
AWQ identifies the weights that matter most for activations and protects them at higher precision.
LLM.int8() keeps outlier dimensions in 16-bit and quantises the rest to 8-bit.
NF4 (used by QLoRA) uses quantisation levels spaced according to a normal distribution, which fits weight distributions better than uniform spacing.
Measuring the outlier collapse
The section above says one outlier forces the scale to cover its range and collapses every other value into a handful of levels. That is a claim with a number attached, so here is the number -- along with the error at each bit width and the exact cost of the group-wise fix.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(7)
W = rng.normal(0, 0.05, 4096) # a typical layer's weights
print("A LAYER'S WEIGHTS: %d values, mean %.5f, std %.5f, range %.4f to %.4f"
% (W.size, W.mean(), W.std(), W.min(), W.max()))
print()
# symmetric linear quantisation: one scale, levels centred on zero
def quantize(w, bits, per_channel_max=None):
qmax = 2 ** (bits - 1) - 1
amax = np.abs(w).max() if per_channel_max is None else per_channel_max
scale = amax / qmax if amax > 0 else 1.0
q = np.clip(np.round(w / scale), -qmax - 1, qmax)
return q, scale
def dequantize(q, scale):
return q * scale
print("THE OPERATION ITSELF is two lines. pick a scale so the largest")
print("weight maps to the largest integer, then round:")
q8, s8 = quantize(W, 8)
print(" scale = max|w| / 127 = %.4f / 127 = %.8f"
% (np.abs(W).max(), s8))
print(" q = round(w / scale)")
print("%-14s %14s %14s %14s" % ("weight", "w / scale", "rounded", "recovered"))
for i in (0, 1, 2, 3):
print("%-14.6f %14.4f %14d %14.6f"
% (W[i], W[i] / s8, int(q8[i]), q8[i] * s8))
print(" the integers are what gets stored. the scale is one float per")
print(" tensor, so the overhead is negligible.")
print()
print("ERROR AT EACH BIT WIDTH:")
print("%-10s %14s %16s %18s %14s"
% ("bits", "levels", "mean abs err", "max err", "err / std"))
for bits in (8, 6, 5, 4, 3, 2):
q, sc = quantize(W, bits)
r = dequantize(q, sc)
e = np.abs(W - r)
print("%-10d %14d %16.6f %18.6f %14.4f"
% (bits, 2 ** bits, e.mean(), e.max(), e.mean() / W.std()))
print(" the error roughly doubles for every bit removed, which is what")
print(" you would expect: half as many levels, twice the gap between")
print(" them, twice the average rounding distance.")
print(" at 8 bits the error is %.1f%% of the weight std -- invisible. at"
% (100 * np.abs(W - dequantize(*quantize(W, 8))).mean() / W.std()))
print(" 2 bits it is %.0f%%, which is not a small perturbation of the"
% (100 * np.abs(W - dequantize(*quantize(W, 2))).mean() / W.std()))
print(" model, it is a different model.")
print()
print("NOW THE THING THAT ACTUALLY BREAKS QUANTISATION: OUTLIERS.")
print("real transformer activations and some weight channels contain a")
print("few values orders of magnitude larger than the rest. add three:")
Wo = W.copy()
Wo[100], Wo[2000], Wo[3500] = 2.5, -3.1, 2.9
print(" %d of %d values are outliers (%.4f%% of the tensor)"
% (3, Wo.size, 100 * 3.0 / Wo.size))
print(" max |w| goes from %.4f to %.4f -- a %.0fx increase"
% (np.abs(W).max(), np.abs(Wo).max(),
np.abs(Wo).max() / np.abs(W).max()))
print()
print("%-24s %16s %16s" % ("", "without outliers", "with outliers"))
for bits in (8, 4):
a = np.abs(W - dequantize(*quantize(W, bits))).mean()
qb, sb = quantize(Wo, bits)
b = np.abs(Wo - dequantize(qb, sb))[np.abs(Wo) < 1.0].mean()
print("%-24s %16.6f %16.6f" % ("mean err at %d bits" % bits, a, b))
print(" the error on the ORDINARY weights got %.0fx worse, and those"
% (np.abs(Wo - dequantize(*quantize(Wo, 4)))[np.abs(Wo) < 1.0].mean()
/ np.abs(W - dequantize(*quantize(W, 4))).mean()))
print(" weights did not change at all. the outliers stretched the scale,")
print(" so every level is now spaced %.0fx further apart and 99.9%% of"
% (np.abs(Wo).max() / np.abs(W).max()))
print(" the tensor is quantised into a handful of levels near zero.")
qb4, _ = quantize(Wo, 4)
print(" distinct integer levels used by the ordinary weights: %d of %d"
% (len(np.unique(qb4[np.abs(Wo) < 1.0])), 16))
print(" three values out of four thousand collapsed the representation")
print(" of everything else. THAT is why naive quantisation of a large")
print(" model produces gibberish, and why every serious method is")
print(" fundamentally an outlier strategy.")
print()
print("THE FIXES, and each one is a different answer to that same problem:")
print()
print("1. GROUPWISE SCALES -- one scale per block of N weights, so an")
print(" outlier only stretches its own block:")
def grouped_err(w, bits, g):
out = np.zeros_like(w)
for i in range(0, w.size, g):
qq, ss = quantize(w[i:i + g], bits)
out[i:i + g] = dequantize(qq, ss)
ordinary = np.abs(w) < 1.0
return float(np.abs(w - out)[ordinary].mean())
print("%-20s %16s %18s" % ("group size", "extra floats", "mean err (4-bit)"))
errs = {}
for g in (Wo.size, 512, 128, 64, 32):
errs[g] = grouped_err(Wo, 4, g)
print("%-20s %16d %18.6f"
% ("whole tensor" if g == Wo.size else str(g), Wo.size // g, errs[g]))
print(" at group size 64 the error on ordinary weights is %.1f%% of what"
% (100 * errs[64] / errs[Wo.size]))
print(" the single whole-tensor scale gave, and it cost %d extra floats"
% (Wo.size // 64))
print(" -- about %.2f of an extra bit per weight. this is exactly what"
% (32.0 / 64))
print(" 'group size 128' means in a quantised model's name, and why")
print(" those names carry a number at all.")
print()
print("2. KEEP THE OUTLIERS IN HIGH PRECISION. store almost all weights")
print(" in 4 bits and the handful of outliers in fp16 -- that is")
print(" LLM.int8() and the mixed-precision family.")
print()
print("3. SMOOTH THEM AWAY FIRST. an outlier in the activations can be")
print(" moved into the weights by rescaling both, since a linear layer")
print(" does not care where a scale factor lives:")
print(" (X / s) @ (s * W) == X @ W")
x = rng.normal(0, 1, (3, 4)); Wm = rng.normal(0, 1, (4, 3)); sc = 4.0
print(" max difference: %.2e"
% np.abs((x / sc) @ (sc * Wm) - x @ Wm).max())
print(" that identity is exact, so you can trade an activation outlier")
print(" for a weight outlier and quantise whichever is easier. that is")
print(" SmoothQuant, and it is the whole idea.")
print()
print("AND ONE NUMBER TO CARRY AWAY from all of this: the difference")
print("between a 4-bit model that works and one that produces gibberish is")
print("not the bit width. it is whether anything was done about the")
print("outliers -- naive and grouped 4-bit differ by %.0fx on exactly the"
% (errs[Wo.size] / errs[64]))
print("same weights, at a cost of half a bit each.")
Output
Try it yourself
Step from FP32 down to INT4. Levels fall from billions to 16, the horizontal grid lines thin out, and the green dots drift visibly away from the grey ones.
Watch RMS error against memory saved. FP16 is nearly free — huge savings, negligible error. That is why it is the default almost everywhere.
Now enable the outlier at INT4. Step size roughly doubles and every ordinary weight is punished for one extreme value.
Switch to asymmetric with the outlier on. Error usually improves, because the grid stops wasting levels on an empty side of the range.
Pick 70B and compare FP16 to INT4. 140 GB versus roughly 35 GB — the difference between a multi-GPU server and a single high-end card. That is the whole reason anyone tolerates the error.
Summing up
Quantization buys memory and speed with precision. The surprise is how cheap the trade is: 8-bit is usually indistinguishable from full precision, and modern 4-bit methods come remarkably close. The size numbers here count weights only — real deployments also need memory for activations and the KV cache, which grows with context length.
Post-training versus quantisation-aware
Post-training quantisation (PTQ) takes a trained model and converts it, using a small calibration dataset to choose scales. Minutes to hours, no training required, and it is what almost everyone uses.
Quantisation-aware training (QAT) simulates quantisation during training so the model learns weights that survive it. Better quality at low bit widths, and it requires the full training pipeline — usually only worth it for very aggressive quantisation or for models shipped at enormous volume.
For LLMs, PTQ with a good method (GPTQ, AWQ) at 4-bit is the standard path.
Format
Character
GGUF
CPU and Apple Silicon inference, llama.cpp ecosystem
GPTQ
GPU inference, widely supported
AWQ
GPU inference, often slightly better than GPTQ at 4-bit
bitsandbytes NF4
Simple to use in transformers; the QLoRA default
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
cfg = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16, # compute in higher precision
bnb_4bit_use_double_quant=True, # quantise the scales too
)
model = AutoModelForCausalLM.from_pretrained("model-name", quantization_config=cfg)
Note compute_dtype: weights are stored in 4-bit and de-quantised to bfloat16 for the actual multiplication. The saving is in memory and bandwidth, not in the arithmetic.
What else can be quantised
Weights are the usual target — they dominate memory and are static, so quantising them is straightforward.
Activations are harder, because they vary per input and contain worse outliers. Weight-only quantisation is much more common for this reason.
The KV cache is increasingly quantised, and for long contexts it can exceed the weights in size. 8-bit cache quantisation is a meaningful saving with a small quality cost.
Gradients and optimiser state during training — 8-bit optimisers reduce Adam's memory, which is twice the model size in 32-bit.
Questions people ask
Which precision should I use? 4-bit for local inference where memory binds, 8-bit when quality matters more than memory, 16-bit when memory is not a constraint.
Does quantisation make inference faster? Usually yes, because generation is bandwidth-bound and smaller weights mean less to read.
Is a quantised large model better than a small one at full precision? Generally yes, for the same memory budget. This is a well-replicated result.
Can I fine-tune a quantised model? With QLoRA, yes — the frozen base is 4-bit and the trained adapters are higher precision.
Which format for a Mac? GGUF via llama.cpp or Ollama, which handles Apple Silicon well.
Does it affect long-context behaviour? Quantising the KV cache can degrade recall over very long contexts more than it affects short-prompt quality.
Recap in one screen
Quantisation stores weights in fewer bits: 4-bit is a quarter the memory of 16-bit.
Generation is memory-bandwidth-bound, so smaller weights are also faster.
Outliers break naive per-tensor quantisation; group-wise scales, GPTQ, AWQ and NF4 all address them.
8-bit is nearly free in quality; 4-bit costs a little, mostly on reasoning tasks.
For a fixed memory budget, a larger quantised model usually beats a smaller full-precision one.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Core Idea: A Coarser Ruler”?
A 32-bit float can express billions of distinct values. An 8-bit integer can express 256, and a 4-bit integer just 16. Quantization finds the range your weights actually occupy and divides it into that many evenly spaced levels:
What does this module say about “Symmetric vs Asymmetric”?
Toggle between them with the outlier enabled and watch the step size and RMS error change.
What does this module say about “Outliers Are the Real Enemy”?
Quantization error does not grow smoothly with bit-width — it is dominated by range . A single weight far from the others stretches the min-max span, and since the step size is that span divided by a fixed number of levels, every other weight gets a coarser grid.
Cheat sheet
Quantization in LLMs
Shrink a model by storing its weights in fewer bits. Watch a real weight distribution snap onto a coarse grid, measure the error you introduce, and see how many gigabytes you save.
GEN AI · vizlearn.in/gen_ai/quantization_in_llms.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.