Shrink a model by storing its weights in fewer bits. Watch a real weight distribution snap onto a coarse grid, measure the error you introduce, and see how many gigabytes you save.
Weights Snapping to the Grid
originalquantized
Grey dots are the original FP32 weights; green dots are where they land after rounding. Horizontal lines are the representable levels.
Per-Weight Detail
Quantization: Trading Precision for Memory
A 70B-parameter model in full 32-bit precision needs about 280 GB just to hold its weights — far beyond any single GPU. Quantization stores those same weights in fewer bits, and it is the single technique most responsible for large models running on ordinary hardware.
The Core Idea: A Coarser Ruler
A 32-bit float can express billions of distinct values. An 8-bit integer can express 256, and a 4-bit integer just 16. Quantization finds the range your weights actually occupy and divides it into that many evenly spaced levels:
Every weight is then snapped to its nearest level. The difference between where a weight was and where it landed is the quantization error — unavoidable, and the entire subject in one sentence.
Symmetric vs Asymmetric
Symmetric centres the grid on zero, using a range of ±max(|w|). Simpler and faster, since zero maps exactly to zero — but if your weights are lopsided, half the levels are wasted on values that never occur.
Asymmetric fits the grid to the actual min and max, adding a zero-point offset. It uses every level, at the cost of slightly more arithmetic at inference.
Toggle between them with the outlier enabled and watch the step size and RMS error change.
Outliers Are the Real Enemy
Quantization error does not grow smoothly with bit-width — it is dominated by range. A single weight far from the others stretches the min-max span, and since the step size is that span divided by a fixed number of levels, every other weight gets a coarser grid.
Tick "Inject an outlier" and watch the step size jump while the ordinary weights suddenly land further from home. This is not a toy effect: emergent outlier features in large transformers are precisely why naive INT8 broke down, and why methods like LLM.int8(), GPTQ and AWQ exist — they isolate or protect the outliers instead of letting them ruin the scale for everyone.
Interactive Exploration Guide
Step from FP32 down to INT4. Levels fall from billions to 16, the horizontal grid lines thin out, and the green dots drift visibly away from the grey ones.
Watch RMS error against memory saved. FP16 is nearly free — huge savings, negligible error. That is why it is the default almost everywhere.
Now enable the outlier at INT4. Step size roughly doubles and every ordinary weight is punished for one extreme value.
Switch to asymmetric with the outlier on. Error usually improves, because the grid stops wasting levels on an empty side of the range.
Pick 70B and compare FP16 to INT4. 140 GB versus roughly 35 GB — the difference between a multi-GPU server and a single high-end card. That is the whole reason anyone tolerates the error.
Key Takeaway
Quantization buys memory and speed with precision. The surprise is how cheap the trade is: 8-bit is usually indistinguishable from full precision, and modern 4-bit methods come remarkably close. The size numbers here count weights only — real deployments also need memory for activations and the KV cache, which grows with context length.
Cheat sheet
Quantization in LLMs
Shrink a model by storing its weights in fewer bits. Watch a real weight distribution snap onto a coarse grid, measure the error you introduce, and see how many gigabytes you save.
GEN AI · vizlearn.in/gen_ai/quantization_in_llms.html