Train a small student to imitate a large teacher. Raise the temperature to expose the dark knowledge hidden in the teacher's wrong answers — the signal a plain label can never carry.
Overview
Dark Knowledge: The Central Idea
A training label says "this is a dog" and nothing else. But a trained teacher, shown that same photo, might output dog 0.90, wolf 0.07, cat 0.02, car 0.001.
Those small numbers are not noise — they encode similarity structure the label cannot express: a dog resembles a wolf far more than a car. Geoffrey Hinton named this dark knowledge. It is the teacher's entire learned view of how the classes relate, and it is free supervision on every single example.
Teacher — softened at T
LARGE MODEL
Hard label (one-hot)
WHAT A NORMAL DATASET GIVES YOU
Student — current beliefs
SMALL MODEL
Knowledge Distillation: Teaching a Small Model to Think Big
A large model is accurate but expensive. A small model is cheap but weaker. Knowledge distillation narrows that gap by training the small student to imitate the large teacher — and remarkably, the student often beats an identical model trained on the original labels.
Why Temperature Is Essential
There is a catch: a confident teacher's non-target probabilities are so tiny that they contribute almost nothing to the gradient. At T = 1 the distribution is nearly one-hot, so the student learns roughly what the plain label already told it.
Raising the temperature in the softmax flattens the distribution and amplifies the ratios between the small values:
pᵢ = exp(zᵢ / T) / Σ⫺ exp(z⫺ / T)
Drag T from 1 to 6 and watch the runner-up bars rise out of nothing. That is the dark knowledge becoming visible — and becoming trainable. Both teacher and student are softened by the same T, and the distillation loss is scaled by T² to keep gradient magnitudes comparable to the hard-label term.
The Combined Loss
In practice the student learns from both sources at once:
The cross-entropy term keeps the student anchored to ground truth, so it cannot inherit the teacher's mistakes wholesale.
The KL divergence term pulls the student's whole distribution toward the teacher's, transferring the similarity structure.
α balances them. Typical values lean toward the teacher, since the soft targets carry strictly more information per example.
Why It Matters for LLMs
Distillation is how most small production models are made. DistilBERT keeps roughly 97% of BERT's performance at 40% of the size and 60% faster inference. The same recipe produced the compact members of most modern model families.
For generative models the idea extends naturally: the teacher's full next-token distribution at every position is a far richer target than the single token that happened to appear in the text. A related variant, sequence-level distillation, simply trains the student on text the teacher generated.
Training a small model to imitate a large one
Distillation trains a small student model to reproduce the behaviour of a large teacher. The result is a model with a fraction of the parameters that retains much of the teacher's quality on the tasks it was distilled for.
The insight is about what the teacher provides. A hard label says "this is a cat". A teacher's full output distribution says "90% cat, 7% dog, 2% fox, 1% everything else" — and that shape carries information about how the classes relate, which a one-hot label does not.
Those richer targets are called soft labels, and they are why a distilled student often outperforms the same architecture trained from scratch on the same data. Each example teaches more.
Hard labels
Soft labels from a teacher
Information per example
One class name
A full distribution
Encodes class similarity
No
Yes
Student quality
Lower
Higher
Requires a teacher
No
Yes
Temperature, and why it is raised
A confident teacher outputs something like [0.99, 0.005, 0.005], which carries almost as little information as a hard label. Dividing the logits by a temperature above 1 before the softmax flattens the distribution and reveals the structure underneath:
Temperature
Distribution
1
0.99, 0.005, 0.005
3
0.79, 0.11, 0.10
5
0.62, 0.20, 0.18
At T = 3 the relative ranking of the non-target classes becomes visible, and that is the signal the student learns from.
The training loss is usually a weighted mixture:
loss = α · KL(studentT ‖ teacherT) + (1 − α) · CE(student, hard label)
The first term matches the teacher's soft distribution; the second keeps the student anchored to the ground truth. Typical settings are T between 2 and 5 and α around 0.5–0.9.
Both models must use the same temperature for the KL term, and the student's inference-time temperature returns to 1.
Distilling language models specifically
For generative models, three approaches are used, and they differ in what the student sees.
Logit distillation. Match the teacher's full next-token distribution at every position. The most informative, and it requires access to the teacher's logits and a shared tokeniser.
Sequence-level distillation. Have the teacher generate outputs, then train the student on them as ordinary text. Works with any teacher — including one behind an API — and discards the distribution information.
Rationale distillation. Have the teacher produce reasoning chains as well as answers, and train the student on both. This transfers some multi-step reasoning that plain answer-matching does not, and it is what "step-by-step distillation" refers to.
DistilBERT is the well-known encoder example: 40% smaller, 60% faster, retaining roughly 97% of BERT's performance on GLUE. It used logit distillation plus a hidden-state matching term and an initialisation taken from every other layer of the teacher.
The dark knowledge, and what temperature exposes
Distillation trains a small model on a large one's full output distribution rather than on the correct answer alone. The extra information is in the wrong answers, and the temperature is what makes it readable -- both of which are visible in the numbers.
example_01.pyNumPy
import numpy as np
CLASSES = ["cat", "dog", "wolf", "car", "spoon"]
# what a well-trained teacher outputs for one image of a cat
LOGITS = np.array([6.0, 3.2, 2.8, -1.0, -2.5])
HARD = np.array([1.0, 0.0, 0.0, 0.0, 0.0]) # the one-hot label
def softmax(z, T=1.0):
z = np.asarray(z, float) / T
z = z - z.max()
e = np.exp(z)
return e / e.sum()
print("A TEACHER'S OUTPUT on one image of a cat, at T=1:")
p = softmax(LOGITS)
print("%-10s %10s %14s" % ("class", "logit", "probability"))
for c, l, q in zip(CLASSES, LOGITS, p):
print("%-10s %10.2f %14.6f" % (c, l, q))
print()
print("THE HARD LABEL says only 'cat'. compare what each target contains:")
print("%-10s %14s %16s" % ("class", "hard label", "teacher at T=1"))
for c, h, q in zip(CLASSES, HARD, p):
print("%-10s %14.1f %16.6f" % (c, h, q))
print(" the teacher's output says the image is a cat AND that it is far")
print(" more dog-like than car-like: P(dog) is %.0fx P(car)."
% (p[1] / p[3]))
print(" that ranking among the WRONG answers is the dark knowledge. it")
print(" is a statement about the shape of the problem -- cats resemble")
print(" dogs, and neither resembles cutlery -- and the one-hot label")
print(" contains none of it.")
print()
print("BUT AT T=1 IT IS ALMOST INVISIBLE. the small probabilities are")
print("crushed against zero, so a loss that averages over them barely")
print("notices they exist:")
print("%-10s %s" % ("class", " ".join("%12s" % ("T=%.0f" % t)
for t in (1, 2, 4, 8))))
for i, c in enumerate(CLASSES):
print("%-10s %s" % (c, " ".join("%12.6f" % softmax(LOGITS, t)[i]
for t in (1, 2, 4, 8))))
print("%-10s %s" % ("P(dog)/P(car)",
" ".join("%12.1f" % (softmax(LOGITS, t)[1]
/ softmax(LOGITS, t)[3])
for t in (1, 2, 4, 8))))
print(" raising T flattens the distribution and lifts the tail into a")
print(" range a gradient can act on. at T=1 the car class carries")
print(" %.2e of the mass; at T=4 it carries %.4f."
% (softmax(LOGITS, 1)[3], softmax(LOGITS, 4)[3]))
print(" the RATIOS do change, and sharply: P(dog)/P(car) falls from")
print(" %.0f to %.1f across that sweep. what is preserved is the ORDER."
% (softmax(LOGITS, 1)[1] / softmax(LOGITS, 1)[3],
softmax(LOGITS, 8)[1] / softmax(LOGITS, 8)[3]))
print(" temperature compresses the differences without ever reordering")
print(" them, which is exactly the trade being made: you give up some")
print(" of the teacher's confidence about how MUCH more dog-like the")
print(" image is, in exchange for the tail being large enough to learn")
print(" from at all.")
print()
print("THE PRICE OF RAISING T, and it is the reason for the 1/T^2 factor")
print("nobody explains. the gradient of the soft loss shrinks as T grows:")
print("%-10s %20s %22s" % ("T", "gradient scale ~1/T^2", "after correction"))
for T in (1, 2, 4, 8):
print("%-10d %20.6f %22.4f" % (T, 1.0 / T ** 2, (1.0 / T ** 2) * T ** 2))
print(" at T=4 the soft-target gradients are %.0fx smaller than the hard"
% (4 ** 2))
print(" ones, so without a correction the soft term would contribute")
print(" almost nothing to training -- exactly the opposite of what you")
print(" raised T for.")
print(" multiplying the soft loss by T^2 restores it, which is why every")
print(" distillation implementation has that factor and why it looks")
print(" arbitrary until you differentiate the softmax.")
print()
print("THE COMBINED LOSS is a weighted sum of the two targets:")
print(" L = alpha * KL(student, teacher at T) * T^2")
print(" + (1 - alpha) * CE(student, the hard label)")
print("%-14s %-34s %s" % ("alpha", "what the student learns from", "risk"))
for a, learns, risk in ((0.0, "the labels only", "no distillation at all"),
(0.5, "both, equally", "the usual starting point"),
(0.9, "mostly the teacher", "inherits its mistakes"),
(1.0, "the teacher only", "cannot beat the teacher")):
print("%-14.1f %-34s %s" % (a, learns, risk))
print(" the hard-label term is what stops the student inheriting the")
print(" teacher's errors wholesale. keeping it is why a distilled model")
print(" can occasionally be RIGHT where its teacher was wrong.")
print()
print("WHY THIS WORKS BETTER THAN TRAINING SMALL FROM SCRATCH. count the")
print("information in one training example:")
print("%-34s %18s %s" % ("target", "numbers per example", "what they say"))
print("%-34s %18d %s" % ("one-hot label", 1, "the answer"))
print("%-34s %18d %s"
% ("teacher distribution", len(CLASSES), "the answer and its neighbours"))
print(" for a language model the second row is the vocabulary size --")
print(" %s numbers per token instead of 1." % "{:,}".format(32000))
print(" that is a much richer signal per example, which is why a")
print(" distilled model reaches a given quality on far less data than")
print(" the same architecture trained from scratch.")
print()
print("WHERE IT SITS AMONG THE COMPRESSION METHODS:")
print("%-24s %-30s %s" % ("method", "what it changes", "needs retraining?"))
for row in (("quantisation", "the precision of weights", "no"),
("pruning", "which weights exist", "usually"),
("distillation", "the whole architecture", "yes, fully"),
("LoRA", "nothing -- it adds", "a small amount")):
print("%-24s %-30s %s" % row)
print(" distillation is the most expensive of the four and the only one")
print(" that can produce a genuinely smaller ARCHITECTURE rather than a")
print(" cheaper copy of the same one. that is the trade: a training run")
print(" against a model that is small by design instead of small by")
print(" compression.")
Output
Guided tour
Look at T = 1. The teacher is nearly one-hot and looks almost identical to the hard label — barely any extra signal.
Raise T to 5. The runner-ups emerge. Note which ones rise: semantically related classes, never random ones. That ordering is the knowledge being transferred.
Press "Train student" repeatedly and watch KL divergence fall as the student's bars converge on the teacher's shape — including the small ones.
Set α = 0 and retrain from reset. The student matches only the one-hot label; its non-target probabilities collapse and the similarity structure is lost.
Try the sentiment example. "not bad at all" is genuinely ambiguous, so the teacher stays uncertain even at T = 1 — and that calibrated uncertainty is itself worth teaching.
In one line
A hard label answers one question. A teacher's full distribution answers every question at once — how likely each alternative is, and therefore how the classes relate. Temperature is the tool that makes that hidden structure large enough to learn from. The student in this lab performs real gradient descent on the combined loss, so the convergence you watch is genuine optimisation, just at a much smaller scale.
Where it fits among the compression methods
Method
What it changes
Quality cost
Distillation
Trains a smaller architecture
Moderate, task-dependent
Quantisation
Stores the same weights in fewer bits
Small
Pruning
Removes weights or heads
Small to moderate
Architecture search
Finds an efficient design
Varies
These combine, and the usual production sequence is distil, then quantise: reduce the parameter count first, then reduce the bits per parameter. A distilled 1B model quantised to 4-bit runs comfortably on a phone.
The comparison worth internalising: quantisation is nearly free and gives you a fixed factor of memory. Distillation costs a training run and can give you a model an order of magnitude smaller — but only for the behaviour you distilled.
That caveat matters. A student distilled on customer-support conversations will handle customer support well and may be markedly worse at anything else. Distillation narrows as it shrinks, which is often exactly what a production task wants.
Practical notes
Generate teacher outputs once, offline. Running the teacher during every training step is expensive; precompute and cache its outputs or logits.
Match the tokeniser for logit distillation. Different vocabularies mean the distributions are over different things and cannot be compared position by position.
Initialise the student from the teacher where architecture allows — taking every other layer, for instance. It converges considerably faster than random initialisation.
Distil on the distribution you will serve. The student learns the teacher's behaviour on the prompts you show it, so those prompts should resemble production traffic.
Watch the licence. Using a commercial model's outputs to train a competing model is prohibited by several providers' terms. This is a legal constraint rather than a technical one, and it is a real one.
When to distil
Reach for it when: inference cost or latency is the binding constraint, the task is narrow and well-defined, you have or can generate plenty of representative inputs, and you have a strong teacher available.
Do not, when: the task is open-ended and you need broad capability, you lack representative prompts, quantising the larger model would already meet the budget, or the licence forbids it.
The honest ordering for most teams: try quantisation first, since it is far cheaper and often sufficient. Distil when quantisation is not enough and the task is narrow enough to survive the specialisation.
Questions people ask
How much smaller can the student be? Commonly 2–10×, with 40–60% reductions retaining most quality on the distilled task.
Do I need the teacher's logits? For logit distillation, yes. Sequence-level distillation works from generated text alone, which is what makes API-based teachers usable at all.
What temperature? 2–5. Higher reveals more structure in confident predictions and eventually flattens away the signal.
Can a student beat its teacher? On the narrow distilled task, occasionally — the soft targets regularise, and the smaller model overfits less. Not in general capability.
Is this the same as fine-tuning? Fine-tuning adapts a model to a task using ground-truth labels. Distillation trains one model to imitate another. They are often combined.
How much data do I need? More than for fine-tuning, because the student is learning behaviour rather than adjusting it — tens of thousands of examples is typical, and they can be generated by the teacher.
Recap in one screen
A small student is trained to reproduce a large teacher's output distribution, not just its answers.
Soft labels carry information about how classes relate, so each example teaches more than a hard label.
Raise the temperature to reveal structure in a confident teacher's distribution.
Logit distillation needs the teacher's internals; sequence-level distillation works from generated text.
Distil then quantise; and expect the student to be narrower as well as smaller.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
A hard label answers one question. A teacher's full distribution answers every question at once — how likely each alternative is, and therefore how the classes relate. Temperature is the tool that makes that hidden structure large enough to learn from. The student in this lab performs real gradient descent on the combined loss, so the convergence you watch is genuine optimisation, just at a much smaller scale.
What does this module say about “Dark Knowledge: The Central Idea”?
A training label says "this is a dog" and nothing else. But a trained teacher, shown that same photo, might output dog 0.90, wolf 0.07, cat 0.02, car 0.001 .
What does this module say about “Why Temperature Is Essential”?
There is a catch: a confident teacher's non-target probabilities are so tiny that they contribute almost nothing to the gradient. At T = 1 the distribution is nearly one-hot, so the student learns roughly what the plain label already told it.
Cheat sheet
Knowledge Distillation in LLMs
Train a small student to imitate a large teacher. Raise the temperature to expose the dark knowledge hidden in the teacher's wrong answers — the signal a plain label can never carry.
GEN AI · vizlearn.in/gen_ai/knowledge_distillation_in_llms.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.