Modules / Gen AI / Distillation Lab

Knowledge Distillation in LLMs

Generative AI & LLMsBy Ashish JangraUpdated 30 September 2026

Train a small student to imitate a large teacher. Raise the temperature to expose the dark knowledge hidden in the teacher's wrong answers — the signal a plain label can never carry.

1. Training Example

true label:

2. Distillation Controls

Temperature (T) 1.0

1 = raw10 = very soft

Loss mix (α) 0.5

hard labels onlyteacher only

Training Signal

KL(teacher‖student)0

Hard-label CE0

Teacher entropy0

Steps trained0

Teacher — softened at T

LARGE MODEL

Hard label (one-hot)

WHAT A NORMAL DATASET GIVES YOU

Student — current beliefs

SMALL MODEL

Overview

Train a small student to imitate a large teacher.

In this guide

Dark Knowledge: The Central Idea
Why Temperature Is Key
The Combined Loss
Why It Matters for LLMs
Training a small model to imitate a large one
Temperature, and why it is raised
Distilling language models in detail
The dark knowledge, and what temperature exposes
Guided tour
In one line
Where it fits among the compression methods
Real notes
When to distil
Questions people ask
Recap in one screen

Knowledge Distillation: Teaching a Small Model to Think Big

A large model is accurate but expensive. A small model is cheap but weaker. Knowledge distillation narrows that gap by training the small student to imitate the large teacher — and remarkably, the student often beats an identical model trained on the original labels.

Dark Knowledge: The Central Idea

A training label says "this is a dog" and nothing else. But a trained teacher, shown that same photo, might output dog 0.90, wolf 0.07, cat 0.02, car 0.001.

Those small counts are not noise — they encode likeness structure the label cannot express. A dog resembles a wolf far more than a car. Geoffrey Hinton named this dark knowledge. It is the teacher's whole learned view of how the classes relate. And it is free supervision on each single case. The student trains on two losses at once. One against the hard labels (the standard cross-entropy) and one against the teacher's soft predictions (the KL divergence between the student's and teacher's output distributions). The total loss is a weighted combination: L = alpha * L_hard + (1 - alpha) * T^2 * L_soft. Where T is the temperature and alpha controls the balance. Temperature softens the teacher's output distribution: at T=1 the distribution is peaked (the top class rules), at T=5 the small chances are amplified and the dark knowledge turns visible. The T-squared scaling compensates for the fact that more temperature reduces the size of the gradients. In practice, T between 2 and 20 and alpha between 0.1 and 0.5 are common. For LLMs, distillation much happens at the sequence level. The student learns to match the teacher's token-level chances across a full generation, not just a single classification. DistilBERT (66M parameters) distilled from BERT-base (110M) retains 97% of BERT's speed at 60% of the size and 2x the speed. This page trains a student from a teacher, shows the soft label distribution at else temperatures, and covers the alpha/T tradeoff.

Why Temperature Is Key

There is a catch: a confident teacher's non-target chances are so tiny that they add almost nothing to the gradient. At T = 1 the distribution is nearly one-hot. So the student learns roughly what the plain label now told it.

Raising the temperature in the softmax flattens the distribution and amplifies the ratios between the small vals:

pᵢ = exp(zᵢ / T) / Σ⫺ exp(z⫺ / T)

Drag T from 1 to 6 and watch the runner-up bars rise out of nothing. That is the dark knowledge turning visible — and turning trainable. Both teacher and student are softened by the same T. And the distillation loss is scaled by T² to keep gradient sizes comparable to the hard-label term.

The Combined Loss

In practice the student learns from both sources at once:

L = (1 − α) · CE(student, true label) + α · T² · KL(teacher ‖ student)

The cross-entropy term keeps the student anchored to ground truth. So it cannot inherit the teacher's mistakes wholesale.
The KL divergence term pulls the student's whole distribution toward the teacher's, transferring the likeness structure.
α balances them. Typical vals lean toward the teacher, since the soft targets carry strictly more data per case.

Why It Matters for LLMs

Distillation is how most small live models are made. DistilBERT keeps roughly 97% of BERT's speed at 40% of the size and 60% faster inference. The same recipe gave the compact members of most modern model families.

For generative models the idea extends naturally: the teacher's full next-token distribution at each spot is a far richer target than the single token that happened to appear in the text. A tied variant, sequence-level distillation, simply trains the student on text the teacher made.

Training a small model to imitate a large one

Distillation trains a small student model to reproduce the behaviour of a large teacher. The result is a model with a fraction of the parameters that retains much of the teacher's grade on the tasks it was distilled for.

The insight is about what the teacher gives. A hard label says "this is a cat". A teacher's full output distribution says "90% cat, 7% dog, 2% fox, 1% all of it else". And that shape carries data about how the classes relate. This a one-hot label does not.

Those richer targets are called soft labels, and they are why a distilled student much beats the same design trained from scratch on the same data. Each case teaches more.

	Hard labels	Soft labels from a teacher
Information per example	One class name	A full distribution
Encodes class similarity	No	Yes
Student quality	Lower	Higher
Requires a teacher	No	Yes

Temperature, and why it is raised

A confident teacher outputs a thing like [0.99, 0.005, 0.005], which carries almost as little data as a hard label. Dividing the logits by a temperature above 1 before the softmax flattens the distribution and reveals the structure below:

Temperature	Distribution
1	0.99, 0.005, 0.005
3	0.79, 0.11, 0.10
5	0.62, 0.20, 0.18

At T = 3 the relative ranking of the non-target classes turns visible. And that is the sign the student learns from.

The training loss is much a weighted mixture:

loss = α · KL(student_T ‖ teacher_T) + (1 − α) · CE(student, hard label)

The first term matches the teacher's soft distribution; the second keeps the student anchored to the ground truth. Typical settings are T between 2 and 5 and α around 0.5–0.9.

Both models must use the same temperature for the KL term. And the student's inference-time temperature returns to 1.

Distilling language models in detail

For generative models, three approaches are used, and they differ in what the student sees.

Logit distillation. Match the teacher's full next-token distribution at each spot. The most informative, and it needs access to the teacher's logits and a shared tokeniser.

Sequence-level distillation. Have the teacher generate outputs, then train the student on them as ordinary text. Works with any teacher — with one behind an API — and discards the distribution data.

Rationale distillation. Have the teacher give reasoning chains and answers, and train the student on both. This transfers some multi-step reasoning that plain answer-matching does not. And it is what "step-by-step distillation" means.

DistilBERT is the well-known encoder case: 40% less, 60% faster, retaining roughly 97% of BERT's speed on GLUE. It used logit distillation plus a hidden-state matching term and an initialisation taken from each else layer of the teacher.

The dark knowledge, and what temperature exposes

Distillation trains a small model on a large one's full output distribution rather than on the correct answer alone. The extra data is in the wrong answers. And the temperature is what makes it readable. Both of which are visible in the counts.

example_01.pyNumPy

import numpy as np

CLASSES = ["cat", "dog", "wolf", "car", "spoon"]
# what a well-trained teacher outputs for one image of a cat
LOGITS = np.array([6.0, 3.2, 2.8, -1.0, -2.5])
HARD = np.array([1.0, 0.0, 0.0, 0.0, 0.0])      # the one-hot label

def softmax(z, T=1.0):
    z = np.asarray(z, float) / T
    z = z - z.max()
    e = np.exp(z)
    return e / e.sum()

print("A TEACHER'S OUTPUT on one image of a cat, at T=1:")
p = softmax(LOGITS)
print("%-10s %10s %14s" % ("class", "logit", "probability"))
for c, l, q in zip(CLASSES, LOGITS, p):
    print("%-10s %10.2f %14.6f" % (c, l, q))
print()
print("THE HARD LABEL says only 'cat'. compare what each target contains:")
print("%-10s %14s %16s" % ("class", "hard label", "teacher at T=1"))
for c, h, q in zip(CLASSES, HARD, p):
    print("%-10s %14.1f %16.6f" % (c, h, q))
print("   the teacher's output says the image is a cat AND that it is far")
print("   more dog-like than car-like: P(dog) is %.0fx P(car)."
      % (p[1] / p[3]))
print("   that ranking among the WRONG answers is the dark knowledge. it")
print("   is a statement about the shape of the problem -- cats resemble")
print("   dogs, and neither resembles cutlery -- and the one-hot label")
print("   contains none of it.")
print()

Output

Building on the same setup, the next block shows the left cases.

example_02.pyNumPy

import numpy as np
CLASSES = ["cat", "dog", "wolf", "car", "spoon"]

print("BUT AT T=1 IT IS ALMOST INVISIBLE. the small probabilities are")
print("crushed against zero, so a loss that averages over them barely")
print("notices they exist:")
print("%-10s %s" % ("class", "  ".join("%12s" % ("T=%.0f" % t)
                                       for t in (1, 2, 4, 8))))
for i, c in enumerate(CLASSES):
    print("%-10s %s" % (c, "  ".join("%12.6f" % softmax(LOGITS, t)[i]
                                     for t in (1, 2, 4, 8))))
print("%-10s %s" % ("P(dog)/P(car)",
                    "  ".join("%12.1f" % (softmax(LOGITS, t)[1]
                                          / softmax(LOGITS, t)[3])
                              for t in (1, 2, 4, 8))))
print("   raising T flattens the distribution and lifts the tail into a")
print("   range a gradient can act on. at T=1 the car class carries")
print("   %.2e of the mass; at T=4 it carries %.4f."
      % (softmax(LOGITS, 1)[3], softmax(LOGITS, 4)[3]))
print("   the RATIOS do change, and sharply: P(dog)/P(car) falls from")
print("   %.0f to %.1f across that sweep. what is preserved is the ORDER."
      % (softmax(LOGITS, 1)[1] / softmax(LOGITS, 1)[3],
         softmax(LOGITS, 8)[1] / softmax(LOGITS, 8)[3]))
print("   temperature compresses the differences without ever reordering")
print("   them, which is exactly the trade being made: you give up some")
print("   of the teacher's confidence about how MUCH more dog-like the")
print("   image is, in exchange for the tail being large enough to learn")
print("   from at all.")
print()

Output

The below block covers the left steps.

example_03.pyNumPy

import numpy as np
CLASSES = ["cat", "dog", "wolf", "car", "spoon"]

print("THE PRICE OF RAISING T, and it is the reason for the 1/T^2 factor")
print("nobody explains. the gradient of the soft loss shrinks as T grows:")
print("%-10s %20s %22s" % ("T", "gradient scale ~1/T^2", "after correction"))
for T in (1, 2, 4, 8):
    print("%-10d %20.6f %22.4f" % (T, 1.0 / T ** 2, (1.0 / T ** 2) * T ** 2))
print("   at T=4 the soft-target gradients are %.0fx smaller than the hard"
      % (4 ** 2))
print("   ones, so without a correction the soft term would contribute")
print("   almost nothing to training -- exactly the opposite of what you")
print("   raised T for.")
print("   multiplying the soft loss by T^2 restores it, which is why every")
print("   distillation implementation has that factor and why it looks")
print("   arbitrary until you differentiate the softmax.")
print()

print("THE COMBINED LOSS is a weighted sum of the two targets:")
print("      L = alpha * KL(student, teacher at T) * T^2")
print("        + (1 - alpha) * CE(student, the hard label)")
print("%-14s %-34s %s" % ("alpha", "what the student learns from", "risk"))
for a, learns, risk in ((0.0, "the labels only", "no distillation at all"),
                        (0.5, "both, equally", "the usual starting point"),
                        (0.9, "mostly the teacher", "inherits its mistakes"),
                        (1.0, "the teacher only", "cannot beat the teacher")):
    print("%-14.1f %-34s %s" % (a, learns, risk))
print("   the hard-label term is what stops the student inheriting the")

Output

The next block goes on with the left demonstrations.

example_04.pyNumPy

print("   teacher's errors wholesale. keeping it is why a distilled model")
print("   can occasionally be RIGHT where its teacher was wrong.")
print()

print("WHY THIS WORKS BETTER THAN TRAINING SMALL FROM SCRATCH. count the")
print("information in one training example:")
print("%-34s %18s %s" % ("target", "numbers per example", "what they say"))
print("%-34s %18d %s" % ("one-hot label", 1, "the answer"))
print("%-34s %18d %s"
      % ("teacher distribution", len(CLASSES), "the answer and its neighbours"))
print("   for a language model the second row is the vocabulary size --")
print("   %s numbers per token instead of 1." % "{:,}".format(32000))
print("   that is a much richer signal per example, which is why a")
print("   distilled model reaches a given quality on far less data than")
print("   the same architecture trained from scratch.")
print()
print("WHERE IT SITS AMONG THE COMPRESSION METHODS:")
print("%-24s %-30s %s" % ("method", "what it changes", "needs retraining?"))
for row in (("quantisation", "the precision of weights", "no"),
            ("pruning", "which weights exist", "usually"),
            ("distillation", "the whole architecture", "yes, fully"),
            ("LoRA", "nothing -- it adds", "a small amount")):
    print("%-24s %-30s %s" % row)
print("   distillation is the most expensive of the four and the only one")
print("   that can produce a genuinely smaller ARCHITECTURE rather than a")
print("   cheaper copy of the same one. that is the trade: a training run")
print("   against a model that is small by design instead of small by")
print("   compression.")

Output

Guided tour

Look at T = 1. The teacher is nearly one-hot and looks almost the same to the hard label. Barely any extra sign.
Raise T to 5. The runner-ups emerge. Note which ones rise: semantically tied classes, never random ones. That sorting is the knowledge being transferred.
Press "Train student" repeatedly and watch KL divergence fall as the student's bars converge on the teacher's shape. With the small ones.
Set α = 0 and retrain from reset. The student matches just the one-hot label; its non-target chances collapse and the likeness structure is lost.
Try the sentiment case. "not bad at all" is truly ambiguous. So the teacher stays unclear still at T = 1. And that calibrated doubt is it worth teaching.

In one line

A hard label answers one question. A teacher's full distribution answers each question at once. How likely each else is, and so how the classes relate. Temperature is the tool that makes that hidden structure large enough to learn from. The student in this lab performs real gradient descent on the combined loss. So the convergence you watch is genuine optimisation, just at a much less scale.

Where it fits among the compression methods

Method	What it changes	Quality cost
Distillation	Trains a smaller architecture	Moderate, task-dependent
Quantisation	Stores the same weights in fewer bits	Small
Pruning	Removes weights or heads	Small to moderate
Architecture search	Finds an efficient design	Varies

These combine, and the usual live sequence is distil. Then quantise: reduce the parameter count first. Then reduce the bits per parameter. A distilled 1B model quantised to 4-bit runs comfortably on a phone.

The check worth internalising: quantisation is nearly free and gives you a fixed factor of memory. Distillation costs a training run and can give you a model an order of size less. But just for the behaviour you distilled.

That caveat matters. A student distilled on customer-support conversations will handle customer support well and may be markedly worse at a thing else. Distillation narrows as it shrinks, which is much just what a live task wants.

Real notes

Generate teacher outputs once, offline. Running the teacher in each training step is costly; precompute and cache its outputs or logits.

Match the tokeniser for logit distillation. Else vocabularies mean the distributions are over else things and cannot be compared spot by spot.

Initialise the student from the teacher where design allows. Taking each else layer, for instance. It converges much faster than random initialisation.

Distil on the distribution you will serve. The student learns the teacher's behaviour on the prompts you show it. So those prompts should resemble live traffic.

Watch the licence. With a commercial model's outputs to train a competing model is prohibited by a few providers' terms. This is a legal constraint rather than a technical one. And it is a real one.

When to distil

Reach for it when: inference cost or latency is the binding constraint, the task is narrow and well-set, you have or can generate plenty of representative inputs. And you have a strong teacher available.

Do not, when: the task is open-ended and you need broad power, you lack representative prompts, quantising the more model would now meet the budget. Or the licence forbids it.

The honest sorting for most teams: try quantisation first, since it is far cheaper and much enough. Distil when quantisation is not enough and the task is narrow enough to survive the specialisation.

Questions people ask

How much less can the student be? Much 2–10×, with 40–60% reductions retaining most grade on the distilled task.

Do I need the teacher's logits? For logit distillation, yes. Sequence-level distillation works from made text alone, which is what makes API-based teachers usable at all.

What temperature? 2–5. More reveals more structure in confident predictions and in time flattens away the sign.

Can a student beat its teacher? On the narrow distilled task, sometimes. The soft targets regularise. And the less model overfits less. Not broadly power.

Is this the same as fine-tuning? Fine-tuning adapts a model to a task with ground-truth labels. Distillation trains one model to imitate one more. They are much combined.

How much data do I need? More than for fine-tuning, since the student is learning behaviour rather than tuning it. Tens of thousands of cases is typical, and they can be made by the teacher.

Recap in one screen

A small student is trained to reproduce a large teacher's output distribution, not just its answers.
Soft labels carry data about how classes relate, so each case teaches more than a hard label.
Raise the temperature to reveal structure in a confident teacher's distribution.
Logit distillation needs the teacher's internals; sequence-level distillation works from made text.
Distil then quantise; and expect the student to be narrower and less.

Recall check

0 of 3

Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.

Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Dark Knowledge: The Central Idea”?
What does this module say about “Why Temperature Is Key”?

Cheat sheet