Cross-Entropy and KL Divergence
Reality is p. Your model says q. Cross-entropy is what your beliefs cost you per symbol, and the KL divergence is the part of that bill you could have avoided.
Reality (p)
p is fixed by the world. q is the only thing you get to change.
Your Model (q)
the weights are normalised, so only their proportions matter
Two Distributions, And The Gap
—p q each outcome's share of the divergence
The Three Numbers
It Is Not Symmetric
Swapping the two arguments gives a different answer, which is why KL is called a divergence rather than a distance.
Cross-Entropy and KL Divergence: A Practical Guide
The loss function almost every classifier is trained with, and the quantity underneath it.
Quick Context
Entropy is the average surprise of a distribution — the shortest average code length you could achieve if you knew the true probabilities. Cross-entropy asks a harsher question: what does it cost if you build your code for q and the data actually comes from p?
The answer is always at least H(p), and the excess is the KL divergence. Every classifier trained with "cross-entropy loss" is being pushed to make that excess smaller.
The three quantities
H(p) = − Σ pᵢ log pᵢ
H(p, q) = − Σ pᵢ log qᵢ
KL(p ‖ q) = Σ pᵢ log(pᵢ / qᵢ) = H(p, q) − H(p)
Read them as costs. Entropy is the unavoidable cost of the randomness in p. Cross-entropy is what you actually pay using q. KL is the difference: pure waste, caused by believing the wrong thing.
Because H(p) does not depend on q at all, minimising cross-entropy over your model and minimising KL are the same optimisation. That is the reason the loss is written as cross-entropy even though the quantity anyone cares about is the divergence.
Interactive Exploration Guide
- Read the default. p is 60/20/15/5 with an entropy of 1.533 bits. q is 40/30/20/10, so the cross-entropy is 1.655 and 0.122 bits per symbol are being wasted.
- Close the gap. Press Set q Equal To p. Cross-entropy drops to exactly the entropy, KL reads 0.000, and no arrangement of q can do better. That is Gibbs' inequality: the divergence is zero only when the two distributions agree everywhere.
- Try to beat it. Nudge any one q weight away from p. Cross-entropy goes up, never down — whichever direction you move.
- Watch the asymmetry. KL(p ‖ q) and KL(q ‖ p) are different numbers for the same pair. Swapping the roles is not a relabelling; it is a different question.
- Meet infinity. Drag Weight On D to 0 while p still gives D a 5% chance. The divergence becomes infinite: your model has declared impossible something that genuinely happens, and no amount of the rest being right can rescue that.
- Use a hard label. Set the true distribution to a hard label. H(p) is 0, so cross-entropy and KL are the same number, and both collapse to −log q(A) — the negative log likelihood of the correct class, which is precisely what a classifier's loss computes.
Why classifiers use it
For a single labelled example, p is one-hot: probability 1 on the true class and 0 elsewhere. Every term of the cross-entropy sum vanishes except one, leaving
loss = −log q(correct class)
which is the negative log likelihood. Minimising cross-entropy over a dataset is therefore maximum likelihood estimation, arrived at from information theory rather than statistics — two different stories about the same formula.
The shape of −log q is what gives the loss its bite: being 99% sure and right costs almost nothing, while being 99% sure and wrong costs a great deal. See Softmax and Cross-Entropy for that curve in a network.
Which direction, and why it matters
KL(p ‖ q) — the "forward" direction used in supervised training — punishes assigning low probability to things that actually happen. A q that is too narrow gets an enormous penalty, so the fitted q tends to spread out and cover all of p's mass. It is mean-seeking.
KL(q ‖ p) — the "reverse" direction, which shows up in variational inference — punishes putting mass where p has none, so the fitted q tends to hide inside a single mode. It is mode-seeking. Same pair of distributions, opposite behaviour, and the choice is a modelling decision rather than a convention.
What usually goes wrong
- A zero in q. One log 0 makes the whole loss infinite, and in floating point it makes it NaN. This is why implementations clamp probabilities and why you should use the framework's fused softmax-cross-entropy rather than composing your own.
- Confusing it with a distance. KL is not symmetric and does not satisfy the triangle inequality. If you need a true metric, the Jensen-Shannon divergence is the symmetric relative built out of it.
- Mixing bits and nats. log₂ gives bits, natural log gives nats, and the ratio is 1.4427. Frameworks report nats; this page reports bits.
- Trusting perfectly confident labels. A one-hot target asks the model to drive its output to exactly 1, which it can only approach by inflating logits without limit. Label smoothing exists to stop that.
Key Takeaway
Cross-entropy H(p, q) is what your beliefs cost when reality is p, entropy H(p) is the part of that cost no model could avoid, and the KL divergence is the difference — the waste that is genuinely your model's fault. KL is never negative and is zero only when q matches p exactly, so minimising cross-entropy and minimising divergence are the same job. With a one-hot label it collapses to −log of the probability given to the right class, which is why classification loss, negative log likelihood and maximum likelihood are three names for one thing. It is not symmetric, and a single zero in q where p is non-zero sends it to infinity.