Cross-Entropy and KL Divergence

The three quantities

H(p)    = − Σ pᵢ log pᵢ

H(p, q) = − Σ pᵢ log qᵢ

KL(p ‖ q) = Σ pᵢ log(pᵢ / qᵢ) = H(p, q) − H(p)

Read them as costs. Entropy is the unavoidable cost of the randomness in p. Cross-entropy is what you actually pay using q. KL is the difference: pure waste, caused by believing the wrong thing.

Because H(p) does not depend on q at all, minimising cross-entropy over your model and minimising KL are the same optimisation. That is the reason the loss is written as cross-entropy even though the quantity anyone cares about is the divergence.

Measuring how wrong a predicted distribution is

Cross-entropy compares two probability distributions: the true one, p, and the model's, q.

H(p, q) = −Σ p(x) log q(x)

For classification the true distribution is one-hot — all the probability on the correct class — so the sum collapses to a single term:

loss = −log(probability the model gave the correct class)

That is the whole loss function, and its behaviour is easy to read off:

The penalty grows without bound as the prediction approaches zero. A model that is confidently wrong is punished far harder than one that is merely uncertain, which is exactly the incentive you want when the output is meant to be a probability.

Compare with squared error, where being wrong by 1.0 caps the penalty at 1.0. That is why cross-entropy trains classifiers faster: its gradient stays large where the model is badly wrong, while squared error's gradient shrinks precisely there.

Entropy, cross-entropy and KL, in one relationship

Three quantities that are constantly confused, and one equation that separates them:

H(p, q) = H(p) + D_KL(p ‖ q)

• H(p), entropy: the irreducible uncertainty in the true distribution. You cannot do better than this.
• D_KL(p‖q), KL divergence: the extra cost of using q when the truth is p. Zero when they match, positive otherwise.
• H(p, q), cross-entropy: the total cost, which is the sum of the two.

Since H(p) does not depend on the model, minimising cross-entropy is exactly minimising KL divergence. Frameworks optimise the first because it is simpler to compute; the second is what it means.

The coding interpretation makes it concrete. Entropy is the average number of bits needed to encode outcomes using an optimal code for p. Cross-entropy is the average when you use a code designed for q instead. KL divergence is the waste.

KL divergence is not a distance

D_KL(p ‖ q) ≠ D_KL(q ‖ p)

The asymmetry is not a technicality — it changes the behaviour of models built on it.

Forward KL, D(p‖q), is heavily penalised wherever p has mass and q has almost none. Minimising it makes q spread out to cover everything the truth does — mode-covering, at the cost of putting probability in places nothing happens.

Reverse KL, D(q‖p), punishes q for putting mass where p has none. Minimising it makes q concentrate on one region and ignore the others — mode-seeking.

Variational autoencoders minimise reverse KL, which is part of why their samples can be blurry and why they sometimes ignore modes. Maximum likelihood training is forward KL, which is why language models trained this way assign small but non-zero probability to almost everything.

When a symmetric measure is genuinely needed, Jensen-Shannon divergence — the average of both directions against their mixture — is the standard choice, and its square root is a true metric.

The loss your classifier actually pays

Cross-entropy is entropy plus the penalty for using the wrong distribution. This separates the two, and shows why KL is not symmetric.

import numpy as np

def H(p):
    p = np.asarray(p, float); p = p[p > 0]
    return -(p * np.log2(p)).sum()

def cross_H(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    return -(p * np.log2(q)).sum()

def KL(p, q):
    return cross_H(p, q) - H(p)

p = np.array([0.7, 0.2, 0.1])          # the truth
for label, q in (("q = p (perfect)",  [0.70, 0.20, 0.10]),
                 ("q close",          [0.60, 0.25, 0.15]),
                 ("q wrong",          [0.10, 0.20, 0.70])):
    print("%-16s H(p)=%.4f  H(p,q)=%.4f  KL=%.4f"
          % (label, H(p), cross_H(p, q), KL(p, q)))
print()
print("cross-entropy = entropy + KL. the entropy part is fixed by the data,")
print("so minimising cross-entropy is minimising KL.")
print()
a, b = [0.9, 0.1], [0.5, 0.5]
print("KL is not symmetric:")
print("  KL(a||b) = %.4f" % KL(a, b))
print("  KL(b||a) = %.4f" % KL(b, a))
print()
print("one training example, in nats, the way a framework reports it:")
for true_class_prob in (0.99, 0.9, 0.5, 0.1, 0.01):
    print("  model gave the right class %.2f -> loss %.4f"
          % (true_class_prob, -np.log(true_class_prob)))

Things to try

1. Read the default. p is 60/20/15/5 with an entropy of 1.533 bits. q is 40/30/20/10, so the cross-entropy is 1.655 and 0.122 bits per symbol are being wasted.
2. Close the gap. Press Set q Equal To p. Cross-entropy drops to exactly the entropy, KL reads 0.000, and no arrangement of q can do better. That is Gibbs' inequality: the divergence is zero only when the two distributions agree everywhere.
3. Try to beat it. Nudge any one q weight away from p. Cross-entropy goes up, never down — whichever direction you move.
4. Watch the asymmetry. KL(p ‖ q) and KL(q ‖ p) are different numbers for the same pair. Swapping the roles is not a relabelling; it is a different question.
5. Meet infinity. Drag Weight On D to 0 while p still gives D a 5% chance. The divergence becomes infinite: your model has declared impossible something that genuinely happens, and no amount of the rest being right can rescue that.
6. Use a hard label. Set the true distribution to a hard label. H(p) is 0, so cross-entropy and KL are the same number, and both collapse to −log q(A) — the negative log likelihood of the correct class, which is precisely what a classifier's loss computes.

Why classifiers use it

For a single labelled example, p is one-hot: probability 1 on the true class and 0 elsewhere. Every term of the cross-entropy sum vanishes except one, leaving

loss = −log q(correct class)

which is the negative log likelihood. Minimising cross-entropy over a dataset is therefore maximum likelihood estimation, arrived at from information theory rather than statistics — two different stories about the same formula.

The shape of −log q is what gives the loss its bite: being 99% sure and right costs almost nothing, while being 99% sure and wrong costs a great deal. See Softmax and Cross-Entropy for that curve in a network.

Which direction, and why it matters

KL(p ‖ q) — the "forward" direction used in supervised training — punishes assigning low probability to things that actually happen. A q that is too narrow gets an enormous penalty, so the fitted q tends to spread out and cover all of p's mass. It is mean-seeking.

KL(q ‖ p) — the "reverse" direction, which shows up in variational inference — punishes putting mass where p has none, so the fitted q tends to hide inside a single mode. It is mode-seeking. Same pair of distributions, opposite behaviour, and the choice is a modelling decision rather than a convention.

What trips people up

• A zero in q. One log 0 makes the whole loss infinite, and in floating point it makes it NaN. This is why implementations clamp probabilities and why you should use the framework's fused softmax-cross-entropy rather than composing your own.
• Confusing it with a distance. KL is not symmetric and does not satisfy the triangle inequality. If you need a true metric, the Jensen-Shannon divergence is the symmetric relative built out of it.
• Mixing bits and nats. log₂ gives bits, natural log gives nats, and the ratio is 1.4427. Frameworks report nats; this page reports bits.
• Trusting perfectly confident labels. A one-hot target asks the model to drive its output to exactly 1, which it can only approach by inflating logits without limit. Label smoothing exists to stop that.

Where that leaves you

Cross-entropy H(p, q) is what your beliefs cost when reality is p, entropy H(p) is the part of that cost no model could avoid, and the KL divergence is the difference — the waste that is genuinely your model's fault. KL is never negative and is zero only when q matches p exactly, so minimising cross-entropy and minimising divergence are the same job. With a one-hot label it collapses to −log of the probability given to the right class, which is why classification loss, negative log likelihood and maximum likelihood are three names for one thing. It is not symmetric, and a single zero in q where p is non-zero sends it to infinity.

Using it correctly in code

Two implementation details cause most of the problems in practice.

Pass logits, not probabilities. Computing a softmax yourself and then taking a log is numerically fragile: large logits overflow the exponential, small probabilities underflow before the log sees them. Frameworks provide fused, stable versions:

import torch.nn as nn

loss = nn.CrossEntropyLoss()        # expects raw logits, applies log-softmax
loss = nn.BCEWithLogitsLoss()       # binary version, also expects logits

nn.CrossEntropyLoss combines log_softmax and negative log-likelihood in one numerically stable step. Using nn.NLLLoss after a manual softmax is the classic double-softmax bug: the model trains slowly and never quite works, with no error message.

Guard the log. If you must write it yourself, add a small epsilon: -log(p + 1e-12). Without it, a single prediction of exactly zero produces infinity and then NaN for the whole batch.

Two variants worth knowing: binary cross-entropy for two-class and multi-label problems (one sigmoid per label, not a softmax), and categorical cross-entropy for mutually exclusive classes. Choosing softmax for a multi-label problem is a common modelling error — softmax forces the outputs to sum to 1, which is wrong when several labels can be true at once.

Where KL divergence turns up

• Variational autoencoders add a KL term pulling the learned latent distribution towards a standard normal, which is what makes the latent space smooth enough to sample from.
• Knowledge distillation minimises the KL between a large teacher's output distribution and a small student's, which transfers far more information per example than hard labels.
• Reinforcement learning from human feedback uses a KL penalty to keep a fine-tuned policy close to the original model, preventing it from drifting into degenerate text that games the reward.
• Drift detection compares the distribution of production inputs against the training distribution; a rising KL or PSI is the alarm.
• Bayesian inference approximates an intractable posterior by minimising KL to a tractable family — the "variational" in variational inference.

Questions people ask

Why not use accuracy as the loss? Accuracy is a step function of the predictions, so its gradient is zero almost everywhere. There is nothing for gradient descent to follow. Train on cross-entropy, evaluate on accuracy.

Can cross-entropy be zero? Only if the model puts probability 1 on the correct class. In practice it approaches but never reaches zero.

What if my loss is NaN? Almost always a log(0), an overflow in the exponential, or a learning rate high enough to send weights to infinity. Use the fused loss functions and check the learning rate.

Why is my loss around 2.3 at the start? Because ln(10) = 2.30 — a model predicting uniformly across 10 classes. Seeing exactly that at initialisation is a healthy sign; not seeing it suggests a bug.

Is KL divergence ever negative? No, never — it is zero only when the distributions are identical.

What is label smoothing doing to this? It replaces the one-hot target with a slightly softened distribution, so the model is no longer pushed towards infinite confidence. It usually improves calibration at a small cost in training accuracy.

Recap in one screen

• Cross-entropy is -log(probability given to the correct class), and it punishes confident mistakes without limit.
• Cross-entropy = entropy + KL divergence, so minimising one minimises the other.
• KL is asymmetric: forward KL covers all the modes, reverse KL concentrates on one.
• Always pass logits to the framework's fused loss; never hand-roll softmax followed by log.
• KL powers VAEs, distillation, RLHF penalties, drift detection and variational inference.