Cross-Entropy and KL Divergence
Reality is p. Your model says q. Cross-entropy is what your beliefs cost you per symbol, and the KL divergence is the part of that bill you could have avoided.
Reality (p)
p is fixed by the world. q is the only thing you get to change.
Your Model (q)
the weights are normalised, so only their proportions matter
Two Distributions, And The Gap
—p q each outcome's share of the divergence
The Three Numbers
It Is Not Symmetric
Swapping the two arguments gives a different answer, which is why KL is called a divergence rather than a distance.
Cross-Entropy and KL Divergence
Cross-entropy scores a predicted distribution against a target; KL divergence measures the extra coding cost from using the wrong distribution.
Classification example
For a known class with target probability 1 and predicted probability q, cross-entropy is −log(q). Predicting q = 0.9 gives a small penalty; q = 0.1 gives a larger one. In practice, stable implementations use logits and log-sum-exp calculations instead of directly taking logs of rounded probabilities. See logarithms and entropy.
Cross-entropy is not always “the same as KL”: for a fixed target P, H(P) is constant with respect to Q, so minimising cross-entropy over Q is equivalent to minimising Dₖₗ(P‖Q). The equivalence depends on P being fixed.
Work a two-class prediction by hand
Suppose the true class distribution is P=(0.8, 0.2) and a model predicts Q=(0.6, 0.4). Using base-2 logarithms, H(P)=−0.8log₂0.8−0.2log₂0.2≈0.722 bits. Cross-entropy is H(P,Q)=−0.8log₂0.6−0.2log₂0.4≈0.854 bits, so DKL(P‖Q)≈0.132 bits. The extra amount is the penalty for using Q instead of the true distribution P. If Q moved closer to P, cross-entropy and KL divergence would shrink, while H(P) would remain fixed.
For a single labelled classification example, the target distribution is often one-hot, so cross-entropy reduces to minus the log of the probability assigned to the correct class. A confident correct prediction gets a small loss; a confident wrong prediction gets a large loss. The order of KL matters: DKL(P‖Q) generally differs from DKL(Q‖P), so the notation identifies which distribution supplies the expectation. If Q gives zero probability to an outcome that P considers possible, the ideal KL is infinite. Implementations often clip or use logits for numerical stability, but those computational details should preserve the intended loss.
Try this above
Keep the target distribution fixed and move the prediction toward it. Compare cross-entropy and KL; their difference should remain the target entropy.
For a dataset with multiple labelled examples, classification cross-entropy is typically averaged across examples. If the model assigns probability qᵢ to each observed class, the loss is the average of −log qᵢ. A single confident error can contribute a large loss, which is useful when confidence matters but can make outliers influential. Class weighting or label smoothing changes the objective and therefore the meaning of the score. Compare models on the same target definition and evaluation set; a smaller training loss alone does not prove better calibrated probabilities.
Sources and next steps
- Cover, T. M. and Thomas, J. A., Elements of Information Theory, develops entropy and relative entropy.
- See information gain for entropy reduction in decision trees.
Overview
Relate the three quantities
For distributions P and Q on the same outcomes, entropy is H(P) = −Σp log p. Cross-entropy is H(P,Q) = −Σp log q. Their difference is the Kullback–Leibler divergence:
KL divergence is nonnegative but generally asymmetric, so it is not a distance metric. It is zero when P and Q agree on outcomes with positive P probability. If Q assigns zero probability where P is positive, the divergence is infinite in the ideal mathematical definition.