Entropy and Information

Surprise first

Start with a single outcome. How surprising is it? Two things should be true: a certain outcome (p = 1) should carry zero surprise, and rarer outcomes should be more surprising. One function does this cleanly:

surprise(p) = −log₂(p)

An outcome with p = 1 gives 0 bits. p = 0.5 gives 1 bit. p = 0.25 gives 2 bits. p = 0.01 gives about 6.6 bits. Halving the probability adds exactly one bit, every time — which is the property that makes the logarithm the right choice rather than an arbitrary one.

Entropy is average surprise

Now take the whole distribution and average the surprise, weighting each outcome by how often it actually occurs:

H = − Σ p_i log₂(p_i)

That is the entire definition. The minus sign is there only because logs of numbers below 1 are negative, and we want a positive answer.

The interpretation worth memorising: entropy is the average number of yes/no questions needed to determine the outcome, if you ask the smartest possible questions. A fair coin needs exactly 1. A fair 8-sided die needs exactly 3, because 2³ = 8.

Surprise, measured in bits

Entropy measures uncertainty. Concretely, it is the average number of yes/no questions needed to identify an outcome — or equivalently, how surprised you should expect to be.

H = −Σ pᵢ log₂(pᵢ)

Three cases make the formula concrete:

• A fair coin. Two outcomes at 0.5: H = −(0.5×−1 + 0.5×−1) = 1 bit. One question, and you know.
• A biased coin (0.9 / 0.1). H = −(0.9×log₂0.9 + 0.1×log₂0.1) = 0.47 bits. You can usually guess correctly, so there is less to learn.
• A two-headed coin. One outcome at probability 1: H = 0 bits. No uncertainty, no information in the answer.

The pattern: entropy is highest when outcomes are equally likely and zero when one outcome is certain. For n equally likely outcomes it is exactly log₂(n), which is why a fair die is 2.58 bits and a fair 8-sided die is exactly 3.

Why rare events carry more information

The term inside the sum, −log₂(p), is the information content of a single outcome — how surprising it is.

"The sun rose today" carries almost no information because it was certain. "A meteor landed in the garden" carries a great deal. Entropy is the average of these surprises, weighted by how often each occurs.

The logarithm is not arbitrary: it makes information additive. Two independent events with probabilities p and q have joint probability pq, and −log(pq) = −log(p) − log(q). Learning two independent facts gives you the sum of their information, which is the property any sensible measure must have.

Where it does work in machine learning

Decision trees. Every split is chosen to reduce entropy the most. A node with 50/50 classes has 1 bit of entropy; a perfectly pure node has 0. The reduction is the information gain, and the algorithm greedily takes the largest.

Cross-entropy loss. The standard classification loss measures how many extra bits are needed to encode the true labels using the model's predicted distribution. Minimising it is making the model's beliefs match reality.

Feature selection. Mutual information — how much knowing one variable reduces uncertainty about another — ranks features without assuming a linear relationship, which correlation cannot do.

Compression. Shannon's source coding theorem says entropy is the hard lower bound on average bits per symbol. English text has roughly 1–1.5 bits of entropy per character, which is why text compresses to a fraction of its size and random bytes do not compress at all.

Language models. Perplexity is 2 raised to the cross-entropy — the effective number of equally likely choices the model is deciding between at each step. Lower is better, and it is entropy wearing a friendlier scale.

Weigh a distribution's surprise

Entropy in bits for several distributions over the same four outcomes, plus the coin curve that peaks exactly at a fair coin.

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float)
    p = p[p > 0]                       # 0 log 0 is defined as 0
    return -(p * np.log2(p)).sum() + 0.0

cases = {
    "uniform over 4":    [0.25, 0.25, 0.25, 0.25],
    "slightly skewed":   [0.40, 0.30, 0.20, 0.10],
    "very skewed":       [0.90, 0.06, 0.03, 0.01],
    "certain":           [1.00, 0.00, 0.00, 0.00],
}
for name, p in cases.items():
    print("  %-18s H = %.4f bits" % (name, abs(entropy(p))))
print()
print("4 equally likely outcomes need log2(4) = %.0f bits. certainty needs 0."
      % np.log2(4))
print()
print("a coin, as it becomes unfair:")
for q in (0.5, 0.6, 0.7, 0.9, 0.99, 0.999):
    print("  P(heads)=%-6s H = %.4f bits" % (q, entropy([q, 1 - q])))
print()
print("surprise per outcome:")
for p in (0.9, 0.5, 0.1, 0.01):
    print("  an event of probability %-5s carries %5.2f bits" % (p, -np.log2(p)))
print()
print("entropy is the average of those surprises, weighted by probability.")

Things to try

1. Find the maximum. Click Make It Maximally Uncertain. Every outcome gets equal probability and entropy hits its ceiling. Uniform is always the most uncertain a distribution can be.
2. Find the floor. Click Make It Certain. All the probability moves onto one outcome and entropy drops to exactly 0 — there is nothing left to learn, so revealing the answer tells you nothing.
3. Watch the curve. Set the Number of Outcomes slider to 2 and drag the Skew slider slowly from 0 to 1. Entropy traces the arch in the lower panel: 1 bit at an even split, falling to 0 at either extreme.
4. Count the bits. Set the Number of Outcomes slider to 8 with skew at 0. Entropy reads exactly 3.000, because eight equally likely outcomes take three yes/no questions to separate.
5. See rare events carry more. Set the Skew slider to about 0.8 and look at the Surprise panel. The rare outcomes have large surprise values but small weights, which is why entropy stays low — you are rarely surprised, even though you would be very surprised if it happened.
6. Compare against Gini. Watch the Gini Impurity readout as you drag. It rises and falls with entropy and shares the same zero point, which is why decision trees can use either and rarely care which.

Why machine learning cares

• Decision trees choose the split that reduces entropy the most. That reduction is called information gain, and it is literally "how much uncertainty did this question remove".
• Cross-entropy loss measures the average surprise of the true labels under your model's distribution. Minimising it means making the truth unsurprising to the model.
• KL divergence is the gap between the cross-entropy and the true entropy — the extra bits you pay for using the wrong distribution.
• Compression gets the same number from the other direction: entropy is the theoretical floor on the average bits per symbol, which is why a file of random bytes will not compress.

Bits, nats and why it rarely matters

Using log base 2 gives bits, which is the interpretable unit and the one this page uses. Machine learning code almost always uses the natural log instead, giving nats. The two differ by a constant factor of ln(2) ≈ 0.693, and since optimisation only cares about which direction is downhill, a constant factor changes nothing about training. It does mean a cross-entropy loss printed by a framework is in nats, so 0.693 is the "fair coin" value there rather than 1.0.

Common mistakes

• Feeding it a zero. log(0) is undefined. The convention is that 0·log(0) = 0, which is the correct limit, but a naive implementation returns NaN. Clamp probabilities away from zero.
• Reading entropy as a probability. It is measured in bits and is unbounded above as outcomes are added — eight outcomes can reach 3, sixteen can reach 4. Only the fraction-of-maximum reading is comparable across different-sized distributions.
• Confusing it with variance. Both measure spread, but entropy ignores the values entirely. Relabelling the outcomes changes variance and leaves entropy untouched.
• Expecting Gini and entropy to disagree. On classification splits they almost never pick different features. Choosing between them is not worth tuning.

Where that leaves you

Entropy is the average surprise of a distribution, −Σp·log(p), and reads as the number of yes/no questions needed to pin down the outcome. It is maximal when every outcome is equally likely and exactly zero when one outcome is certain, which is why it works as a measure of uncertainty. Every classification loss in machine learning is built from it: cross-entropy is the average surprise of the true labels under your model, and information gain — the entropy a split removes — is how a decision tree decides what to ask.

Information gain, worked through

A decision tree splitting 10 samples that are 5 yes and 5 no:

• Before: H = 1.0 bit (maximum uncertainty).
• Split on "is it sunny?": sunny gives 4 samples, all yes (H = 0); not sunny gives 6 samples, 1 yes and 5 no (H = 0.65).
• Weighted entropy after: (4/10)×0 + (6/10)×0.65 = 0.39 bits.
• Information gain: 1.0 − 0.39 = 0.61 bits.

The tree computes this for every feature and every threshold, and keeps the winner. Gini impurity is the cheaper alternative that almost always picks the same split — it approximates the same idea without the logarithm.

There is a known bias worth guarding against: information gain favours features with many distinct values, because splitting into many small pure groups always looks good. A unique ID column would score perfectly and generalise not at all. The gain ratio, which divides by the entropy of the split itself, corrects for this.

The family of related measures

Mutual information deserves particular attention as a feature-selection tool. Unlike correlation, it detects any dependence, including non-linear ones: a perfect parabolic relationship has near-zero correlation and high mutual information. It is zero exactly when the variables are independent.

KL divergence is not symmetric — D(p||q) ≠ D(q||p) — so it is a divergence rather than a distance. That asymmetry has real consequences in variational inference, where which direction you minimise determines whether the fitted distribution covers all the modes or concentrates on one.

Questions people ask

Why base 2? Because it gives the answer in bits, which correspond to yes/no questions. Base e gives nats, and base 10 gives dits — only the units change.

What is the maximum entropy? log₂(n) for n outcomes, reached when all are equally likely.

Is high entropy good or bad? Neither — it depends. High entropy in your labels means a hard problem; high entropy in a decision tree leaf means the split did not help; high entropy in a generated distribution means diversity.

What is the difference between entropy and cross-entropy? Entropy is the uncertainty of one distribution. Cross-entropy measures one distribution against another, and equals entropy plus KL divergence.

Why do trees use Gini instead of entropy? It is cheaper (no logarithm) and agrees with entropy on the chosen split the overwhelming majority of the time.

Can entropy be negative? Not for discrete distributions. Differential entropy for continuous ones can be, which is one of several reasons the continuous version needs more care.

Recap in one screen

• Entropy is average surprise, in bits: H = -Σ p log₂ p.
• Equally likely outcomes give maximum entropy; a certain outcome gives zero.
• Rare events carry more information, and the log makes information additive.
• Decision trees split to reduce entropy; cross-entropy loss trains classifiers; perplexity is entropy for language models.
• Mutual information detects any dependence, not just linear ones, which makes it a stronger feature-selection signal than correlation.