Entropy and Information
Drag the bars and watch uncertainty rise and fall. Entropy is how many yes/no questions you would need to pin the answer down.
Distribution
0 = uniform, 1 = all on one outcome
The Distribution
sum = 1.000Drag any bar to change its probability. The rest rescale to keep the total at 1.
Entropy of a Two-Outcome Event
Maximum at p = 0.5, zero at both ends. The marker is the current split when there are two outcomes.
Uncertainty
Surprise per Outcome
Each outcome's surprise is −log₂(p). Entropy is the average surprise, weighted by how often each outcome happens.
Entropy and Information: A Practical Guide
One number for "how uncertain is this?", and the quantity every classification loss is built from.
Quick Context
Suppose I am about to tell you the outcome of an event, and you want to know how much you are about to learn. If the event is a coin flip, you will learn something. If it is "will the sun rise tomorrow", you will learn essentially nothing, because you already knew.
Entropy puts a number on that. It measures the uncertainty in a probability distribution — equivalently, how much information you gain when the outcome is revealed. It is the foundation of cross-entropy loss, of how decision trees pick their splits, and of compression.
Surprise first
Start with a single outcome. How surprising is it? Two things should be true: a certain outcome (p = 1) should carry zero surprise, and rarer outcomes should be more surprising. One function does this cleanly:
surprise(p) = −log2(p)
An outcome with p = 1 gives 0 bits. p = 0.5 gives 1 bit. p = 0.25 gives 2 bits. p = 0.01 gives about 6.6 bits. Halving the probability adds exactly one bit, every time — which is the property that makes the logarithm the right choice rather than an arbitrary one.
Entropy is average surprise
Now take the whole distribution and average the surprise, weighting each outcome by how often it actually occurs:
H = − Σ pi log2(pi)
That is the entire definition. The minus sign is there only because logs of numbers below 1 are negative, and we want a positive answer.
The interpretation worth memorising: entropy is the average number of yes/no questions needed to determine the outcome, if you ask the smartest possible questions. A fair coin needs exactly 1. A fair 8-sided die needs exactly 3, because 2³ = 8.
Interactive Exploration Guide
- Find the maximum. Click Make It Maximally Uncertain. Every outcome gets equal probability and entropy hits its ceiling. Uniform is always the most uncertain a distribution can be.
- Find the floor. Click Make It Certain. All the probability moves onto one outcome and entropy drops to exactly 0 — there is nothing left to learn, so revealing the answer tells you nothing.
- Watch the curve. Set the Number of Outcomes slider to 2 and drag the Skew slider slowly from 0 to 1. Entropy traces the arch in the lower panel: 1 bit at an even split, falling to 0 at either extreme.
- Count the bits. Set the Number of Outcomes slider to 8 with skew at 0. Entropy reads exactly 3.000, because eight equally likely outcomes take three yes/no questions to separate.
- See rare events carry more. Set the Skew slider to about 0.8 and look at the Surprise panel. The rare outcomes have large surprise values but small weights, which is why entropy stays low — you are rarely surprised, even though you would be very surprised if it happened.
- Compare against Gini. Watch the Gini Impurity readout as you drag. It rises and falls with entropy and shares the same zero point, which is why decision trees can use either and rarely care which.
Why machine learning cares
- Decision trees choose the split that reduces entropy the most. That reduction is called information gain, and it is literally "how much uncertainty did this question remove".
- Cross-entropy loss measures the average surprise of the true labels under your model's distribution. Minimising it means making the truth unsurprising to the model.
- KL divergence is the gap between the cross-entropy and the true entropy — the extra bits you pay for using the wrong distribution.
- Compression gets the same number from the other direction: entropy is the theoretical floor on the average bits per symbol, which is why a file of random bytes will not compress.
Bits, nats and why it rarely matters
Using log base 2 gives bits, which is the interpretable unit and the one this page uses. Machine learning code almost always uses the natural log instead, giving nats. The two differ by a constant factor of ln(2) ≈ 0.693, and since optimisation only cares about which direction is downhill, a constant factor changes nothing about training. It does mean a cross-entropy loss printed by a framework is in nats, so 0.693 is the "fair coin" value there rather than 1.0.
What usually goes wrong
- Feeding it a zero. log(0) is undefined. The convention is that 0·log(0) = 0, which is the correct limit, but a naive implementation returns NaN. Clamp probabilities away from zero.
- Reading entropy as a probability. It is measured in bits and is unbounded above as outcomes are added — eight outcomes can reach 3, sixteen can reach 4. Only the fraction-of-maximum reading is comparable across different-sized distributions.
- Confusing it with variance. Both measure spread, but entropy ignores the values entirely. Relabelling the outcomes changes variance and leaves entropy untouched.
- Expecting Gini and entropy to disagree. On classification splits they almost never pick different features. Choosing between them is not worth tuning.
Key Takeaway
Entropy is the average surprise of a distribution, −Σp·log(p), and reads as the number of yes/no questions needed to pin down the outcome. It is maximal when every outcome is equally likely and exactly zero when one outcome is certain, which is why it works as a measure of uncertainty. Every classification loss in machine learning is built from it: cross-entropy is the average surprise of the true labels under your model, and information gain — the entropy a split removes — is how a decision tree decides what to ask.