Entropy and Information

Entropy depends on the distribution and units

Entropy is highest for a uniform distribution over a fixed finite set: if there are n equally likely outcomes, H = log₂(n). Changing the logarithm base changes units, not the underlying uncertainty: base 2 gives bits and base e gives nats.

Entropy does not measure the importance of an outcome or uncertainty about every aspect of a system. It is defined relative to a probability model. In machine learning, it appears in cross-entropy and KL divergence and in information gain.

Calculate uncertainty for a biased coin

Suppose a coin lands heads with probability 0.9 and tails with probability 0.1. The entropy is −0.9log₂(0.9)−0.1log₂(0.1)≈0.469 bits, less than the 1 bit for a fair coin. The rare tail carries more self-information (−log₂0.1≈3.32 bits) than the common head (−log₂0.9≈0.15 bits), but the tail occurs less often; entropy averages both contributions using their probabilities. If the outcome is certain, its surprise is zero and entropy is zero.

Entropy measures uncertainty about the outcome under a specified distribution, not the importance of the outcome or the amount of data stored by every encoding system. For n equally likely outcomes it is log₂n bits, and no distribution on those same n outcomes has higher entropy. Entropy also depends on how outcomes are defined: grouping several outcomes together changes the distribution and its entropy. Use the same outcome categories and logarithm base when comparing values. In machine learning, entropy can quantify label uncertainty before and after a split; information gain makes that reduction explicit.

Try this above

Make the outcome probabilities more or less balanced. Entropy should increase as the distribution approaches uniformity and decrease as one outcome becomes nearly certain.

Entropy is additive for independent discrete variables: if X and Y are independent, H(X,Y)=H(X)+H(Y). If they are dependent, knowing X can reduce uncertainty about Y, and H(X,Y)=H(X)+H(Y|X). Conditional entropy measures the remaining average uncertainty after X is known. This makes entropy useful for reasoning about compression and prediction, but the result depends on the probability model and the granularity of the categories. A model that assigns incorrect probabilities gives an entropy value for that model, not an objective measure of every kind of uncertainty.

For a fair six-sided die, entropy is log₂6≈2.585 bits. This is an average uncertainty, so it need not be a whole number even though any one outcome can be represented with a finite code. An efficient code assigns shorter descriptions to common outcomes and longer ones to rare outcomes; entropy is the ideal lower bound on average code length under suitable coding assumptions. It does not prescribe a unique code or count the physical bits in every data format.

Sources and next steps

• Shannon, “A Mathematical Theory of Communication”, introduced entropy as an information measure.
• Continue with cross-entropy and KL divergence to compare two distributions.