Information Gain
Slide a threshold through the data and watch the disorder on each side fall. How far it falls is the information the question bought you — and picking the question that buys the most is the entire decision tree algorithm.
The Question
everything below goes left, everything above goes right
The Split
Twenty Students, One Question
—passed failed
Entropy
Information Gain
Information Gain
Information gain measures how much a split reduces uncertainty about a target variable.
The high-cardinality trap
Information gain can favour features that create many small groups, including identifiers. A tree may appear to fit training data better while generalising worse. Validation, regularisation, and alternative criteria such as gain ratio can address this, depending on the task. Entropy itself is defined in entropy and information; the decision-tree lesson applies the split criterion.
Calculate a weighted split
Suppose a dataset has 8 examples: 4 positive and 4 negative. Its target entropy is 1 bit. A feature divides it into two groups of four. If the first group has 3 positive and 1 negative, and the second has 1 positive and 3 negative, each child entropy is about 0.811 bits. The weighted child entropy is also 0.811, so the information gain is about 1−0.811=0.189 bits. A split creates useful information when its groups are more label-pure than the original set.
The weights are group proportions, not an optional detail. If one child contains 90% of the examples and the other 10%, compute 0.9H(child₁)+0.1H(child₂); a tiny pure group should not outweigh the large group's uncertainty. Information gain is measured in the same units as entropy, commonly bits. It is a training-set criterion: choosing the split with the largest gain does not guarantee the best performance on new data. A feature with a unique value per row can create pure singleton leaves and receive high gain, so decision trees use depth limits, pruning, gain ratio, or validation to control overfitting.
Try this above
Compare a balanced split with an uneven split and inspect the weighted child entropy. Then add a split that isolates individual records to see why raw information gain can overvalue it.
Entropy reduction can also be written as mutual information between the target Y and the split outcome G: I(Y;G)=H(Y)−H(Y|G). It is zero when knowing the group gives no information about the label and reaches H(Y) when every group is pure. This interpretation clarifies that information gain is about association between a candidate split and labels, not causal influence. A feature may predict labels in training data because of leakage or chance; evaluate on held-out data and inspect whether the feature would be available at prediction time.
Sources and next steps
- Quinlan, “Induction of Decision Trees”, discusses information-based tree splitting.
- See cross-entropy and KL divergence for related distribution comparisons.
Overview
Entropy before and after a split
For target labels Y, entropy is H(Y) = −Σᵧ p(y)log₂p(y). A feature split divides the data into groups. Information gain is the original entropy minus the weighted entropy remaining in those groups:
Suppose a balanced binary target has entropy 1 bit. A split that creates two pure groups has conditional entropy 0 and information gain 1 bit. A split that leaves both groups equally mixed gives no gain. Weighting by group size matters: a tiny pure group should not count as much as a large one.