Information Gain
Slide a threshold through the data and watch the disorder on each side fall. How far it falls is the information the question bought you — and picking the question that buys the most is the entire decision tree algorithm.
The Question
everything below goes left, everything above goes right
The Split
Twenty Students, One Question
—passed failed
Entropy
Information Gain
Information Gain: A Practical Guide
How a decision tree decides what to ask, one question at a time.
Quick Context
Entropy measures how mixed a set of labels is: 1 bit for an even fifty-fifty split of two classes, 0 bits when every label is the same. Information gain is simply how much that number falls when you split the set in two.
A decision tree does nothing cleverer than this. At every node it tries every feature and every threshold, computes the gain for each, and keeps the winner. Then it repeats on both children until some stopping rule fires.
The formula
Gain = H(parent) − [ (nᴱ/n)·H(left) + (nᵣ/n)·H(right) ]
The weights matter. A child holding two points and a child holding eighteen do not deserve an equal say, so each child's entropy is weighted by the share of the data that landed in it. Without that weighting, a split that peels off one lucky point would look like a triumph.
Gain is never negative, and it is zero when the split changes nothing — when both sides have the same label mix as the parent, the question told you nothing you did not already know.
Interactive Exploration Guide
- Look at the parent. Twenty students, 11 passed and 9 failed, so entropy before any split is 0.993 bits — almost a full bit, because the classes are nearly balanced and knowing nothing else you would be nearly guessing.
- Split badly on purpose. Drag Threshold down to 1.5. One student goes left and nineteen go right, so the right-hand child still reads 11 pass / 8 fail — almost exactly the parent — and the gain collapses to 0.060 bits. You learned something about one student and nothing about the rest.
- Now press Find The Best Split. The threshold jumps from 4.5 to 6.5, the peak of the gain curve. Everyone who studied more than six hours passed, so that child is pure — entropy exactly 0.000 — and the gain climbs from 0.181 to 0.414 bits.
- Read the curve. The lower panel plots gain at every candidate threshold. It is jagged rather than smooth, because moving the threshold changes nothing at all until it crosses an actual data point. This is why trees only ever consider midpoints between observed values.
- Switch features. Change Split On Feature to attendance and press Find The Best Split again. The best it can manage is 0.145 bits at 82.5% — a third of what hours bought. That comparison — best gain from feature 1 against best gain from feature 2 — is precisely the choice the tree makes at each node.
- Find a worthless split. Push the threshold to either extreme so that one side is empty. The gain drops to 0.000: a split that separates nothing tells you nothing.
Gini, and why the choice barely matters
Many implementations measure impurity with the Gini index, 1 − Σ pᵢ², rather than entropy. Both are zero for a pure node and maximal for an even mix; Gini avoids a logarithm and is marginally cheaper. In practice the two pick the same split the overwhelming majority of the time, and scikit-learn's default of Gini is a performance decision, not a statistical one.
Entropy has the better story though: it is measured in bits, and gain is literally the number of bits of uncertainty the answer removed.
Where raw gain misleads
- High-cardinality features. Split on a column of unique IDs and every child holds one row, so every child is pure and the gain is maximal — on a feature with no predictive value whatsoever. Gain ratio, which divides by the entropy of the split itself, exists to punish exactly this.
- Greed. The tree takes the best split now, never the pair of splits that would be best together. It is a greedy algorithm, and greedy is not optimal — XOR-shaped data defeats the first split entirely.
- Overfitting. Keep splitting and gain stays non-negative all the way down to one point per leaf, at which point the tree has memorised the training set. Depth limits, minimum samples per leaf and pruning are all answers to this.
- Instability. Two candidate splits with nearly equal gain make the tree's choice a coin flip, and a different sample flips it. This variance is exactly what bagging was built to average away.
Key Takeaway
Information gain is the drop in entropy a split buys: the parent's entropy minus the size-weighted average of its children's. The weighting is what stops a tiny lucky child from looking impressive, and the gain is zero exactly when the split leaves the label mix unchanged. A decision tree builds itself by evaluating this quantity for every feature and every candidate threshold and taking the largest, which also explains its known weaknesses — the measure rewards splitting on high-cardinality columns, the search is greedy rather than globally optimal, and nothing in the gain itself ever tells you to stop.