How a split is chosen
The tree looks for a feature question that makes the resulting groups less mixed. For binary classes with positive fraction p, entropy is:
H(p) = −p log₂(p) − (1 − p) log₂(1 − p)
Entropy is 0 when a group contains one class only and 1 bit when the classes are evenly mixed. Information gain is the parent’s entropy minus the size-weighted average entropy of its child groups. The tree picks the split with the greatest gain, then repeats the process in each child until a stopping rule applies.
For example, start with 10 cases: five Yes and five No, so entropy is 1 bit. Suppose a “sunny?” question makes one child with four Yes cases and no No cases (entropy 0), and a second child with one Yes and five No (entropy about 0.65). The weighted child entropy is 0.4 × 0 + 0.6 × 0.65 = 0.39 bits. The split gains about 1 − 0.39 = 0.61 bits.
This page uses information gain and categorical features. Other tree implementations may use Gini impurity or continuous thresholds; check the estimator’s splitting rules before assuming every tree grows the same way.
Follow the split in the visualiser
- At the root, compare the displayed information gain for the available features. Best Split selects the feature with the largest gain.
- After splitting on Outlook, select the Overcast child. Its examples all have the same Play label, so it is a pure leaf and needs no further split.
The example shows the rule-building process, not an automatic claim that the chosen tree will generalise. A tree can keep splitting until it memorises its training examples.
When trees are useful
Tree paths are readable as a sequence of decisions, and the model can represent nonlinear boundaries and feature interactions without scaling numeric inputs. A small, pruned tree can be a useful explanation or baseline.
The main risk is overfitting. Limit depth, require enough samples per leaf, or prune using validation data. A high information-gain feature is useful for this tree’s split; it does not prove that the feature causes the outcome. Deep trees can also be unstable: a small change in training data may produce a different early split and a different tree. Ensembles such as random forests reduce some of that instability at the cost of a less transparent prediction process.
Sources and next steps
For scikit-learn’s tree implementation and its splitting criteria, see the official decision-tree guide. Continue with random forests to see how many trees are combined.