A supervised learning algorithm that recursively splits data based on attributes yielding the highest Information Gain to form a predictive tree.
Overview
Overview
A Decision Tree is one of the most intuitive and interpretable models in machine learning. It makes predictions by learning a series of simple "if-then-else" rules from the data, forming a tree-like structure. This lab visualizes the ID3 (Iterative Dichotomiser 3) algorithm, which builds the tree by asking one simple question at each step: "Which feature gives me the most information to help classify my data?"
Node Data
Subset: Root
Splitting Actions
Decision Arena
Active Selected Leaf
Click a node to inspect its data or use the controls to split the active node.
Building a Decision Tree (ID3)
A Decision Tree is one of the most intuitive and interpretable models in machine learning. It makes predictions by learning a series of simple "if-then-else" rules from the data, forming a tree-like structure. This lab visualizes the ID3 (Iterative Dichotomiser 3) algorithm, which builds the tree by asking one simple question at each step: "Which feature gives me the most information to help classify my data?"
The Core Idea: Maximizing Information Gain
The goal of the ID3 algorithm is to build a tree that separates the data into pure groups (i.e., groups containing only a single class) as efficiently as possible. It does this by choosing the best feature to split on at each node. The "best" feature is the one that results in the most significant reduction in uncertainty, a concept measured by Information Gain.
Entropy: A Measure of Impurity
First, we measure the "impurity" or "randomness" of a set of data using a metric called Entropy. A dataset with a perfect mix of all classes (e.g., 50% 'Yes', 50% 'No') has the highest entropy (maximum uncertainty). A dataset with only one class (e.g., 100% 'Yes') has zero entropy (perfect certainty).
Information Gain: The Reduction in Entropy
For each feature, the algorithm calculates what the entropy would be after splitting the data based on that feature's values. Information Gain is simply the initial entropy minus the weighted average entropy after the split. The feature with the highest information gain is chosen for the split because it does the best job of creating purer, more organized subgroups.
The twenty questions analogy
A decision tree is the game of twenty questions, learned from data instead of played by instinct.
Each internal node asks one question about one feature — "is income above £40,000?", "is the weather sunny?" — and each answer sends you down a branch. Keep answering until you reach a leaf, and the leaf holds the prediction: a class for classification, an average value for regression.
That is why trees are the model you can show to a non-technical colleague without apologising. The path from root to leaf reads as a sentence: "applicants under 25 with less than two years of credit history and an existing loan were rejected 82% of the time." No other family of models hands you that for free.
The only real question in building one is which feature to ask about first, and the answer is: whichever question separates the classes best. A good first question splits a mixed pile into two much purer piles. A bad one leaves both halves as mixed as the original.
Measuring "purer", with real numbers
Two measures dominate, and both answer the same question: how mixed is this group?
Gini impurity is the chance you would label a randomly chosen item wrongly if you guessed labels at the group's own proportions. For a group that is a fraction p positive:
Gini = 1 − (p² + (1−p)²)
Entropy measures the same idea in bits of surprise:
Entropy = −p log₂(p) − (1−p) log₂(1−p)
Both are 0 for a pure group and highest for a 50/50 group — 0.5 for Gini, 1.0 bit for entropy.
Work one split through. Start with 10 items: 5 yes, 5 no. Entropy = 1.0 bit, as mixed as it gets. Now split on "is it sunny?":
The weighted entropy after the split is (4/10)×0 + (6/10)×0.65 = 0.39 bits. The information gain is 1.0 − 0.39 = 0.61 bits. The algorithm computes exactly this for every feature and every possible threshold, and keeps the winner. Then it repeats on each child, and keeps going until a stopping rule fires.
In practice Gini and entropy pick the same split the vast majority of the time. Gini is marginally cheaper because it has no logarithm, which is why it is scikit-learn's default. This is not a choice worth agonising over.
Why an unpruned tree always overfits
Left alone, a tree keeps splitting until every leaf is pure. On training data that means 100% accuracy, and on new data it means trouble, because the last few splits were made to accommodate individual rows — a leaf holding one customer is a rule about that customer, not about customers.
The controls that stop this are the ones worth knowing by name:
max_depth — how many questions deep the tree may go. The bluntest and most effective control. Depths of 3 to 8 cover most tabular problems.
min_samples_leaf — refuse to create a leaf with fewer than this many rows. Setting it to 20 or 50 kills the one-row rules directly.
min_samples_split — refuse to split a node that is already small.
max_features — consider only a random subset of features at each split. Mostly used inside forests.
ccp_alpha — cost-complexity pruning. Grow the tree fully, then remove the branches that do not pay for their complexity. Often the cleanest approach of all.
There is a second, subtler instability. Change a handful of training rows and the root split can change, which changes everything below it, which produces a visibly different tree that performs about the same. This high variance is not a bug you can tune away — it is the reason random forests and gradient boosting exist, and why a single tree is more often a communication tool than a production model.
How to Use the Interactive Lab
This lab lets you manually build a decision tree step-by-step to understand the ID3 algorithm's logic.
The Data: The table on the left shows the dataset for the currently selected node (initially, the whole dataset). The goal is to predict whether to 'Play' based on 'Outlook', 'Temp', 'Humidity', and 'Wind'.
Splitting Actions: The buttons under "Splitting Actions" correspond to the available features you can use to split the current node. Clicking one will show you the Information Gain for that split.
Building the Tree: When you click a split button (e.g., "Split on Outlook"), the tree in the "Decision Arena" expands. New child nodes are created, each containing the subset of data corresponding to a value of that feature (e.g., 'Sunny', 'Overcast', 'Rain').
Leaf Nodes: The process stops at a node when all data points in it belong to the same class. This is a "pure" node and becomes a leaf node (colored green), which makes the final prediction.
Read the tree it actually built
A depth-3 tree printed as the rules it learned, with the impurity drop at every split. Nothing here is a black box.
example_01.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split, cross_val_score
X, y = make_classification(n_samples=2000, n_features=6, n_informative=3,
n_redundant=0, flip_y=0.06, class_sep=1.3,
random_state=0)
names = ["f%d" % i for i in range(6)]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(Xtr, ytr)
print(export_text(tree, feature_names=names))
print("test accuracy %.4f" % tree.score(Xte, yte))
print()
print("how it chose the first split -- gini impurity before and after:")
def gini(labels):
if len(labels) == 0:
return 0.0
p = np.bincount(labels, minlength=2) / len(labels)
return 1 - (p ** 2).sum()
before = gini(ytr)
print(" impurity of the root: %.4f" % before)
best = None
for col in range(6):
for t in np.percentile(Xtr[:, col], [20, 40, 50, 60, 80]):
left, right = ytr[Xtr[:, col] <= t], ytr[Xtr[:, col] > t]
after = (len(left) * gini(left) + len(right) * gini(right)) / len(ytr)
if best is None or before - after > best[0]:
best = (before - after, col, t)
print(" best split found by brute force: f%d <= %.4f, impurity drop %.4f"
% (best[1], best[2], best[0]))
print(" scikit-learn chose: f%d <= %.4f"
% (tree.tree_.feature[0], tree.tree_.threshold[0]))
print(" same feature. the thresholds differ because this loop tried five")
print(" candidate cut points and scikit-learn tries the midpoint between")
print(" every adjacent pair of values -- the idea is identical, the search")
print(" is just finer.")
print()
print("if two sibling leaves show the SAME class, that is not a mistake. a")
print("split is chosen because it lowers impurity -- it makes one side purer")
print("-- and that can happen without either side's majority label flipping.")
print("export_text prints the label, not the mix behind it:")
for leaf in np.where(tree.tree_.children_left == -1)[0]:
counts = tree.tree_.value[leaf][0] * tree.tree_.n_node_samples[leaf]
print(" leaf %2d: %5d rows, %s -> class %d"
% (leaf, tree.tree_.n_node_samples[leaf],
np.round(counts).astype(int), counts.argmax()))
print()
print("a tree is a set of boxes, so it can only cut along the axes:")
print(" it will never draw a diagonal line, no matter how much data you give it.")
print(" it models x > 3 in one split and x + y > 3 only by stacking many.")
print()
print("depth is the whole trade-off:")
print("%8s %10s %10s %10s" % ("depth", "leaves", "train", "test"))
for d in (1, 2, 3, 5, 10, None):
t = DecisionTreeClassifier(max_depth=d, random_state=0).fit(Xtr, ytr)
print("%8s %10d %10.4f %10.4f"
% (d, t.get_n_leaves(), t.score(Xtr, ytr), t.score(Xte, yte)))
print()
print("feature importances are impurity drops added up, weighted by how many")
print("rows passed through each split:")
for n, imp in sorted(zip(names, tree.feature_importances_),
key=lambda kv: -kv[1]):
print(" %-4s %.4f %s" % (n, imp, "#" * int(imp * 50)))
print()
print("note the zeros -- a feature the tree never split on scores 0, even if")
print("it is genuinely predictive. it just lost every contest to a better one.")
Output
Experiments to try
Follow these steps to see the algorithm in action.
Find the Best First Split: Start with the root node selected. Click each of the four split buttons ('Outlook', 'Temp', etc.). Observe the Information Gain value displayed for each. You'll notice that 'Outlook' provides the highest gain. This is the split the ID3 algorithm would choose automatically. Click the "Best Split" button to confirm this.
Follow a Path to a Leaf Node: After splitting on 'Outlook', the tree has three new nodes. Click on the 'Overcast' node. Look at the data table on the left. You'll see that all four data points in this subset have a 'Play' value of 'Yes'. This is a pure node! The algorithm stops here and creates a leaf node that predicts 'Yes'.
Perform a Recursive Split: Now, select the 'Sunny' node. The data subset for this node is still impure (it contains both 'Yes' and 'No' values). You can split it again! Check the Information Gain for the remaining features ('Temp', 'Humidity', 'Wind'). Find the best feature to split the 'Sunny' node and continue the process until all branches end in pure leaf nodes.
The short of it
Greedy Algorithm: ID3 is a "greedy" algorithm. At each step, it picks the split that looks best at that moment (the one with the highest Information Gain) without looking ahead to see if a different choice might lead to a better overall tree.
Interpretability is a Superpower: The final tree structure is easy for humans to read and understand. You can trace the path for any new data point to see exactly how the model arrived at its prediction.
Foundation for Advanced Models: While simple, decision trees are the building blocks for more powerful ensemble models like Random Forests and Gradient Boosted Trees (like XGBoost).
Overfitting Risk: If a decision tree is grown too deep, it can perfectly memorize the training data, including its noise. This leads to overfitting. Techniques like pruning or setting a maximum depth are used to combat this.
Where trees are used, and where they are not
Trees handle several things that trip other models up. They need no feature scaling, because a threshold on income does not care whether income is in pounds or thousands of pounds. They cope with a mix of numeric and categorical features. They find interactions on their own — "young and in London" is just two levels of the same branch. And missing values can be handled by sending them down a default branch rather than being imputed.
What they are bad at is equally specific. A smooth linear relationship needs a staircase of many splits to approximate, so a plain line beats a tree on genuinely linear data. They cannot extrapolate at all: a tree trained on houses up to 200 square metres predicts the same value for a 500-square-metre house, because it has no leaf beyond the range it saw. And a single tree's predictions jump in steps rather than moving smoothly.
Situation
Single tree
Random forest
Gradient boosting
You must explain every decision
Best
Hard
Hard
Highest accuracy on tabular data
Weakest
Good
Usually best
Sensitive to noisy labels
Very
Less
More than a forest
Training cost
Trivial
Moderate, parallel
Higher, sequential
Tuning needed
Little
Little
Considerable
The honest summary: use one tree when a human has to read the model, and an ensemble of trees when a machine has to act on it.
Questions people ask
Do decision trees need scaled features? No. Splits are comparisons on one feature at a time, so units are irrelevant. This is one of the few places you can genuinely skip the scaler.
Gini or entropy? Either. They agree on the split almost always; Gini is faster. Spend the effort on depth and leaf size instead.
How do trees handle categories? Depends on the library. Scikit-learn expects numbers, so you encode first; LightGBM and CatBoost handle categorical columns natively, which is usually better because one-hot encoding a high-cardinality column produces many weak, sparse splits.
Why does feature importance disagree with my intuition? The built-in importance counts how much each feature reduced impurity, which is biased towards high-cardinality features — a column with many distinct values gets more chances to look useful. Permutation importance is a fairer measure.
Can trees do regression? Yes. Everything is identical except that the split criterion becomes variance reduction (or mean squared error) and each leaf predicts the average of its rows.
Why did my tree change completely after adding ten rows? That is the high-variance behaviour, not a bug. If stability matters, use a forest, or prune harder so the tree depends on broad patterns rather than fine ones.
Recap in one screen
A tree is a chain of yes/no questions ending in a prediction, learned greedily one split at a time.
Each split is chosen to make the resulting groups purer, measured by Gini impurity or entropy.
Grown without limits it will memorise the training data, so depth, leaf size and pruning are not optional.
No scaling needed, handles mixed data types, finds interactions, cannot extrapolate.
One tree to explain, an ensemble of trees to predict.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Overview”?
A Decision Tree is one of the most intuitive and interpretable models in machine learning. It makes predictions by learning a series of simple "if-then-else" rules from the data, forming a tree-like structure. This lab visualizes the ID3 (Iterative Dichotomiser 3) algorithm, which builds the tree by asking one simple question at each step: "Which feature gives me the most information to help classify my data?"
What does this module say about “The Core Idea: Maximizing Information Gain”?
The goal of the ID3 algorithm is to build a tree that separates the data into pure groups (i.e., groups containing only a single class) as efficiently as possible. It does this by choosing the best feature to split on at each node. The "best" feature is the one that results in the most significant reduction in uncertainty, a concept measured by Information Gain .
What does this module say about “Entropy: A Measure of Impurity”?
First, we measure the "impurity" or "randomness" of a set of data using a metric called Entropy . A dataset with a perfect mix of all classes (e.g., 50% 'Yes', 50% 'No') has the highest entropy (maximum uncertainty). A dataset with only one class (e.g., 100% 'Yes') has zero entropy (perfect certainty). Information Gain: The Reduction in Entropy
Cheat sheet
Decision Tree Analysis
A supervised learning algorithm that recursively splits data based on attributes yielding the highest Information Gain to form a predictive tree.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.