The one model you can read end to end - and the one whose answer changes when you resample the data.
Overview
How it works
Decision trees are one of the most interpretable models in machine learning. You can print the tree, read the splits, and explain every prediction to someone with no technical background. That transparency is why they remain popular even when other models score higher.
A decision tree learns a set of if-then rules from the data. Each rule tests one feature against one threshold. The rules are arranged in a tree structure: the first rule splits all the data, each subsequent rule splits a subset. At the bottom, each leaf holds a prediction.
Trees require no scaling and handle both numeric and categorical features naturally. They are fast to train, fast to predict, and easy to visualise. They also overfit aggressively. An unconstrained tree will memorise the training data perfectly. Controlling depth, minimum samples per leaf and minimum impurity decrease is how you trade accuracy for generalisation.
Understanding trees matters even if you plan to use random forests or gradient boosting. Both are built from trees. The parameters that control a single tree -- max_depth, min_samples_leaf, min_samples_split -- are the same parameters that control the trees inside an ensemble.
A choice tree asks a series of yes/no questions about one feature at a time. Each answer sends the sample down a branch. At the bottom, a leaf holds a guess. The majority class for a classifier, the mean target for a regressor.
Fitting means choosing the questions. At each node the method tries each feature and each threshold, scores how well each split separates the classes. Keeps the best. Then it repeats on each side, recursively, until a stopping rule fires.
The scoring function is criterion. Gini or entropy for classification, squared error for regression. It's the param people reach for first and the one that matters least. The editor above shows the three classification criteria within half a point of each other. This Is common. Depth and leaf size are the params that change the model.
Worth knowing
A tree is a sequence of yes/no questions on one feature at a time; export_text prints the whole model.
It needs no scaling and handles mixed feature ranges, because a split is a threshold on a single column.
It captures interactions and non-linearity without being told to look for them, which is why trees dominate on tabular data.
Left unconstrained it grows until every leaf is pure - a training score of 1.0 and a lookup table.
It is unstable: resampling the data changes the structure, which is exactly what ensembles average away.
It cannot extrapolate - predictions outside the training range are the nearest leaf's average, flat forever.
Decision Trees: A Practical Guide
The only model in this track whose fitted form a person can read and check. That readability comes with an instability that shows why trees are almost always used in groups.
The whole model, printed
No other estimator can do this - the fitted model is a set of questions you can read.
example_01.pyscikit-learn
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris
data = load_iris()
tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(data.data,
data.target)
print(export_text(tree, feature_names=list(data.feature_names)))
print("that is the entire model - a set of if/else questions")
Output
Which features it actually used
feature_importances_ sums to 1, and features never split on get exactly zero.
example_02.pyscikit-learn
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
data = load_iris()
tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(data.data,
data.target)
for name, imp in sorted(zip(data.feature_names, tree.feature_importances_),
key=lambda p: -p[1]):
print("%-22s %.4f" % (name, imp))
print()
print("sum:", round(tree.feature_importances_.sum(), 6))
print("two of the four features are never used at all")
Output
The split criterion barely matters
Which is worth knowing, because it is the parameter people reach for first and the one that changes least.
example_03.pyscikit-learn
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score
X, y = load_iris(return_X_y=True)
print("criterion accuracy")
for crit in ("gini", "entropy", "log_loss"):
s = cross_val_score(DecisionTreeClassifier(criterion=crit, random_state=0),
X, y, cv=5).mean()
print("%-11s %.4f" % (crit, s))
print()
print("the split rule almost never matters. depth and leaf size do.")
Output
One tree is unstable
Six resamples of the same data, six trees, and they do not agree.
example_04.pyscikit-learn
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
import numpy as np
X, y = make_classification(n_samples=200, n_features=8, n_informative=4,
n_redundant=2, random_state=0)
rng = np.random.RandomState(0)
firsts, preds = [], []
for _ in range(6):
idx = rng.choice(len(X), len(X), replace=True) # a bootstrap sample
t = DecisionTreeClassifier(random_state=0).fit(X[idx], y[idx])
firsts.append(int(t.tree_.feature[0]))
preds.append(t.predict(X[:40]))
print("feature chosen for the first split, six resamples:", firsts)
disagree = sum(len(set(col)) > 1 for col in zip(*preds))
print("of 40 rows, predictions disagree on:", disagree)
print()
print("resampling the same data changes the tree's structure and")
print("its answers. That instability is what a forest averages away.")
Output
It cannot extrapolate
A perfectly straight line, and a tree asked about values past the end of the training range.
example_05.pyscikit-learn
from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import LinearRegression
import numpy as np
x = np.arange(0, 20).reshape(-1, 1).astype(float)
y = 3.0 * x.ravel() # a perfectly straight line
tree = DecisionTreeRegressor(max_depth=3, random_state=0).fit(x, y)
lin = LinearRegression().fit(x, y)
future = np.array([[25.0], [30.0]])
print("truth at x=25, 30 :", [75.0, 90.0])
print("linear model :", lin.predict(future).round(1).tolist())
print("decision tree :", tree.predict(future).round(1).tolist())
print()
print("a tree predicts the average of a training leaf, so it cannot")
print("extrapolate beyond the range it was shown.")
Output
Units are irrelevant to it
One column multiplied by a million, and the cross-validated score does not move at all.
example_06.pyscikit-learn
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
X, y = make_classification(n_samples=300, n_features=8, n_informative=3,
random_state=0)
base = cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=5).mean()
X2 = X.copy()
X2[:, 0] *= 1_000_000
scaled = cross_val_score(DecisionTreeClassifier(random_state=0), X2, y, cv=5).mean()
print("original :", round(base, 4))
print("one column x1000000 :", round(scaled, 4))
print()
print("splits are thresholds on one column, so any monotonic change")
print("of units leaves the tree identical.")
Output
What makes trees attractive
They're readable.export_text prints the entire model. For a shallow tree that output is a thing a domain expert can check line by line. No other model in this track offers that. When somebody has to justify a choice. A loan refusal, a clinical flag. Being able to point at the path a case took is worth a great deal.
They need no preparation. No scaling. A split is a threshold on one column and units can't affect which threshold is best. The last editor times a column by a million and the cross-validated score doesn't move by a thousandth. No distributional assumptions. Outliers affect only the leaf they land in.
They find structure without being told. Interplays between features come for free. A split lower in the tree is conditioned on each split above it. Non-linear boundaries come for free, because a staircase of thresholds approximates any shape. A linear model needs to be handed x1 * x2 and x**2 by name. A tree discovers both.
They handle mixed data. Numeric and cat features side by side, no common scale needed.
What makes one tree a poor final answer
Left alone, it stores. The default stopping rule is to keep splitting until each leaf is pure. This On any dataset with noise means one leaf per training row. Training fit 1.0. A lookup table. each tree needs constraining. This Is what max_depth, min_samples_leaf and ccp_alpha are for.
It's unstable. This is the deeper problem. The fourth editor takes six bootstrap resamples of the same data, fits a tree to each. Finds them disagreeing on 18 of 40 guesses. With the first split falling on a different feature in one of the six.
The cause is the greedy fitting. The best split is chosen at each node without regard to what comes below. When two pick splits score nearly the same, a handful of resampled rows decides between them. That choice then finds all underneath. A small change at the top rewrites the whole tree.
The consequence is that a single tree's structure shouldn't be over-read. "The model says petal width is the key variable" is a statement about one fit on one sample. A different sample might have said a thing else.
It can't extrapolate. A leaf predicts an mean of training values. Beyond the range of the training data a tree returns a constant. The fifth editor fits a perfect straight line and asks for values past the end. The linear model gives 75 and 90, right. The tree gives 54 for both. For any of it with a trend. Time, growth, prices. This is disqualifying on its own.
Axis-aligned splits. Each split is on one feature. A main-line boundary has to be approximated by a staircase. Trees handle it, inefficiently. A linear model handles a main-line just.
Feature importances, and their bias
feature_importances_ reports how much each feature reduced the impurity across all the splits it appears in, normed to sum to 1. Features never split on get just zero. This Is a truly handy signal. The editor above shows two of iris's four features unused.
Three cautions before trusting the numbers.
Linked features split the credit arbitrarily. If two columns carry the same info, whichever is chosen first takes the importance and the other looks worthless. Neither is. The split between them isn't stable.
High-cardinality features are favoured. A column with many distinct values offers more pick thresholds and so more chances to look good on the training data. This is the impurity-based measure's known bias. It inflates the apparent importance of ids and smooth variables scaled to binary ones.
It's measured on the training data. A feature that helped fit noise still scores.
permutation_importance avoids all three by measuring the drop in test-set score when a column is shuffled. It costs more and answers the question people really mean.
Constraining it
Four params, of which the middle two are often the most handy.
max_depth caps the number of questions. Simple, blunt, and the first thing to try.
min_samples_leaf refuses to create a leaf with fewer than n samples. This is often better than depth. It reacts to the density of the data rather than imposing a fixed count. A tree can stay deep where there's data to support it.
min_samples_split is the related control on splitting rather than on the resulting leaves.
ccp_alpha grows the tree fully and then prunes back branches whose part doesn't justify their cost. It's the most principled of the four. The overfitting module found it giving the best test score of three constraints tried.
The right values are found by cross-validation, not chosen. The overfitting module also showed that a constraint can make things worse: min_samples_leaf=20 lowered the test score on data where max_depth=3 raised it.
Where a single tree belongs
Rarely as the final model. A random forest or a grad booster beats it on almost any dataset. The next modules cover both.
It belongs when the model has to be shown. A shallow tree is a flowchart somebody can audit. That is at times the need. It belongs as a fast baseline that tells you whether the problem has non-linear structure. And it belongs as the thing to understand first. Forests and boosting are both built out of trees. Their params are largely tree params.
Categorical features, and the one that is missing
scikit-learn's DecisionTreeClassifier can't take a string column. It needs numbers. This Means cat features have to be coded before they reach it. And the choice of encoding interacts badly with how trees split.
One-hot encoding a category with twenty values gives the tree twenty binary columns. A split on a binary column separates one category from the other nineteen. Isolating a group of three categories then takes three levels of depth. This The depth budget has to pay for. The tree can express it. It's just costly.
Ordinal encoding gives one column and lets a single threshold separate a contiguous run of categories. But the run is contiguous in what any order the encoder chose. The groups it can form cheaply are the wrong groups.
Neither is the native answer. This Is to consider any subsets of categories at each split. HistGradientBoostingClassifier supports that through categorical_features. It's a genuine reason to prefer it on data with many categories. LightGBM and CatBoost handle it too. It's one of the things they're known for.
For a plain tree with a handful of categories, one-hot and enough depth is fine. For a high-cardinality column, it's a real limitation rather than a detail.
Reading the structure programmatically
Beyond export_text, the fitted tree exposes its structure through tree_. A few fields answer handy questions right.
tree_.node_count and get_n_leaves() say how large the model is. A leaf count approaching the number of training rows is the clearest signal of memorisation available, and it costs one call.
tree_.feature and tree_.threshold are arrays giving, per node. This Feature was split on and at what value. Which is how the instability demo above read the first split without parsing text.
get_depth() says how deep it really grew. This Is often less than max_depth when another constraint bound first.
decision_path(X) returns which nodes each sample passed through. This Is the basis of any per-guess explanation. For a given row, the path is the reason.
These matter because they turn "the tree looks complicated" into numbers. A tree with 300 leaves on 400 training rows doesn't need a diagnosis. The leaf count is the diagnosis.
Is gini or entropy better? Neither, . They agree on almost each split and differ by a fraction of a point. Spend the effort on depth and leaf size.
Can a tree do regression? Yes - DecisionTreeRegressor predicts the mean of each leaf. The output is a step function. This Is why it can't extrapolate.
Should I prune or cap the depth?ccp_alpha prunes where pruning helps, rather than truncating all over at the same level. It often does better. It costs one more setting to tune.
Why is my tree different each run? Ties between equally good splits are broken at random. random_state fixes it. that it matters is the instability in miniature.
What "greedy" means, and why it matters
The fitting method is greedy: at each node it picks the split that looks best *right now*, without considering what splits become available afterwards.
That is a real limitation rather than an setup shortcut. Finding the truly best tree. The smallest one reaching a given fit. Is computationally unsolvable for any realistic dataset. Each hands-on setup is greedy. The consequence is that a tree can miss structure that needs two splits to reveal. If neither of a pair of features looks handy alone. But Their combination separates the classes just right, the greedy step may never take the first one.
The classic example is exclusive-or. With two binary features where the answer is "one but not both", neither feature alone carries any info at all. The first split scores nothing on either. A tree will eventually get there on real data with noise to guide it. It takes more depth than the structure warrants.
This also shows the instability from the other direction. When two pick splits score nearly the same way, the greedy choice between them is decided by whichever is fractionally better on this given sample. And all below inherits that any choice.
Ensembles help here too, and not only by blending. A random forest restricts each split to a random subset of features. This Forces different trees to take different first steps and at times finds the structure a single greedy path would have missed.
Things to try with Decision Trees
Read the model. The first editor prints the whole thing. Raise the depth to 4 and watch it stop being readable.
Watch the instability. In the fourth editor, change random_state on the resampler and see how much the six trees disagree this time.
Confirm the flat line. In the fifth editor, ask for x=100. The tree still says 54.
Compare the importances. Fit at depth 1 and at depth 5 and see how the numbers redistribute.
Questions people ask about Decision Trees
Do decision trees need scaling? No. Splits compare a feature to a threshold, so monotonic rescaling makes no difference.
Why do trees overfit? An unconstrained tree keeps splitting until each leaf is pure, memorising the training data. Limit it with max_depth or min_samples_leaf.
What do gini and entropy mean? Both measure how mixed a node is. In practice they give similar trees, so Gini is a fine default.
Can I see the tree? Yes. plot_tree draws it, and export_text prints the rules as text.
Are feature importances trustworthy? They are biased towards features with many distinct values. Permutation importance on held-out data is a safer check.
Do trees handle missing values? Recent versions of DecisionTreeClassifier support NaN natively, but check your installed version first.
Where this leaves you
A model you can read, that needs no preparation, that finds interplays on its own, that stores unless constrained, that changes when you resample. That can't see past the edge of its training data. The next two modules address the instability by using many of them at once.
Check yourself
0 of 4
Answer without scrolling back up.
Why do decision trees not need feature scaling?
Multiplying a column by a million left the cross-validated score identical to the fourth decimal.
What does an unconstrained decision tree do to the training data?
The default stopping rule is purity, which on noisy data means roughly one leaf per row - a lookup table.
What happens when a tree regressor is asked about values beyond its training range?
On a perfect straight line the tree predicted 54 for both x=25 and x=30, where the truth was 75 and 90.
Why is feature_importances_ unreliable for correlated features?
The credit goes to whichever was chosen, and that choice is unstable. permutation_importance measures the question people actually mean.
Cheat sheet
Decision Trees
Fit a decision tree, read its splits as plain rules, and see why a single tree overfits and how pruning brings it back.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.