The one model you can read end to end - and the one whose answer changes when you resample the data.
Overview
How it works
A decision tree asks a series of yes/no questions about one feature at a time, and each answer sends the sample down a branch. At the bottom, a leaf holds a prediction: the majority class for a classifier, the mean target for a regressor.
Fitting means choosing the questions. At each node the algorithm tries every feature and every threshold, scores how well each split separates the classes, and keeps the best. Then it repeats on each side, recursively, until a stopping rule fires.
The scoring function is criterion — gini or entropy for classification, squared error for regression. It is the parameter people reach for first and the one that matters least: the editor above shows the three classification criteria within half a point of each other, which is typical. Depth and leaf size are the parameters that change the model.
Worth knowing
A tree is a sequence of yes/no questions on one feature at a time; export_text prints the whole model.
It needs no scaling and handles mixed feature ranges, because a split is a threshold on a single column.
It captures interactions and non-linearity without being told to look for them, which is why trees dominate on tabular data.
Left unconstrained it grows until every leaf is pure - a training score of 1.0 and a lookup table.
It is unstable: resampling the data changes the structure, which is exactly what ensembles average away.
It cannot extrapolate - predictions outside the training range are the nearest leaf's average, flat forever.
Decision Trees: A Practical Guide
The only model in this track whose fitted form a person can read and check. That readability comes with an instability that explains why trees are almost always used in groups.
The whole model, printed
No other estimator can do this - the fitted model is a set of questions you can read.
example_01.pyscikit-learn
Output
Which features it actually used
feature_importances_ sums to 1, and features never split on get exactly zero.
example_02.pyscikit-learn
Output
The split criterion barely matters
Which is worth knowing, because it is the parameter people reach for first and the one that changes least.
example_03.pyscikit-learn
Output
One tree is unstable
Six resamples of the same data, six trees, and they do not agree.
example_04.pyscikit-learn
Output
It cannot extrapolate
A perfectly straight line, and a tree asked about values past the end of the training range.
example_05.pyscikit-learn
Output
Units are irrelevant to it
One column multiplied by a million, and the cross-validated score does not move at all.
example_06.pyscikit-learn
Output
What makes trees attractive
They are readable.export_text prints the entire model, and for a shallow tree that output is something a domain expert can check line by line. No other model in this track offers that. When somebody has to justify a decision — a loan refusal, a clinical flag — being able to point at the path a case took is worth a great deal.
They need no preparation. No scaling, because a split is a threshold on one column and units cannot affect which threshold is best. The last editor multiplies a column by a million and the cross-validated score does not move by a thousandth. No distributional assumptions, and outliers affect only the leaf they land in.
They find structure without being told. Interactions between features come for free, because a split lower in the tree is conditioned on every split above it. Non-linear boundaries come for free, because a staircase of thresholds approximates any shape. A linear model needs to be handed x1 * x2 and x**2 explicitly; a tree discovers both.
They handle mixed data. Numeric and categorical features side by side, no common scale needed.
What makes one tree a poor final answer
Left alone, it memorises. The default stopping rule is to keep splitting until every leaf is pure, which on any dataset with noise means one leaf per training row. Training accuracy 1.0, and a lookup table. Every tree needs constraining, which is what max_depth, min_samples_leaf and ccp_alpha are for.
It is unstable. This is the deeper problem. The fourth editor takes six bootstrap resamples of the same data, fits a tree to each, and finds them disagreeing on 18 of 40 predictions — with the first split falling on a different feature in one of the six.
The cause is the greedy fitting. The best split is chosen at each node without regard to what comes below, and when two candidate splits score nearly the same, a handful of resampled rows decides between them. That choice then determines everything underneath, so a small change at the top rewrites the whole tree.
The consequence is that a single tree's structure should not be over-interpreted. "The model says petal width is the key variable" is a statement about one fit on one sample, and a different sample might have said something else.
It cannot extrapolate. A leaf predicts an average of training values, so beyond the range of the training data a tree returns a constant. The fifth editor fits a perfect straight line and asks for values past the end: the linear model gives 75 and 90, correctly, and the tree gives 54 for both. For anything with a trend — time, growth, prices — this is disqualifying on its own.
Axis-aligned splits. Each split is on one feature, so a diagonal boundary has to be approximated by a staircase. Trees handle it, inefficiently, and a linear model handles a diagonal exactly.
Feature importances, and their bias
feature_importances_ reports how much each feature reduced the impurity across all the splits it appears in, normalised to sum to 1. Features never split on get exactly zero, which is a genuinely useful signal — the editor above shows two of iris's four features unused.
Three cautions before trusting the numbers.
Correlated features split the credit arbitrarily. If two columns carry the same information, whichever is chosen first takes the importance and the other looks worthless. Neither is, and the split between them is not stable.
High-cardinality features are favoured. A column with many distinct values offers more candidate thresholds and so more chances to look good on the training data. This is the impurity-based measure's known bias, and it inflates the apparent importance of ids and continuous variables relative to binary ones.
It is measured on the training data. A feature that helped fit noise still scores.
permutation_importance avoids all three by measuring the drop in test-set score when a column is shuffled. It costs more and answers the question people actually mean.
Constraining it
Four parameters, of which the middle two are usually the most useful.
max_depth caps the number of questions. Simple, blunt, and the first thing to try.
min_samples_leaf refuses to create a leaf with fewer than n samples. This is often better than depth, because it responds to the density of the data rather than imposing a fixed count — a tree can stay deep where there is data to support it.
min_samples_split is the related control on splitting rather than on the resulting leaves.
ccp_alpha grows the tree fully and then prunes back branches whose contribution does not justify their complexity. It is the most principled of the four, and the overfitting module found it giving the best test score of three constraints tried.
The right values are found by cross-validation, not chosen. The overfitting module also showed that a constraint can make things worse: min_samples_leaf=20 lowered the test score on data where max_depth=3 raised it.
Where a single tree belongs
Rarely as the final model. A random forest or a gradient booster beats it on almost any dataset, and the next modules cover both.
It belongs when the model has to be explained — a shallow tree is a flowchart somebody can audit, and that is occasionally the requirement. It belongs as a fast baseline that tells you whether the problem has non-linear structure. And it belongs as the thing to understand first, because forests and boosting are both built out of trees, and their parameters are largely tree parameters.
Categorical features, and the one that is missing
scikit-learn's DecisionTreeClassifier cannot take a string column. It requires numbers, which means categorical features have to be encoded before they reach it — and the choice of encoding interacts badly with how trees split.
One-hot encoding a category with twenty values gives the tree twenty binary columns, and a split on a binary column separates one category from the other nineteen. Isolating a group of three categories then takes three levels of depth, which the depth budget has to pay for. The tree can express it; it is just expensive.
Ordinal encoding gives one column and lets a single threshold separate a contiguous run of categories — but the run is contiguous in whatever arbitrary order the encoder chose, so the groups it can form cheaply are the wrong groups.
Neither is the native answer, which is to consider arbitrary subsets of categories at each split. HistGradientBoostingClassifier supports that through categorical_features, and it is a genuine reason to prefer it on data with many categories. LightGBM and CatBoost handle it too, and it is one of the things they are known for.
For a plain tree with a handful of categories, one-hot and enough depth is fine. For a high-cardinality column, it is a real limitation rather than a detail.
Reading the structure programmatically
Beyond export_text, the fitted tree exposes its structure through tree_, and a few attributes answer useful questions directly.
tree_.node_count and get_n_leaves() say how large the model is. A leaf count approaching the number of training rows is the clearest signal of memorisation available, and it costs one call.
tree_.feature and tree_.threshold are arrays giving, per node, which feature was split on and at what value — which is how the instability demonstration above read the first split without parsing text.
get_depth() says how deep it actually grew, which is often less than max_depth when another constraint bound first.
decision_path(X) returns which nodes each sample passed through, which is the basis of any per-prediction explanation: for a given row, the path is the reason.
These matter because they turn "the tree looks complicated" into numbers. A tree with 300 leaves on 400 training rows does not need a diagnosis; the leaf count is the diagnosis.
Is gini or entropy better? Neither, in practice. They agree on almost every split and differ by a fraction of a point. Spend the effort on depth and leaf size.
Can a tree do regression? Yes - DecisionTreeRegressor predicts the mean of each leaf. The output is a step function, which is why it cannot extrapolate.
Should I prune or cap the depth?ccp_alpha prunes where pruning helps, rather than truncating everywhere at the same level. It usually does better, and it costs one more hyperparameter to tune.
Why is my tree different each run? Ties between equally good splits are broken randomly. random_state fixes it, and the fact that it matters is the instability in miniature.
What "greedy" means, and why it matters
The fitting algorithm is greedy: at every node it picks the split that looks best *right now*, without considering what splits become available afterwards.
That is a real limitation rather than an implementation shortcut. Finding the genuinely optimal tree — the smallest one achieving a given accuracy — is computationally intractable for any realistic dataset, so every practical implementation is greedy. The consequence is that a tree can miss structure that requires two splits to reveal: if neither of a pair of features looks useful alone, but their combination separates the classes perfectly, the greedy step may never take the first one.
The classic example is exclusive-or. With two binary features where the answer is "one but not both", neither feature alone carries any information at all, so the first split scores nothing on either. A tree will eventually get there on real data with noise to guide it, and it takes more depth than the structure warrants.
This also explains the instability from the other direction. When two candidate splits score nearly identically, the greedy choice between them is decided by whichever is fractionally better on this particular sample — and everything below inherits that arbitrary decision.
Ensembles help here too, and not only by averaging. A random forest restricts each split to a random subset of features, which forces different trees to take different first steps and occasionally finds the structure a single greedy path would have missed.
Things to try
Read the model. The first editor prints the whole thing. Raise the depth to 4 and watch it stop being readable.
Watch the instability. In the fourth editor, change random_state on the resampler and see how much the six trees disagree this time.
Confirm the flat line. In the fifth editor, ask for x=100. The tree still says 54.
Compare the importances. Fit at depth 1 and at depth 5 and see how the numbers redistribute.
Where this leaves you
A model you can read, that needs no preparation, that finds interactions on its own, that memorises unless constrained, that changes when you resample, and that cannot see past the edge of its training data. The next two modules address the instability by using many of them at once.
Check yourself
0 of 4
Answer without scrolling back up.
Why do decision trees not need feature scaling?
Multiplying a column by a million left the cross-validated score identical to the fourth decimal.
What does an unconstrained decision tree do to the training data?
The default stopping rule is purity, which on noisy data means roughly one leaf per row - a lookup table.
What happens when a tree regressor is asked about values beyond its training range?
On a perfect straight line the tree predicted 54 for both x=25 and x=30, where the truth was 75 and 90.
Why is feature_importances_ unreliable for correlated features?
The credit goes to whichever was chosen, and that choice is unstable. permutation_importance measures the question people actually mean.
Cheat sheet
Decision Trees
The only model in this track whose fitted form a person can read and check. That readability comes with an instability that explains why trees are almost always used in groups.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.