Many unstable trees, deliberately made to disagree, averaged into something stable - and the strongest default on tabular data.
Overview
The idea
A random forest is an ensemble of decision trees. It fits many trees on slightly different versions of the data and averages their predictions. The average is almost always more accurate than any single tree.
Random forests are the default first model for many practitioners. They work on classification and regression, handle numeric and categorical features, require little tuning, and resist overfitting better than single trees. On tabular data, they are competitive with gradient-boosted trees and far simpler to configure.
The forest's main parameter is the number of trees. More trees give a better average, up to a point of diminishing returns. The cost is linear in the number of trees, but the trees are independent and can be fitted in parallel across CPU cores.
A single choice tree is unstable: the greedy choice at each node is decided by a handful of rows. A different sample makes a different tree. That instability is variance, and variance is error.
Blending free guesses reduces variance. If the trees made *the same* errors, blending would achieve nothing. If their errors are partly free, the errors cancel and the signal lives. So the whole design problem is making the trees disagree.
A random forest adds disagreement twice.
Bootstrap sampling. Each tree is fitted on a sample of the training rows drawn with swap, the same size as the first. About 63% of the rows appear in any given sample. Each tree sees a different subset.
Random feature subsets. At each split, the tree considers only a random subset of the features rather than all of them. This is the more key of the two. It's what sets apart a random forest from plain bagging.
The second matters because bootstrapping alone leaves the trees too similar. If one feature is strongly predictive, each tree splits on it first and the trees end up nearly the same. Restricting the choice forces different trees down different paths. This Is just the freedom the blending needs.
Worth knowing
Each tree is fitted on a bootstrap sample and picks each split from a random subset of features - two sources of disagreement, deliberately introduced.
Averaging cancels errors that are independent, so the more the trees differ, the more the averaging buys.
n_estimators cannot overfit; more trees only cost time.
oob_score=True gives an estimate from the rows each tree did not see, for the price of a single fit.
feature_importances_ is biased towards high-cardinality columns - permutation_importance answers the question people mean.
It inherits the tree's inability to extrapolate: predictions past the training range are flat.
Random Forests: A Practical Guide
The previous module ended with a model that changes its mind when you resample the data. This one fixes that by building hundreds of them and taking a vote.
More accurate, and far more stable
The accuracy gain is modest. The stability gain is the real one, and it has to be measured across refits rather than across folds.
example_01.pyscikit-learn
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = make_classification(n_samples=600, n_features=20, n_informative=6,
random_state=0)
print("accuracy:")
print(" one tree :", round(cross_val_score(DecisionTreeClassifier(random_state=0),
X, y, cv=5).mean(), 4))
print(" a forest :", round(cross_val_score(RandomForestClassifier(random_state=0),
X, y, cv=5).mean(), 4))
# The variance that averaging removes is in the predictions, not in the
# fold scores - so measure it by refitting on resampled data.
rng = np.random.RandomState(0)
for label, make in [("one tree", lambda: DecisionTreeClassifier(random_state=0)),
("a forest", lambda: RandomForestClassifier(n_estimators=50,
random_state=0))]:
preds = []
for _ in range(5):
idx = rng.choice(len(X), len(X), replace=True)
preds.append(make().fit(X[idx], y[idx]).predict(X[:60]))
disagree = sum(len(set(col)) > 1 for col in zip(*preds))
print("%-10s disagrees with itself on %2d of 60 rows across refits"
% (label, disagree))
Output
n_estimators is the one free parameter
Raising it cannot overfit. It only costs time.
example_02.pyscikit-learn
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
X, y = make_classification(n_samples=500, n_features=20, n_informative=5,
random_state=0)
print("n_estimators accuracy")
for n in (1, 5, 25, 100, 300):
s = cross_val_score(RandomForestClassifier(n_estimators=n, random_state=0),
X, y, cv=5).mean()
print("%12d %.4f" % (n, s))
print()
print("one tree is much worse; after about 25 the differences are")
print("fold noise. more trees costs time and does not overfit -")
print("it is the one parameter you can raise without a trade.")
Output
Two importances, one of them misleading
A pure-noise column with many distinct values, and the built-in measure gives it more credit than the other noise column.
example_03.pyscikit-learn
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
import numpy as np
rng = np.random.RandomState(0)
n = 500
signal = rng.normal(size=n)
X = np.column_stack([
signal, # genuinely useful
rng.normal(size=n), # noise
rng.randint(0, 400, size=n), # noise, many distinct values
])
y = (signal > 0).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
rf = RandomForestClassifier(random_state=0).fit(X_tr, y_tr)
names = ["signal", "noise", "noise_many_values"]
perm = permutation_importance(rf, X_te, y_te, n_repeats=10, random_state=0)
print("%-20s %10s %12s" % ("", "impurity", "permutation"))
for i, nm in enumerate(names):
print("%-20s %10.4f %12.4f"
% (nm, rf.feature_importances_[i], perm.importances_mean[i]))
print()
print("impurity gives both noise columns a score, and more to the one")
print("with more distinct values. permutation gives both exactly zero.")
Output
A free estimate, without a second fit
Each tree misses about a third of the rows, so those rows can score it.
example_04.pyscikit-learn
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
X, y = make_classification(n_samples=600, n_features=20, n_informative=6,
random_state=0)
rf = RandomForestClassifier(n_estimators=200, oob_score=True,
random_state=0).fit(X, y)
print("out-of-bag score :", round(rf.oob_score_, 4))
print("5-fold cross-validated:",
round(cross_val_score(RandomForestClassifier(n_estimators=200,
random_state=0),
X, y, cv=5).mean(), 4))
print()
print("each tree is scored on the rows its bootstrap sample left out,")
print("which gives an estimate for the cost of a single fit.")
Output
max_features is the decorrelation dial
Letting every tree see every feature makes them agree, and agreement is what averaging cannot fix.
example_05.pyscikit-learn
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
X, y = make_classification(n_samples=400, n_features=30, n_informative=8,
random_state=0)
print("max_features accuracy")
for mf in ("sqrt", 0.5, None):
s = cross_val_score(RandomForestClassifier(n_estimators=50, max_features=mf,
random_state=0), X, y, cv=5).mean()
print("%12s %.4f" % (str(mf), s))
print()
print("None means every tree sees every feature, so the trees resemble")
print("each other more and the averaging buys less.")
Output
Still cannot extrapolate
The one limitation the ensemble inherits whole from its trees.
example_06.pyscikit-learn
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
import numpy as np
x = np.arange(0, 20).reshape(-1, 1).astype(float)
y = 3.0 * x.ravel()
rf = RandomForestRegressor(n_estimators=50, random_state=0).fit(x, y)
lin = LinearRegression().fit(x, y)
future = np.array([[25.0], [40.0]])
print("truth at x=25, 40 :", [75.0, 120.0])
print("linear :", lin.predict(future).round(1).tolist())
print("random forest :", rf.predict(future).round(1).tolist())
print()
print("averaging many trees does not fix extrapolation - every tree")
print("is flat past the edge, so the average is too.")
Output
What it buys
The fit improvement is real and often modest. 0.79 to 0.83 in the first editor.
The steadiness improvement is the larger one. It has to be measured properly. Fold-to-fold variation in a cross-validation score is *not* it. That measures how much the score depends on which rows were held out, and a forest isn't above all better there. What blending fixes is how much the *model* changes when the training data changes.
The first editor measures that right: refitting on five bootstrap resamples, one tree disagrees with itself on 14 of 60 rows. A fifty-tree forest on 6. That is the variance reduction. It's why a forest's guesses can be leaned on in a way a single tree's can't.
The parameters worth setting
n_estimators is unusual: it can't overfit. More trees means a better guess of the mean. This Settles rather than degrading. The only cost is time and memory. a thing like 100 to 500 is normal. The editor above shows the curve flattening after about 25. Past which the differences are fold noise rather than improvement.
That makes it the one cost-shaped param with no trade-off. This Is worth stating because it runs against the pattern of all else in the track.
max_features controls the decorrelation. "sqrt" is the default for classification and is often right. Setting it to None gives each tree each feature. The third editor shows the cost: 0.8500 against 0.8650. The trees became too alike. Lower values mean more diverse and on their own weaker trees. This Is often a good trade.
max_depth and min_samples_leaf are inherited from the tree. A forest needs them far less. Single trees are allowed to overfit. Their overfitting is unlinked and means out. Leaving them unconstrained is a reasonable default. Constraining them slightly can help on small or very noisy data.
n_jobs=-1 fits the trees in parallel across cores. The trees are free. This is close to free.
Out-of-bag scoring
Because each tree misses about 37% of the rows, those rows can score it. And blending that across trees gives a validation guess without a separate fit.
oob_score=True finds it, and oob_score_ holds the result. The editor above gives 0.8683 against a cross-validated 0.8383. This Is the usual tie: OOB tends to run slightly optimistic scaled to k-fold. Each OOB guess comes from the subset of trees that missed that row rather than from a full forest.
It's truly handy when fitting is costly. It costs nothing beyond the fit you were doing anyway. It isn't a swap for cross-validation when you're checking models or tuning, both because of the optimism and because it only exists for bagged ensembles.
Feature importances, and the trap
feature_importances_ is the field people reach for. It has a known bias worth seeing rather than being told about.
The third editor gives a forest three columns. One carrying the signal, one pure noise. One pure noise with 400 distinct values. The impurity-based measure gives the signal 0.89. Correct. And then gives both noise columns a non-zero score, with more going to the one with more distinct values.
The cause is that a column with many possible thresholds gets many chances to look good on the training data. Impurity importance is found on the training data. permutation_importance. This Measures how much the *test* score drops when a column is shuffled, gives both noise columns just 0.0000.
Two further cautions apply to both measures. Linked features split the credit arbitrarily. A truly key feature can look weak because a linked one absorbed it. And importance isn't causation. It says the model used the column, not that the column drives the outcome.
The hands-on rule: use permutation importance when the answer matters, on the test set. Treat the built-in field as a rough first look.
What it does not fix
Extrapolation. each tree is flat past the edge of its training range. The mean of many trees is flat too. The last editor asks for x=25 and x=40 on a perfect straight line and gets 55.3 for both. No number of trees changes this. It disqualifies forests for any of it with a trend unless the trend is removed first.
Readability. A single shallow tree can be printed and audited. Three hundred trees can't. The model turns a black box that happens to be made of transparent parts.
Size and speed. A forest holds each tree. A model fitted on a large dataset can be hundreds of megabytes. Guess means traversing each tree.
Extreme sparsity. On very high-dimensional sparse data. Text, mostly — linear models often win.
Where it sits
A random forest is the strongest thing you can fit with no tuning. That is its main claim. Defaults plus n_estimators=200 makes a competitive model on most tabular problems, needing no scaling, no encoding choices beyond making the columns numeric, and no search.
Grad boosting often beats it given tuning effort. Grad boosting *needs* that effort. A badly tuned booster is worse than a default forest. The sensible sequence is a linear model for the baseline, a forest for the strong default. Boosting when the difference is worth the work.
The rest of the bagging family
Three near relatives share the design and differ in one choice each. Knowing what separates them saves trying all four blindly.
ExtraTreesClassifier — extremely randomised trees — goes further than a forest. Rather than searching for the best threshold on each pick feature, it picks thresholds at random and keeps the best of those. That makes each tree weaker and the trees more different. This At times wins and is always faster to fit. The costly threshold search disappears. It's worth trying when a forest is your best model. It takes one word to swap.
BaggingClassifier is the general form: bootstrap samples and blending, wrapped around *any* estimator. A forest is bagging plus feature subsetting, specialised to trees. Bagging a linear model reaches almost nothing. Linear models are stable and there's no variance to mean away. Which is the clearest statement of when the technique applies at all.
RandomForestRegressor is the same method predicting the mean of each leaf rather than the majority class. All on this page transfers, including the extrapolation limit. This Bites harder in regression because a flat guess past the training range is more obviously wrong than a class label.
The pattern: bagging reduces variance and does nothing for bias. It helps unstable, low-bias models. Deep trees. And wastes time on stable ones.
Costs worth knowing before deploying one
A forest is cheap to fit and costly to keep.
Memory. each tree is stored in full. A forest of 500 unconstrained trees on a hundred thousand rows can run to hundreds of megabytes. The pickle is the same size. Constraining max_depth or min_samples_leaf shrinks it a lot at a small fit cost. This Is often the right trade for a thing that has to be shipped.
guess latency. each guess traverses each tree. That is fast per tree and adds up: 500 trees is 500 traversals. For batch scoring it's irrelevant. For a request-response service with a latency budget it can decide the design.
Fitting time scales with trees times rows times features. Parallelises almost just right with n_jobs=-1.
The lever for all three is the same. Fewer, shallower trees. Measuring how much fit that really costs, rather than given, often shows the curve is flat well below the default.
How many trees should I use? Enough that the score has flattened - often 100 to 500. Plot it once for your data and stop where the curve does.
Does a forest need scaling? No, for just the reason a single tree doesn't: splits are thresholds on one column.
Are the chances from predict_proba trustworthy? They're the proportion of trees voting for each class. This Ranks well and is systematically pushed towards the middle. Calibrate if you need them as numbers.
Can it handle missing values? Not in scikit-learn - impute first, or use HistGradientBoosting. This Handles them natively.
Why averaging works at all
The mechanism is worth one paragraph of math. It shows just when the technique helps and when it does nothing.
Blending n guesses whose errors are fully free cuts the variance by n. Blending n guesses whose errors are the same cuts it by nothing at all. The mean is the same as any one of them. Real ensembles sit between. The variance falls towards a floor set by how much the members share. No number of extra members goes below that floor.
That single fact shows each design choice in a random forest. Bootstrap sampling and feature subsetting exist to push the link between trees down. The link is what sets the floor. It shows why max_features=None performs worse. The trees become too alike and the floor rises. It shows why more trees stop helping. You approach the floor and then sit on it.
It also shows what bagging can't do. Blending drops variance and leaves bias untouched. If each tree is systematically wrong in the same direction. Is the mean. That is why a forest can't extrapolate. each tree is flat past the training range, that flatness is bias rather than variance. Blending the same flatness gives flatness.
Boosting attacks the other half of the problem, fitting each new model to what the previous ones got wrong. This Reduces bias and is why it needs more careful tuning to avoid the variance coming back.
Things to try with Random Forests
Watch the steadiness. The first editor's last two lines are the point: 14 rows against 6.
Push the trees up. In the second editor, add 1000 and confirm the score doesn't fall.
Compare the importances. The third editor's two columns disagree about a column that is pure noise.
Break the decorrelation. In the fifth editor, try max_features=1 and see whether so much range starts to hurt.
Questions people ask about Random Forests
Why is a forest better than one tree? Averaging many de-correlated trees cancels out much of their individual variance, so the ensemble is steadier.
How many trees do I need? A few hundred is usually plenty. More trees rarely hurt accuracy but add time and memory.
What is out-of-bag scoring? Each tree skips about a third of the rows, so those rows act as a free validation set. Enable it with oob_score=True.
Does a random forest need scaling? No, because trees split on thresholds and are unaffected by feature scale.
Can a forest still overfit? Yes, though less than one tree. Limiting depth and leaf size and using max_features helps.
Are forest importances reliable? Impurity-based ones favour high-cardinality features. Permutation importance on a held-out set is more honest.
Where this leaves you
Many on purpose different trees, averaged. More trees never hurt, max_features is the dial that matters, out-of-bag scoring is free, the built-in importances are biased towards many-valued columns. The whole thing still can't see past the edge of its training data.
Check yourself
0 of 4
Answer without scrolling back up.
What are the two sources of randomness in a random forest?
Bootstrapping alone leaves the trees too similar when one feature dominates; restricting the features at each split is what forces genuine diversity.
Can raising n_estimators cause overfitting?
It is the one complexity-shaped parameter with no trade-off. The cost is time and memory, not generalisation.
Why is feature_importances_ misleading?
In the editor a pure-noise column with 400 distinct values scored higher than another noise column. Permutation importance gave both exactly zero.
What does a random forest predict beyond its training range?
It predicted 55.3 for both x=25 and x=40 on a perfect straight line. Averaging does not fix what every member shares.
Cheat sheet
Random Forests
Combine hundreds of decision trees into a random forest that averages out each tree's instability and resists overfitting.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.