Random Forest and Bagging
One deep tree memorises the noise. Add more trees, each trained on a different sample, and the jagged boundary averages into a smooth one.
Forest
Decision Boundary
1 treeShading is the share of trees voting for orange. Colour strength is the ensemble's confidence.
Accuracy
Diversity
press Regrow Forest to compare runs
How far test accuracy moves between regrows of the same settings. That swing is the variance bagging removes.
Overview
Random forests combine many decision trees so that individual quirks have less influence on the final prediction. The method works best when the trees are reasonably accurate but make different errors.
From one tree to a forest
A deep decision tree can fit its training data closely, including accidental patterns. Bagging (bootstrap aggregating) trains each tree on a bootstrap sample: a same-size sample drawn with replacement. A bootstrap sample contains about 63% of the distinct training rows on average; the remaining rows are out of bag for that tree.
For classification, the forest usually combines tree votes; for regression, it averages predictions. Averaging reduces variability when the trees do not all make the same mistakes. If every tree were identical, combining them would add little.
A random forest adds feature randomness: at each split, a tree considers only a random subset of candidate features. This can stop one strong feature from dominating every tree and helps decorrelate their errors. Row sampling and feature sampling create diversity; aggregation uses that diversity to stabilise predictions.
Out-of-bag evaluation
Each training row is left out of roughly 36.8% of bootstrap samples. For that row, the trees that did not train on it can produce an out-of-bag prediction. Aggregating these predictions gives an out-of-bag score without reserving a separate validation split.
Out-of-bag evaluation is a useful estimate, not a replacement for a final held-out test set. Its default score for a classifier is accuracy, which can be misleading for imbalanced labels. Pick a metric that matches the task, and use a separate test set for the final evaluation.
Watch the ensemble grow
- Set number of Trees to 1 and click Regrow Forest a few times. The fitted boundary and test score can vary with the sampled data.
- Increase number of Trees to 40 and regrow. The ensemble averages more trees, so its predictions and score spread should become more stable. More trees cost more time; they do not repair a poor feature set or biased training data.
The visualiser uses a small two-dimensional classification example. Real forests can have many features and trees, and their performance still needs validation on data representative of deployment.
When to use one
Random forests are a strong general-purpose baseline for tabular data. They can model nonlinear boundaries and feature interactions without scaling numeric columns. They are less suitable when a compact, directly interpretable model is required, or when extrapolation beyond the training range matters. A single tree is easier to inspect; boosted trees may perform better on some structured datasets but need their own tuning and validation.
Sources and next steps
See scikit-learn’s random forest documentation for parameters and out-of-bag scoring. Continue with decision trees to see the base learner, or cross-validation to compare models across splits.