Gradient Boosting
Add one small tree at a time, each fitted to what the previous ones got wrong. Watch the residuals shrink toward zero.
Boosting
depth 1 = a stump
The Ensemble So Far
1 treeResiduals — What the Next Tree Will Fit
Each bar is one point's error. The next tree is trained on these, not on the original labels.
Error
Error Curve
Overview
Gradient boosting builds an ensemble sequentially: each new, usually shallow tree is fitted to reduce the current model’s loss. This can produce strong predictions, but learning rate, tree count, and depth must be tuned together to manage overfitting and training cost.
The sequential fit
- Start with a constant prediction — the mean of the target. It is a bad model, but it is a start.
- Compute the residual for every training point: how far off the current prediction is.
- Fit a small tree to the residuals.
- Add a fraction of that tree's predictions to the current model. The fraction is the learning rate.
- Repeat.
This residual description is exact for squared-error regression, which the visualizer demonstrates. More generally, gradient boosting fits each new learner to the negative gradient of the chosen loss, so the target depends on the loss function.
The residual panel above is step 2 made visible. Watch it after each tree: the bars shrink, and where they are still tall is exactly where the next tree will spend its capacity.
Exploration guide
- Watch it build. Press Reset to Zero Trees, then click Add One Tree a few times. Each stump adds a single step to the prediction and flattens one part of the residual plot. After three or four trees the shape of the underlying curve is already visible.
- Let it run. Click Add Trees One by One and watch the step function converge on the true curve while both error curves fall.
Boosting versus bagging
| Random Forest | Gradient Boosting | |
|---|---|---|
| Trees are built | independently, so training can be parallelized | in sequence, each depending on the current ensemble |
| Typical trees | often deeper | often shallow |
| Combined by | averaging or voting | adding each learner’s contribution |
| Main effect | often reduces variance | can reduce bias and variance |
| Tuning | usually forgiving | learning rate, depth, and tree count interact |
Both can perform well on tabular data. A random forest is a useful baseline; boosting may improve the fit but generally needs careful validation and tuning.
The learning rate and tree count are one knob
Reducing the learning rate often requires more trees to reach a similar training fit, but the trade-off is not exact. Tune learning rate and tree count together using validation data or early stopping; choose tree depth and other regularization settings for the task rather than relying on a universal value.
Traps worth knowing
- Judging by training error. It falls in effect forever. If you are not watching a validation curve you have no signal at all about when to stop.
- Deep trees. Greater depth lets each learner fit more interactions and can overfit. Choose depth using validation data.
- Expecting parallel training. Tree n cannot start until tree n−1 has finished, by construction. Libraries parallelise the work inside a single tree, not across trees, so boosting is slower to train than a forest of the same size.
- Ignoring outliers under squared error. Residuals from a wild point stay large and every subsequent tree keeps chasing them. Huber or absolute-error losses exist for exactly this.
When to use it, and when not to
Gradient boosting is a strong candidate for tabular data, but it is not best for every dataset. Compare it with a simple baseline and other suitable models using a validation design that reflects deployment.
It is usually not the first choice for raw images, audio, or text, where specialized representations are important. Tree ensembles are also hard to interpret directly and do not extrapolate smoothly beyond observed feature ranges. Explainability tools can help inspect predictions, but they do not turn feature attribution into causal evidence.
Further reading
For implementation details, see the Ensemble methods. Continue with random forests to connect this topic to a related part of the workflow.