Gradient Boosting
Add one small tree at a time, each fitted to what the previous ones got wrong. Watch the residuals shrink toward zero.
Boosting
depth 1 = a stump
The Ensemble So Far
1 treeResiduals — What the Next Tree Will Fit
Each bar is one point's error. The next tree is trained on these, not on the original labels.
Error
Error Curve
Gradient Boosting: A Practical Guide
The method that wins on tabular data, built from models too weak to be useful alone.
Quick Context
A random forest builds many strong trees independently and averages away their variance. Boosting does close to the opposite: it builds many deliberately weak trees in sequence, each one aimed squarely at the mistakes the previous ones left behind.
Nothing here votes. The trees are added together, and each new tree is a correction rather than an opinion.
The loop
- Start with a constant prediction — the mean of the target. It is a bad model, but it is a start.
- Compute the residual for every training point: how far off the current prediction is.
- Fit a small tree to those residuals, not to the original labels.
- Add a fraction of that tree's predictions to the running total. The fraction is the learning rate.
- Repeat.
The residual panel above is step 2 made visible. Watch it after each tree: the bars shrink, and where they are still tall is exactly where the next tree will spend its capacity.
Why "gradient"
For squared-error loss, the negative gradient of the loss with respect to the prediction is the residual. So fitting a tree to the residuals is the same as taking a gradient descent step in function space — each tree is one step downhill, and the learning rate is the step size, exactly as in ordinary gradient descent.
That framing is what makes the method general. Swap in a different loss and the "residual" becomes that loss's negative gradient instead, which is how the same algorithm handles classification, ranking and count data without changing shape.
Interactive Exploration Guide
- Watch it build. Press Reset to Zero Trees, then click Add One Tree a few times. Each stump adds a single step to the prediction and flattens one part of the residual plot. After three or four trees the shape of the underlying curve is already visible.
- Let it run. Click Add Trees One by One and watch the step function converge on the true curve while both error curves fall.
- Turn the rate up. Set the Learning Rate slider to 1. The fit lurches at the data and reaches its best around round 20 rather than round 80 — then stops improving and drifts slightly worse. With stumps the damage is mild; combine a rate of 1 with a Tree Depth of 4 and the model is overfitting from the very first round.
- Turn it down. Set the Learning Rate slider to 0.02. Progress is slow and the model needs far more trees, but the final fit is smoother and the test curve stays flat rather than turning up. This is the trade every boosting library is tuned around.
- Make the trees too strong. Set the Tree Depth slider to 5 with 120 trees. Deep trees fit the noise directly and test error rises sharply while train error keeps falling. Boosting wants weak learners.
- Remove the noise. Set the Noise slider to 0. Now more trees never hurt — there is nothing to memorise. Boosting overfits noise, not complexity as such.
Boosting versus bagging
| Random Forest | Gradient Boosting | |
|---|---|---|
| Trees are built | independently, in parallel | in sequence, each on the last one's errors |
| Individual trees are | deep and strong | shallow and weak |
| Combined by | voting or averaging | addition |
| Attacks | variance | bias |
| More trees | never hurts | eventually overfits |
| Tuning | forgiving | needs care |
That last row is the honest trade. Boosting usually wins on tabular data, and it will happily overfit if you let it. A forest is much harder to get wrong.
The learning rate and tree count are one knob
Halving the learning rate roughly doubles the number of trees you need to reach the same fit. Because of this they should never be tuned independently: the usual recipe is to fix a small learning rate (0.05 or 0.1), then add trees until held-out error stops improving, and stop there. That early stop is the main regulariser in practice, which is why every serious implementation has it built in.
What usually goes wrong
- Judging by training error. It falls essentially forever. If you are not watching a validation curve you have no signal at all about when to stop.
- Deep trees. Depth 3 to 6 is the normal range. Going deeper turns each correction into a memorisation step and the ensemble overfits within a few dozen rounds.
- Expecting parallel training. Tree n cannot start until tree n−1 has finished, by construction. Libraries parallelise the work inside a single tree, not across trees, so boosting is slower to train than a forest of the same size.
- Ignoring outliers under squared error. Residuals from a wild point stay large and every subsequent tree keeps chasing them. Huber or absolute-error losses exist for exactly this.
Key Takeaway
Gradient boosting fits each new small tree to the residuals left by everything built so far, then adds a fraction of it to the running prediction — which is gradient descent performed in function space, with the learning rate as the step size. Because it attacks bias by sequential correction rather than variance by averaging, its trees must be weak and its round count must be limited, and unlike a random forest it will overfit if you keep going. Fix a small learning rate, add trees while held-out error improves, and stop the moment it does not.