Two failures that need opposite treatments, and one number that tells you which one you have.
Overview
The two failures
Overfitting is learning the training data rather than the pattern in it. The model captures noise, coincidences and details specific to the rows it was shown, none of which recur. Training score high, test score much lower.
Underfitting is the opposite: the model is too simple, or too constrained, to represent the structure that is there. It does badly on the training data and equally badly on the test data.
Both produce a disappointing test score, which is why a single number cannot distinguish them — and why the responses are opposite. Overfitting needs a simpler model, more regularisation, or more data. Underfitting needs a more flexible model, better features, or less regularisation. Applying the wrong remedy makes things worse, and guessing is a coin flip.
Worth knowing
Overfitting: training score high, test score much lower. The model learned the training rows rather than the pattern.
Underfitting: both scores low. The model cannot capture the structure even on data it has seen.
The gap between train and test is the diagnostic - which is why return_train_score=True is worth the extra computation.
validation_curve sweeps one hyperparameter; learning_curve sweeps the amount of data.
More data helps overfitting and does nothing for underfitting - the learning curve tells you which case you are in.
Every constraint trades training accuracy for generalisation, and the trade does not always pay.
Overfitting and Underfitting: A Practical Guide
Two failures that look similar in a single number and need opposite treatments. Telling them apart takes one extra column.
The gap opening up
One model, six complexities. Watch the training score reach 1.0 and the test score stop improving.
example_01.pyscikit-learn
Output
validation_curve does the sweep for you
The same picture, cross-validated, in one call.
example_02.pyscikit-learn
Output
Would more data help?
learning_curve answers the question that otherwise gets guessed at.
example_03.pyscikit-learn
Output
The question people guess at, and learning_curve answers it directly by fitting at several training-set sizes and reporting both scores.
Two shapes matter. A large gap that is still narrowing as the training set grows means more data would help — the model has capacity it cannot yet use reliably. Two curves that have converged means more data will not help; the model has extracted what it can, and improving requires a different model or better features.
The third editor shows the converged case: at 80 training samples the gap is 0.900 against 0.798, and by 1600 it is 0.844 against 0.838. Collecting another thousand rows would be wasted effort, and knowing that before commissioning the collection is worth a great deal.
The training score falling as the training set grows is normal and often surprises people. Fitting 80 points is easy; fitting 1600 is not.
Both failures, on one dataset
The truth is a parabola. Degree 1 cannot bend, and degree 15 bends to every point it was shown.
example_04.pyscikit-learn
Output
Variance, measured
How much the model changes when the training data changes - which is what averaging many models fixes.
example_05.pyscikit-learn
Output
Three constraints, two of which help
Every one of them lowers the training score. That is the trade, and it does not always pay.
example_06.pyscikit-learn
Output
The gap is the diagnostic
Compare the training score with the test score.
Both high, small gap. Working. Nothing to fix.
Train high, test much lower. Overfitting. The size of the gap is roughly how much of the training performance was memorisation.
Both low, small gap. Underfitting. The model is not capturing what is there.
Train lower than test. Unusual. Legitimate when regularisation or dropout is active during training but not scoring; otherwise a sign the folds are not comparable.
cross_validate(..., return_train_score=True) gives you both columns. It costs an extra scoring pass per fold and it is the single most informative argument in the model-selection module, because it converts "the score is disappointing" into "the score is disappointing *for this reason*".
The first editor shows the gap opening as a tree is allowed to grow: +0.070 at depth 1, +0.193 unconstrained, with the test score peaking in the middle and then going nowhere while the training score climbs to a perfect 1.000.
Complexity is the dial
Every model has something that controls how much structure it can represent.
Tree depth, and the minimum samples in a leaf. The number of neighbours in k-NN — fewer neighbours is *more* complex, which is the one that runs backwards. The degree of a polynomial expansion. The C in an SVM or a logistic regression, where larger is less regularised. The number of estimators and the learning rate in a boosting model.
Turn the dial up and the training score rises monotonically; the test score rises, peaks, and falls. The peak is what you are looking for, and validation_curve finds it by sweeping one parameter with cross-validation at each setting.
The fourth editor shows both ends on the same data. The truth is a parabola. Degree 1 scores 0.008 on training — it cannot bend at all. Degree 2 is right. Degree 15 scores 0.894 on training and −17.888 on test, which is far worse than predicting the mean, and is what memorising sixty points with sixteen coefficients produces.
Bias, variance, and why averaging works
The two failures have formal names. Bias is error from a model too rigid to represent the truth — underfitting. Variance is error from a model so flexible that it changes substantially when the training data changes — overfitting.
Variance is measurable, and the fifth editor measures it: refitting a single deep decision tree on five different halves of the data moves one prediction by 270. The same exercise with a fifty-tree random forest moves it by 48.
That reduction is the entire idea behind ensembles. Each tree is high-variance; the errors are partly independent; averaging cancels them. It is why a random forest is almost always better than the tree inside it, and why bagging works on unstable models and does nothing for stable ones.
The classic framing is a trade-off, and modern practice complicates it — very large models sometimes improve again past the point of interpolating the training data. For everything in this track, the trade-off framing holds: reducing one usually increases the other, and the useful setting is where their sum is smallest.
The remedies, and that they can fail
For overfitting: constrain the model, add regularisation, reduce the number of features, get more data, or use an ensemble to average the variance away.
For underfitting: use a more flexible model, add features or interactions, reduce regularisation, or train for longer if the fit was cut short.
The last editor is worth dwelling on because it shows the trade honestly. Three constraints applied to an overfitting tree: all three lower the training score, as they must. max_depth=3 and ccp_alpha=0.01 raise the test score. min_samples_leaf=20 lowers it, from 0.826 to 0.806 — it constrained the model past the useful point and turned overfitting into underfitting.
So "add regularisation" is a direction rather than a fix, and the amount has to be found by cross-validation rather than chosen. That is precisely what the tuning module is for.
Overfitting the validation set
There is a third failure, and it is the one that catches experienced people rather than beginners.
Every time you look at a validation score and change something in response, you use that score to make a decision. Do it once and the effect is negligible. Do it two hundred times — trying models, sweeping hyperparameters, adding and removing features, each time keeping what scored better — and the winner has been selected for performing well on those particular folds, including on their noise.
The result is a cross-validated score that is optimistic even though every individual fit was honest. Nothing in the procedure was wrong; the selection was the leak. It is the same mechanism as picking the best of twenty test scores, just slower and less obviously a mistake.
Three defences. Keep a final test set that is touched exactly once, at the end, after every decision has been made. Prefer a small number of deliberate comparisons to a long undirected search. And when the difference between two candidates is smaller than the standard deviation across folds, treat them as tied rather than picking a winner — because at that point you are choosing noise.
Nested cross-validation is the rigorous version: an inner loop tunes and an outer loop scores the whole tuning procedure. It costs a multiple of the fits and is the right answer when the number being reported matters.
Signals from the model itself
Besides the gap, three things hint at overfitting before you have measured anything.
Coefficients that are enormous. A linear model whose coefficients run to thousands is balancing large positive and negative terms that nearly cancel — a fit that is precariously tuned to the training rows. Regularisation exists to prevent exactly this, and the size of the coefficients is a reasonable proxy for how much is needed.
A tree with as many leaves as samples. If tree.get_n_leaves() is close to the number of training rows, every leaf is one or two samples and the model is a lookup table.
Scores that move a lot between folds. A high standard deviation across cross-validation folds means the model is sensitive to which rows it saw, which is variance by definition.
None of these is conclusive on its own. All three are cheap to check and worth a glance before the more expensive diagnosis.
Is a perfect training score always bad? Not by itself. An unconstrained tree reaches 1.0 by construction. It is bad when the test score is far below it.
Does more data ever cause overfitting? No - more data only ever helps generalisation. More *features* on the same rows is what makes it worse.
Which comes first, tuning or feature work? Usually features. A learning curve that has converged says the model has extracted what it can, and no amount of tuning changes that.
What is ccp_alpha? Cost-complexity pruning: it grows the tree fully and then removes branches whose contribution is not worth their complexity. Often better than capping the depth, because it prunes where pruning helps rather than everywhere.
Which dial belongs to which model
The complexity control has a different name in every estimator, which makes the general idea harder to see than it should be. The mapping is worth having in one place.
Decision trees.max_depth caps how many questions can be asked. min_samples_leaf and min_samples_split stop the tree splitting groups that are already small — often a better control than depth, because it responds to the data rather than to a fixed count. ccp_alpha prunes after growing.
Random forests. The same tree parameters, plus n_estimators, which is the one that does *not* overfit — more trees only ever helps, at the cost of time. max_features controls how decorrelated the trees are and is the main variance dial.
Gradient boosting.learning_rate and n_estimators trade against each other: a smaller rate needs more trees and generalises better. max_depth is kept small, typically 3 to 6, because the ensemble supplies the flexibility.
Linear models.alpha in Ridge and Lasso, where larger is more regularised. C in LogisticRegression and the SVMs, where larger is *less* regularised — the one inversion in the library, and the reason to check which you are holding.
k-NN.n_neighbors, where more neighbours is a simpler model. This runs backwards from most intuitions about "bigger numbers mean more".
Polynomial features.degree, which increases complexity faster than anything else on this list — degree 3 on ten features is 286 columns.
The pattern underneath: every one of these controls how much of the training data the model is permitted to represent exactly.
Can a model overfit with only a few features? Yes. Degree-15 polynomial features on one input column scored -17.888 on test in the fourth editor. What matters is flexibility relative to the number of rows, not the raw column count.
Do I fix underfitting by removing regularisation entirely? Try it to confirm the diagnosis, then find the amount by cross-validation. Zero is rarely the best value.
Things to try
Watch the gap grow. The first editor's last column goes from +0.070 to +0.193 while the test score stops improving after depth 3.
Find the peak. In the second editor, note that the training score keeps rising after the test score has stopped — that divergence is the definition.
Ask whether data would help. In the third editor, cut n_samples to 300 and see whether the curves have converged by then.
Over-constrain deliberately. In the last editor, try min_samples_leaf=100 and watch both scores fall together.
Where this leaves you
One extra column tells you which failure you have. validation_curve finds the right complexity, learning_curve says whether more data would help, and every constraint is a trade that has to be measured rather than assumed.
Check yourself
0 of 4
Answer without scrolling back up.
Training accuracy 1.00, test accuracy 0.72. What is happening?
A large gap with a perfect training score means the model learned the rows rather than the pattern. Both scores being low would be underfitting.
What does return_train_score=True buy you?
The two failures look identical in a test score alone and need opposite remedies. The train column is what tells them apart.
The learning curve's train and test scores have converged. What does that mean?
Converged curves mean the model has extracted what it can from data of this kind. Improving needs a different model or better features.
Why does a random forest have lower variance than one deep tree?
Measured in the editor: one tree's prediction moved by 270 across refits, the forest's by 48. That averaging is the point of bagging.
Cheat sheet
Overfitting and Underfitting
Overfitting is learning the training data rather than the pattern in it. The model captures noise, coincidences and details specific to the rows it was shown, none of which recur. Training score high, test score much lower.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.