Two failures that need opposite treatments, and one number that tells you which one you have.
Overview
The two failures
Every model sits somewhere on a spectrum between too simple and too complex. Too simple means it cannot capture the pattern in the data. Too complex means it captures the pattern plus the noise. The first is underfitting. The second is overfitting. Telling them apart is the most important diagnostic skill in machine learning.
The fix for overfitting is more constraint: lower depth, stronger regularisation, more data, fewer features. The fix for underfitting is less constraint: higher depth, weaker regularisation, more features, a more flexible model family. Applying the wrong fix makes things worse. Adding regularisation to an underfitting model pushes it further from the pattern.
The diagnostic is the gap between the training score and the test score. A large gap means overfitting. A small gap with a low score means underfitting. This page shows both in practice, with learning curves that make the gap visible as a function of training-set size.
Understanding this trade-off also tells you when to stop tuning. If the training score is high and the test score is close to it, the model is doing as well as it can on this data. More tuning will not help. More or better features might.
Overfitting is learning the training data rather than the pattern in it. The model captures noise, coincidences and details exact to the rows it was shown, none of which recur. Training score high, test score much lower.
Underfitting is the opposite: the model is too simple, or too constrained, to represent the structure that is there. It does badly on the training data and equally badly on the test data.
Both give a disappointing test score. This Is why a single number can't set apart them. And why the responses are opposite. Overfitting needs a simpler model, more smoothing, or more data. Underfitting needs a more flexible model, better features, or less smoothing. Applying the wrong remedy makes things worse. Guessing is a coin flip.
Worth knowing
Overfitting: training score high, test score much lower. The model learned the training rows rather than the pattern.
Underfitting: both scores low. The model cannot capture the structure even on data it has seen.
The gap between train and test is the diagnostic - which is why return_train_score=True is worth the extra computation.
validation_curve sweeps one hyperparameter; learning_curve sweeps the amount of data.
More data helps overfitting and does nothing for underfitting - the learning curve tells you which case you are in.
Every constraint trades training accuracy for generalisation, and the trade does not always pay.
Overfitting and Underfitting: A Practical Guide
Two failures that look similar in a single number and need opposite treatments. Telling them apart takes one extra column.
The gap opening up
One model, six complexities. Watch the training score reach 1.0 and the test score stop improving.
example_01.pyscikit-learn
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=500, n_features=20, n_informative=5,
random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
print(" depth train test gap")
for depth in (1, 2, 3, 5, 10, None):
t = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_tr, y_tr)
tr, te = t.score(X_tr, y_tr), t.score(X_te, y_te)
print("%6s %.3f %.3f %+.3f" % (depth, tr, te, tr - te))
Output
validation_curve does the sweep for you
The same picture, cross-validated, in one call.
example_02.pyscikit-learn
from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
import numpy as np
X, y = make_classification(n_samples=500, n_features=20, n_informative=5,
random_state=0)
depths = [1, 2, 3, 5, 8, 12, 20]
train, test = validation_curve(
DecisionTreeClassifier(random_state=0), X, y,
param_name="max_depth", param_range=depths, cv=5)
print(" depth train test")
for d, tr, te in zip(depths, train.mean(axis=1), test.mean(axis=1)):
print("%6d %.3f %.3f" % (d, tr, te))
print()
print("best depth by cross-validation:", depths[int(np.argmax(test.mean(axis=1)))])
Output
Would more data help?
learning_curve answers the question that otherwise gets guessed at.
example_03.pyscikit-learn
from sklearn.model_selection import learning_curve
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=2000, n_features=20, n_informative=5,
random_state=0)
sizes, train, test = learning_curve(
LogisticRegression(max_iter=1000), X, y,
train_sizes=[0.05, 0.1, 0.25, 0.5, 1.0], cv=5)
print(" n_train train test")
for n, tr, te in zip(sizes, train.mean(axis=1), test.mean(axis=1)):
print("%9d %.3f %.3f" % (n, tr, te))
print()
print("the two curves converging means more data will not help much;")
print("a wide gap that is still closing means it would.")
Output
The question people guess at, and learning_curve answers it right by fitting at a few training-set sizes and reporting both scores.
Two shapes matter. A large gap that is still narrowing as the training set grows means more data would help. The model has room it can't yet use well. Two curves that have settled means more data won't help. The model has extracted what it can. Raising needs a different model or better features.
The third editor shows the settled case: at 80 training samples the gap is 0.900 against 0.798. By 1600 it's 0.844 against 0.838. Collecting another thousand rows would be wasted effort. Knowing that before commissioning the collection is worth a great deal.
The training score falling as the training set grows is normal and often surprises people. Fitting 80 points is easy. Fitting 1600 isn't.
Both failures, on one dataset
The truth is a parabola. Degree 1 cannot bend, and degree 15 bends to every point it was shown.
example_04.pyscikit-learn
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
import numpy as np
rng = np.random.RandomState(0)
x = np.sort(rng.uniform(-3, 3, 60)).reshape(-1, 1)
y = x.ravel() ** 2 + rng.normal(scale=1.5, size=60)
x_tr, x_te, y_tr, y_te = train_test_split(x, y, test_size=0.4, random_state=0)
print("degree train R2 test R2")
for deg in (1, 2, 5, 15):
m = make_pipeline(PolynomialFeatures(deg), LinearRegression()).fit(x_tr, y_tr)
print("%6d %8.3f %7.3f" % (deg, m.score(x_tr, y_tr), m.score(x_te, y_te)))
print()
print("degree 1 underfits, degree 2 is the truth, degree 15 memorises.")
Output
Variance, measured
How much the model changes when the training data changes - which is what averaging many models fixes.
example_05.pyscikit-learn
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
import numpy as np
X, y = make_regression(n_samples=300, n_features=10, noise=20.0, random_state=0)
preds = []
for seed in range(5):
X_tr, _, y_tr, _ = train_test_split(X, y, test_size=0.5, random_state=seed)
preds.append(DecisionTreeRegressor(random_state=0).fit(X_tr, y_tr).predict(X[:5]))
print("one deep tree, 5 refits - spread of the first prediction: %.1f"
% np.ptp([p[0] for p in preds]))
preds = []
for seed in range(5):
X_tr, _, y_tr, _ = train_test_split(X, y, test_size=0.5, random_state=seed)
preds.append(RandomForestRegressor(n_estimators=50, random_state=0)
.fit(X_tr, y_tr).predict(X[:5]))
print("a forest of 50, same 5 refits - spread: %.1f"
% np.ptp([p[0] for p in preds]))
Output
Three constraints, two of which help
Every one of them lowers the training score. That is the trade, and it does not always pay.
example_06.pyscikit-learn
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate
X, y = make_classification(n_samples=500, n_features=20, n_informative=5,
random_state=0)
settings = [("unconstrained", {}),
("max_depth=3", {"max_depth": 3}),
("min_samples_leaf=20", {"min_samples_leaf": 20}),
("ccp_alpha=0.01", {"ccp_alpha": 0.01})]
print("%-22s %7s %7s %7s" % ("", "train", "test", "gap"))
for label, kw in settings:
out = cross_validate(DecisionTreeClassifier(random_state=0, **kw), X, y,
cv=5, return_train_score=True)
tr, te = out["train_score"].mean(), out["test_score"].mean()
print("%-22s %7.3f %7.3f %+7.3f" % (label, tr, te, tr - te))
print()
print("every constraint lowers the training score. Two of them")
print("raise the test score, which is the trade being made.")
Output
The gap is the diagnostic
Compare the training score with the test score.
Both high, small gap. Working. Nothing to fix.
Train high, test much lower. Overfitting. The size of the gap is roughly how much of the training speed was memorisation.
Both low, small gap. Underfitting. The model isn't capturing what is there.
Train lower than test. Unusual. Legitimate when smoothing or dropout is active during training but not scoring. Or else a sign the folds aren't close.
cross_validate(..., return_train_score=True) gives you both columns. It costs an extra scoring pass per fold and it's the single most helpful arg in the model-pick module. It converts "the score is disappointing" into "the score is disappointing *for this reason*".
The first editor shows the gap opening as a tree is allowed to grow: +0.070 at depth 1, +0.193 unconstrained, with the test score peaking in the middle and then going nowhere while the training score climbs to a perfect 1.000.
Complexity is the dial
each model has a thing that controls how much structure it can represent.
Tree depth, and the min samples in a leaf. The number of neighbours in k-NN. Fewer neighbours is *more* complex. This Is the one that runs backwards. The degree of a polynomial expansion. The C in an SVM or a logistic regression. Here Larger is less smoothed. The number of estimators and the learning rate in a boosting model.
Turn the dial up and the training score rises monotonically. The test score rises, peaks. Falls. The peak is what you're looking for. validation_curve finds it by sweeping one param with cross-validation at each setting.
The fourth editor shows both ends on the same data. The truth is a parabola. Degree 1 scores 0.008 on training. It can't bend at all. Degree 2 is right. Degree 15 scores 0.894 on training and −17.888 on test. This Is far worse than predicting the mean. Is what storing sixty points with sixteen coeffs makes.
Bias, variance, and why averaging works
The two failures have formal names. Bias is error from a model too rigid to represent the truth. Underfitting. Variance is error from a model so flexible that it changes a lot when the training data changes — overfitting.
Variance is measurable, and the fifth editor measures it. Refitting a single deep choice tree on five different halves of the data moves one guess by 270. The same exercise with a fifty-tree random forest moves it by 48.
That reduction is the entire idea behind ensembles. Each tree is high-variance. The errors are partly free. Blending cancels them. It's why a random forest is almost always better than the tree inside it. Why bagging works on unstable models and does nothing for stable ones.
The classic framing is a trade-off. Modern practice complicates it. Very large models at times improve again past the point of interpolating the training data. For all in this track, the trade-off framing holds. Reducing one often grows the other, and the handy setting is where their sum is smallest.
The remedies, and that they can fail
For overfitting: constrain the model, add smoothing, reduce the number of features, get more data, or use an ensemble to mean the variance away.
For underfitting: use a more flexible model, add features or interplays, reduce smoothing, or train for longer if the fit was cut short.
The last editor is worth dwelling on because it shows the trade honestly. Three constraints applied to an overfitting tree. All three lower the training score, as they must. max_depth=3 and ccp_alpha=0.01 raise the test score. min_samples_leaf=20 lowers it, from 0.826 to 0.806. It constrained the model past the handy point and turned overfitting into underfitting.
So "add smoothing" is a direction rather than a fix. The amount has to be found by cross-validation rather than chosen. That is just what the tuning module is for.
Overfitting the validation set
There's a third failure, and it's the one that catches seen people rather than beginners.
each time you look at a validation score and change a thing in response, you use that score to make a choice. Do it once and the effect is negligible. Do it two hundred times. Trying models, sweeping settings, adding and dropping features, each time keeping what scored better. And the winner has been selected for performing well on those given folds, including on their noise.
The result is a cross-validated score that is optimistic though each single fit was honest. Nothing in the method was wrong. The pick was the leak. It's the same mechanism as picking the best of twenty test scores, just slower and less obviously a mistake.
Three defences. Keep a final test set that is touched just once, at the end, after each choice has been made. Prefer a small number of deliberate checks to a long undirected search. And when the difference between two picks is smaller than the standard deviation across folds, treat them as tied rather than picking a winner. Because at that point you're choosing noise.
Nested cross-validation is the rigorous version: an inner loop tunes and an outer loop scores the whole tuning method. It costs a multiple of the fits and is the right answer when the number being reported matters.
Signals from the model itself
Besides the gap, three things hint at overfitting before you've measured any of it.
coeffs that are huge. A linear model whose coeffs run to thousands is weighing large plus and minus terms that nearly cancel. A fit that is precariously tuned to the training rows. Smoothing exists to prevent just this. The size of the coeffs is a reasonable proxy for how much is needed.
A tree with as many leaves as samples. If tree.get_n_leaves() is close to the number of training rows, each leaf is one or two samples and the model is a lookup table.
Scores that move a lot between folds. A high standard deviation across cross-validation folds means the model is sensitive to which rows it saw. This Is variance by meaning.
None of these is conclusive on its own. All three are cheap to check and worth a glance before the more costly diagnosis.
Is a perfect training score always bad? Not by itself. An unconstrained tree reaches 1.0 by build. It's bad when the test score is far below it.
Does more data ever cause overfitting? No - more data only ever helps generalisation. More *features* on the same rows is what makes it worse.
Which comes first, tuning or feature work? often features. A learning curve that has settled says the model has extracted what it can. No amount of tuning changes that.
What is ccp_alpha? Cost-cost pruning: it grows the tree fully and then drops branches whose part isn't worth their cost. Often better than capping the depth. It prunes where pruning helps rather than all over.
Which dial belongs to which model
The cost control has a different name in each estimator. This Makes the general idea harder to see than it should be. The mapping is worth having in one place.
choice trees.max_depth caps how many questions can be asked. min_samples_leaf and min_samples_split stop the tree splitting groups that are already small. Often a better control than depth. It reacts to the data rather than to a fixed count. ccp_alpha prunes after growing.
Random forests. The same tree params, plus n_estimators. This Is the one that does *not* overfit. More trees only ever helps, at the cost of time. max_features controls how decorrelated the trees are and is the main variance dial.
Grad boosting.learning_rate and n_estimators trade against each other. A smaller rate needs more trees and generalises better. max_depth is kept small, often 3 to 6. The ensemble supplies the flexibility.
Linear models.alpha in Ridge and Lasso. Here Larger is more smoothed. C in LogisticRegression and the SVMs. Here Larger is *less* smoothed. The one inversion in the library. The reason to check which you're holding.
k-NN.n_neighbors, where more neighbours is a simpler model. This runs backwards from most intuitions about "bigger numbers mean more".
Polynomial features.degree, which grows cost faster than any of it else on this list. Degree 3 on ten features is 286 columns.
The pattern underneath: each one of these controls how much of the training data the model is permitted to represent just.
Can a model overfit with only a few features? Yes. Degree-15 polynomial features on one input column scored -17.888 on test in the fourth editor. What matters is flexibility scaled to the number of rows, not the raw column count.
Do I fix underfitting by dropping smoothing fully? Try it to confirm the diagnosis, then find the amount by cross-validation. Zero is rarely the best value.
Things to try with Overfitting and Underfitting
Watch the gap grow. The first editor's last column goes from +0.070 to +0.193 while the test score stops raising after depth 3.
Find the peak. In the second editor, note that the training score keeps rising after the test score has stopped. That divergence is the meaning.
Ask whether data would help. In the third editor, cut n_samples to 300 and see whether the curves have settled by then.
Over-constrain on purpose. In the last editor, try min_samples_leaf=100 and watch both scores fall together.
Questions people ask about Overfitting and Underfitting
How can I tell overfitting from underfitting? Compare scores. High training and low validation scores mean overfitting, and low scores on both mean underfitting.
Will more data fix overfitting? Often, yes, if the model is flexible. It does not help an underfit model, which needs more capacity or better features.
What reduces overfitting? Simpler models, regularisation, fewer features, early stopping or more training data. Pick whichever matches the model.
What fixes underfitting? A more flexible model, added or better features, or weaker regularisation.
What is a learning curve? A plot of training and validation score against training set size. It shows whether more data would still help.
Can I overfit the validation set? Yes. Trying many settings against one validation score picks noise, so keep a final test set untouched.
Where this leaves you
One extra column tells you which failure you've. validation_curve finds the right cost, learning_curve says whether more data would help. Each constraint is a trade that has to be measured rather than assumed.
Check yourself
0 of 4
Answer without scrolling back up.
Training accuracy 1.00, test accuracy 0.72. What is happening?
A large gap with a perfect training score means the model learned the rows rather than the pattern. Both scores being low would be underfitting.
What does return_train_score=True buy you?
The two failures look identical in a test score alone and need opposite remedies. The train column is what tells them apart.
The learning curve's train and test scores have converged. What does that mean?
Converged curves mean the model has extracted what it can from data of this kind. Improving needs a different model or better features.
Why does a random forest have lower variance than one deep tree?
Measured in the editor: one tree's prediction moved by 270 across refits, the forest's by 48. That averaging is the point of bagging.
Cheat sheet
Overfitting and Underfitting
Spot overfitting and underfitting from training and test scores, and apply the opposite fix each failure needs to close the gap.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.