Learning Curves

Two lines that tell you whether to collect more data or change the model — and they give different answers.

Overview

What is plotted

Train the model on 40 examples, record the error on those examples and on a held-out validation set. Repeat with 80, 120, and so on. Plot both against training-set size.

Grey is training error, orange is validation error. The shape of the pair is the diagnosis, and the diagnosis is more useful than either number alone.

Learning Curves

This module needs JavaScript: the numbers are computed in the page rather than recorded.

Worth knowing

Plot training and validation error against the number of training examples. The shape is the diagnosis.
Both high and converged: bias. The model is too simple, and more data will not help.
Low training error with a large gap: variance. More data or less capacity will help.
A validation curve that has flattened is the honest answer to 'should we collect more data'. It is no.

Learning Curves

The plot that answers whether to collect more data, and saves you from collecting it pointlessly.

Underfitting

Set capacity to 1 or 2. Both curves rise quickly to a plateau, sit close together, and stay high.

Training error is high, which is the tell. The model cannot fit the data it has already seen, so nothing about unseen data is the problem. This is bias: the model is too simple for the structure present.

The important consequence is negative. More data will not help. The curves have already converged; adding examples moves neither. Money spent on collection here is wasted, and this plot is the cheapest way to find that out before spending it.

What helps instead: more capacity, better features, less regularisation, a different model class.

Overfitting

Set capacity to 8 or 9. Training error drops close to zero, validation error stays well above it, and a wide gap opens between them.

The model fits its training data nearly perfectly and generalises poorly. This is variance: capacity is being spent memorising noise.

Here more data *does* help, and the curve says so — the validation line is still falling at the right-hand edge. Every additional example makes the noise harder to memorise. Other options: reduce capacity, add regularisation, augment the data, or ensemble.

About right

Set capacity to 4 or 5. The curves converge to a small gap at a low error, and both have flattened.

The readout says as much. There is no obvious bias and no obvious variance, and the flattening is the answer to "should we collect more data": no, not for this model. Improvement would have to come from features or from a different model class, not from volume.

The summary

Training errorGapDiagnosisDo
HighSmallBiasmore capacity, better features
LowLargeVariancemore data, regularisation, less capacity
LowSmallFinefeatures or a different model
HighLargeSomething is wrongcheck the split and the labels

That last row is worth keeping. High training error *and* a large gap should not happen from bias or variance alone, and usually means a bug — leakage in reverse, mislabelled data, or a validation set drawn from a different distribution.

Where the curve is still falling

The single most valuable thing here is being able to answer "will more data help" with evidence.

If the validation curve is still descending at your current dataset size, collecting more will improve the model, and the slope gives a rough sense of how much. If it has been flat for the last few points, more data buys nothing, and the effort belongs elsewhere.

That question is otherwise answered by intuition, and it usually involves real money.

Practical notes

The curves are noisy at small sizes, because a model trained on 40 examples depends heavily on which 40. Average over several splits, and plot the spread if you can.

Use the same validation set at every size. Growing it alongside the training set confounds two changes.

And do not confuse this with a validation curve, which plots error against a hyperparameter at fixed data size. Both are useful and they answer different questions: learning curves ask about data, validation curves ask about a setting.

Where it goes wrong

Reading a single run. Small-sample noise looks like structure.

Plotting accuracy on imbalanced data. The curves will be flat and uninformative. Plot the metric you actually care about.

Concluding 'more data' from a gap alone. Check the validation curve is still falling. A converged gap means variance you cannot fix with volume.

Check yourself

0 of 3

Answer without scrolling back up.

  1. Training error is high and the two curves have converged. What does that mean?

  2. What tells you whether collecting more data will help?

  3. High training error AND a large gap suggests what?

Cheat sheet

Learning Curves

Train the model on 40 examples, record the error on those examples and on a held-out validation set. Repeat with 80, 120, and so on. Plot both against training-set size.

MACHINE LEARNING · vizlearn.in/machine_learning/learning_curves.html

About the author

Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.