Simulate and understand how Loss and Accuracy evolve over time. Tune the architecture mathematically to observe underfitting, ideal fits, and overfitting.
A training curve plots loss (or accuracy) against epochs for both the training set and the validation set. The training curve tells you how well the net fits the data it has seen. The validation curve tells you how well it generalises to data it has not. The gap between them is the single most useful diagnostic in a training run.
Without a validation curve, a falling training loss looks like progress. With one, you can tell whether the net is really learning the pattern or simply memorising the cases. Overfitting shows up as the gap widening. Underfitting shows up as both curves staying high. Early stopping uses the validation curve to decide when to stop training before overfitting sets in.
This page shows you how to read each shape the two curves can form, what it means for your net. And what action to take next. Each shape maps to a clear diagnosis and a clear next step:
The visualisation above lets you drag the epoch slider and watch both curves respond in real time. So each of these shapes is something you can build and inspect right.
Simulate and understand how Loss and Accuracy evolve over time.
Two lines tell you almost everything about a training run. Learning to read the gap between them is the fastest diagnostic skill in machine learning.
A training curve plots loss (or accuracy) against epochs for both the training set and the validation set. The training curve tells you how well the net fits the data it has seen. The validation curve tells you how well it generalises to data it has not. The gap between them is the single most useful diagnostic in a training run.
Without a validation curve, a falling training loss looks like progress. With one, you can tell whether the net is really learning the pattern or simply memorising the cases. Overfitting shows up as the gap widening. Underfitting shows up as both curves staying high. Early stopping uses the validation curve to decide when to stop training before overfitting sets in.
This page shows you how to read each shape the two curves can form, what it means for your net. And what action to take next. Each shape maps to a clear diagnosis and a clear next step:
The visualisation above lets you drag the epoch slider and watch both curves respond in real time. So each of these shapes is something you can build and inspect right.
Training loss 0.08, validation loss 0.34. The gap of 0.26 is the net's memorisation of the training set — that is overfitting, no matter how good 0.08 looks in isolation.
Training loss 0.31, validation loss 0.33. A tiny gap, but both are high. The net is not overfitting. It simply is not good enough yet. These two cases need opposite replies. It is just why looking at one curve is not enough.
The two curves matter less individually than the distance between them. And how that distance changes over time.
A gap that stays narrow while both curves fall is a healthy run. A gap that opens steadily — training continuing down while validation flattens or rises — is overfitting. And the epoch where validation turns is where early stopping should trigger. Both curves flat and high means the net has not got started: check the learning rate before assuming the design is too small.
The size of the gap is not it a problem. A large but stable gap on a net whose validation loss is still the best you have reached is fine. It is the trend in validation loss that decides whether to stop.
A steep initial drop as the net learns the easy structure, then a long shallow decline as it refines. Validation tracks training closely at first and gradually separates. Both curves are noisy step to step — that is mini-batch sampling, not instability — and the trend over a window of epochs is what to judge.
If loss is still falling meaningfully when training ends, you stopped too early. If validation has been flat or rising for many epochs, you trained too long and the useful checkpoint is behind you.
Two lines — training loss and validation loss against epochs — diagnose more problems faster than any other single output.
Read them together, never apart. The level of the training curve measures bias: how well the net can fit at all. The gap between the curves measures variance: how much of that fit was memorisation.
| Shape | Diagnosis | Action |
|---|---|---|
| Both falling, close together | Healthy, still learning | Keep training |
| Both flat and high | Underfitting | Bigger model, better features, higher rate |
| Training falls, validation rises | Overfitting | Regularise, augment, stop earlier |
| Training falls, validation flat | Mild overfitting or a ceiling | More data or regularisation |
| Both flat from step one | Learning rate far too low, or a bug | Check gradients and the rate |
Loss jumps to NaN | Rate too high, or log(0) | Lower the rate, clip gradients |
| Validation below training | Augmentation or dropout artefact, or a leak | Check the pipeline |
That last row is worth expanding. Validation loss below training loss is often benign: dropout and augmentation are active in training and disabled at validation. So the training count is measured under harder conditions. If neither is in use, suspect a leak.
Beyond the broad diagnosis, the curve's shape carries set information.
A sharp initial drop then a long plateau is normal. The net learns the easy structure — class priors, obvious features — in the first epoch or two. And the rest is slow refinement.
A staircase means a step learning-rate schedule. Each drop is a rate reduction letting the net settle into a narrower min.
A sudden improvement in the final epochs is the cosine schedule's low-rate phase working as intended. Expect it, and set early-stopping patience long enough not to cut it off.
A sudden permanent jump upwards means the learning rate was too high for the sharper region the net had entered. The weights have been thrown somewhere worse. A decaying schedule stops it.
Noise batch to batch, smooth across epochs is expected. Judge the trend over a few epochs. A single bad epoch is not a signal.
Learning rate. Plot it alongside the loss and the connection between schedule and progress turns visible at once. It also catches the common bug of a scheduler stepped at the wrong interval.
Gradient norm per layer. A steady decay from output to input means vanishing gradients. Vals in the thousands mean exploding. This is the fastest diagnosis on hand for a network that will not train.
The task metric. Accuracy or F1 alongside the loss, since they can diverge — loss rising while accuracy holds means growing overconfidence rather than worse classification.
The fraction of zero activations. Around 50% after ReLU is healthy. 90% and rising means units are dying.
Logging all four costs almost nothing and turns debugging from guesswork into reading.
Judging on training loss alone. It almost always keeps falling. A net can drive training loss to nearly zero while getting steadily worse at its actual job.Training for a fixed epoch count. The right count is wherever validation loss bottoms out. And that moves with each change to the data, the net or the learning rate.Reading noise as trend. With a small validation set, a wobble of a couple of percent between epochs is sampling noise. Wait for a sustained rise before concluding anything.
The gap between the curves is the diagnosis. Either curve alone is not.
A learning curve plots performance against the amount of training data rather than against epochs. And it answers a question no other diagnostic does: would more data help?
Train on 10%, 20%, … 100% of the data and plot both scores:
Both curves converging to a high error means you are limited by the net, not the data. More rows will not help. More capacity or better features will.
A large left gap, with validation error still falling means you are limited by data. Collecting more is the highest-value action on hand.
Validation error flat as data grows
That distinction is worth an afternoon of compute before committing to an costly annotation project.
When a training run looks wrong, work through this in order:
Most training failures are caught by the first four.
Training curves are the main diagnostic you get. And most of their shapes have a set cause. Here are six, made from those causes, with the reading for each.
The left three curves cover a learning rate that is too high, too low. And the confusing case where validation sits below training.
Two real rules summarise the diagnostic, plus a reminder about log-scale plotting.
How many epochs should I train for? Until validation stops improving, with early stopping deciding rather than a fixed count.
Should the training loss reach zero? On real data, no — noise sets a floor. Reaching zero often means memorisation.
Why is my validation loss so noisy? A small validation set. Or a high learning rate. Increase the set size or smooth over a few epochs.
My training loss is less than validation from the start. Overfitting? Not necessarily — some gap is normal from the first epoch. Watch whether it widens.
Should I plot loss or accuracy? Both. Loss is the smoother signal. Accuracy is what you report. And the two can disagree informatively.
What if both curves are flat at just the same value? The net is predicting a constant. Check the learning rate, the initialisation. And whether the optimiser is stepping at all.
Answer without scrolling back up.
Training loss keeps falling while validation loss rises. The right reading is:
The divergence point is where the model stopped learning the pattern and started learning the training set. Everything after it is overfitting.
Both curves are flat and high from the very first epoch. Most likely:
A model that never improves has not converged, it has failed to start. Check the learning rate first, then whether the labels and the loss are actually connected.
Validation loss sits consistently *below* training loss. The usual explanation is:
Dropout is on during training and off during validation, so the training number is measured under harder conditions. It is normal early on and not a cause for alarm.
Read training and validation curves like a practitioner: spot underfitting, a healthy fit and overfitting from their shape alone.