Simulate and understand how Loss and Accuracy evolve over time. Tune the architecture mathematically to observe underfitting, ideal fits, and overfitting.
Two lines tell you almost everything about a training run. Learning to read the gap between them is the fastest diagnostic skill in machine learning.
Training loss 0.08, validation loss 0.34. The gap of 0.26 is the model's memorisation of the training set — that is overfitting, no matter how good 0.08 looks in isolation.
Training loss 0.31, validation loss 0.33. A tiny gap, but both are high. The model is not overfitting; it simply is not good enough yet. These two situations need opposite responses, which is exactly why looking at one curve is not enough.
The two curves matter less individually than the distance between them, and how that distance changes over time.
A gap that stays narrow while both curves fall is a healthy run. A gap that opens steadily — training continuing down while validation flattens or rises — is overfitting, and the epoch where validation turns is where early stopping should trigger. Both curves flat and high means the model has not got started: check the learning rate before assuming the architecture is too small.
The size of the gap is not itself a problem. A large but stable gap on a model whose validation loss is still the best you have achieved is fine; it is the trend in validation loss that decides whether to stop.
A steep initial drop as the model learns the easy structure, then a long shallow decline as it refines. Validation tracks training closely at first and gradually separates. Both curves are noisy step to step — that is mini-batch sampling, not instability — and the trend over a window of epochs is what to judge.
If loss is still falling meaningfully when training ends, you stopped too early. If validation has been flat or rising for many epochs, you trained too long and the useful checkpoint is behind you.
Two lines — training loss and validation loss against epochs — diagnose more problems faster than any other single output.
Read them together, never separately. The level of the training curve measures bias: how well the model can fit at all. The gap between the curves measures variance: how much of that fit was memorisation.
| Shape | Diagnosis | Action |
|---|---|---|
| Both falling, close together | Healthy, still learning | Keep training |
| Both flat and high | Underfitting | Bigger model, better features, higher rate |
| Training falls, validation rises | Overfitting | Regularise, augment, stop earlier |
| Training falls, validation flat | Mild overfitting or a ceiling | More data or regularisation |
| Both flat from step one | Learning rate far too low, or a bug | Check gradients and the rate |
Loss jumps to NaN | Rate too high, or log(0) | Lower the rate, clip gradients |
| Validation below training | Augmentation or dropout artefact, or a leak | Check the pipeline |
That last row is worth expanding. Validation loss below training loss is usually benign: dropout and augmentation are active during training and disabled at validation, so the training number is measured under harder conditions. If neither is in use, suspect a leak.
Beyond the broad diagnosis, the curve's shape carries specific information.
A sharp initial drop then a long plateau is normal. The model learns the easy structure — class priors, obvious features — in the first epoch or two, and the rest is slow refinement.
A staircase means a step learning-rate schedule; each drop is a rate reduction letting the model settle into a narrower minimum.
A sudden improvement in the final epochs is the cosine schedule's low-rate phase working as intended. Expect it, and set early-stopping patience long enough not to cut it off.
A sudden permanent jump upwards means the learning rate was too high for the sharper region the model had entered. The weights have been thrown somewhere worse. A decaying schedule prevents it.
Noise batch to batch, smooth across epochs is expected. Judge the trend over several epochs; a single bad epoch is not a signal.
Learning rate. Plot it alongside the loss and the connection between schedule and progress becomes visible immediately. It also catches the common bug of a scheduler stepped at the wrong interval.
Gradient norm per layer. A steady decay from output to input means vanishing gradients; values in the thousands mean exploding. This is the fastest diagnosis available for a network that will not train.
The task metric. Accuracy or F1 alongside the loss, because they can diverge — loss rising while accuracy holds means growing overconfidence rather than worse classification.
The fraction of zero activations. Around 50% after ReLU is healthy; 90% and rising means units are dying.
Logging all four costs almost nothing and turns debugging from guesswork into reading.
Judging on training loss alone. It almost always keeps falling. A model can drive training loss to nearly zero while getting steadily worse at its actual job.Training for a fixed epoch count. The right number is wherever validation loss bottoms out, and that moves with every change to the data, the model or the learning rate.Reading noise as trend. With a small validation set, a wobble of a couple of percent between epochs is sampling noise. Wait for a sustained rise before concluding anything.
The gap between the curves is the diagnosis; either curve alone is not.
A learning curve plots performance against the amount of training data rather than against epochs, and it answers a question no other diagnostic does: would more data help?
Train on 10%, 20%, … 100% of the data and plot both scores:
Both curves converging to a high error means you are limited by the model, not the data. More rows will not help; more capacity or better features will.
A large remaining gap, with validation error still falling means you are limited by data. Collecting more is the highest-value action available.
Validation error flat as data grows
That distinction is worth an afternoon of compute before committing to an expensive annotation project.
When a training run looks wrong, work through this in order:
Most training failures are caught by the first four.
Training curves are the main diagnostic you get, and most of their shapes have a specific cause. Here are six, generated from those causes, with the reading for each.
How many epochs should I train for? Until validation stops improving, with early stopping deciding rather than a fixed number.
Should the training loss reach zero? On real data, no — noise sets a floor. Reaching zero usually means memorisation.
Why is my validation loss so noisy? A small validation set, or a high learning rate. Increase the set size or smooth over several epochs.
My training loss is lower than validation from the start. Overfitting? Not necessarily — some gap is normal from the first epoch. Watch whether it widens.
Should I plot loss or accuracy? Both. Loss is the smoother signal; accuracy is what you report, and the two can disagree informatively.
What if both curves are flat at exactly the same value? The model is predicting a constant. Check the learning rate, the initialisation, and whether the optimiser is stepping at all.
Answer without scrolling back up.
Training loss keeps falling while validation loss rises. The right reading is:
The divergence point is where the model stopped learning the pattern and started learning the training set. Everything after it is overfitting.
Both curves are flat and high from the very first epoch. Most likely:
A model that never improves has not converged, it has failed to start. Check the learning rate first, then whether the labels and the loss are actually connected.
Validation loss sits consistently *below* training loss. The usual explanation is:
Dropout is on during training and off during validation, so the training number is measured under harder conditions. It is normal early on and not a cause for alarm.
Simulate and understand how Loss and Accuracy evolve over time. Tune the architecture mathematically to observe underfitting, ideal fits, and overfitting.