Prevent overfitting by halting training when the model stops generalizing! The network monitors Validation Loss. If it fails to improve for a set number of epochs (Patience), training stops and the weights are restored to the Best Epoch.
Overview
The shape every training run has
Training loss falls more or less monotonically: the model keeps getting better at the data it can see. Validation loss falls too, up to a point — then turns and rises.
That turning point is where the model stops learning generalisable structure and starts memorising the training set. Everything after it makes the model worse on new data while the training curve continues to look excellent. Early stopping simply keeps the weights from the bottom of the validation curve and discards everything after.
20
Number of epochs to wait for improvement before stopping.
Training Metrics
Status
READY
Click Start to begin training simulation.
Train Loss
--
Val Loss
--
Epoch: 0
Best Epoch: 0
Train
Val
Best
Drag to Pan | Scroll to Zoom
Early Stopping: A Practical Guide
Stop training when validation loss stops improving. It is the cheapest regulariser available - it costs nothing, needs no extra term in the loss, and often works as well as anything else.
Patience, and why you cannot stop at the first uptick
Validation loss is noisy. Mini-batch sampling, dropout and a finite validation set mean it rises and falls between epochs even while the underlying trend is still improving. Stopping at the first increase would stop almost immediately and almost always too early.
Patience is the number of epochs to keep training without improvement before giving up. With patience 10, the run continues 10 epochs past the best score and stops only if nothing beats it in that window.
The other half is restoring weights. Because training continued past the best point, the final weights are not the best weights — the best checkpoint must be saved when it occurs and reloaded at the end. Skipping that gives you a model that is deliberately patience epochs overfit.
Stopping at the right moment
Train a network long enough and training loss keeps falling while validation loss stops improving and then rises. That turning point is where the model stops learning the pattern and starts memorising the noise.
Early stopping watches validation performance and halts when it stops improving. It is the cheapest regulariser available — free in compute, since it saves time — and it requires no tuning beyond one number.
The mechanism has three parts:
Monitor a validation metric after each epoch.
Track the best value seen, and save a checkpoint whenever it improves.
Stop when it has not improved for patience epochs, and restore the best checkpoint.
That third detail is the one people miss. Stopping without restoring leaves you with the weights from patience epochs past the best point — the worst of both worlds.
Patience, and why it is not zero
Validation loss is noisy. It dips and recovers, sometimes for several epochs, particularly with small validation sets or aggressive augmentation. Stopping at the first non-improvement would end most runs prematurely.
Patience
Behaviour
0–1
Stops on noise; almost always too early
5–10
The usual range
20+
For noisy metrics, or long schedules
Patience also interacts with the learning-rate schedule. With a cosine decay, the final low-rate epochs often produce a distinct improvement — so patience must be long enough to survive the plateau before it, or you will stop just short of the best result the schedule was building towards.
best, wait = float("inf"), 0
for epoch in range(max_epochs):
train_one_epoch()
val = validate()
if val < best - 1e-4: # min_delta: ignore trivial changes
best, wait = val, 0
torch.save(model.state_dict(), "best.pt")
else:
wait += 1
if wait >= patience:
break
model.load_state_dict(torch.load("best.pt")) # restore the best, not the last
What to monitor
Validation loss is the default, and it is not always the right choice.
Loss is smooth and sensitive, which makes it a good early-stopping signal. But it can rise while accuracy holds steady — a model becoming overconfident on examples it already classifies correctly. Stopping there discards a model that was still improving on the thing you care about.
The task metric — accuracy, F1, IoU, AUC — is what you will report, so stopping on it aligns the decision with the goal. It is noisier, so it needs more patience.
The practical answer: monitor the metric that matters for the decision, with enough patience to ride out its noise, and log both.
Note that the validation set is now part of the training procedure. Every early-stopping decision uses it, so its score is no longer an unbiased estimate of generalisation. Keep a separate test set for the number you report.
Interactive Exploration Guide
Watch the curves diverge. Press Start Training and follow both lines. They fall together, then separate — training keeps dropping while validation flattens and turns up. That gap is overfitting, drawn directly.
Stop too eagerly. Set Patience (Epochs) to 5 and press Reset Model, then train. The run often halts during a temporary plateau, before the model has finished improving — underfitting caused by impatience.
Give it room. Set Patience (Epochs) to 60 and train again. Now the run survives noisy stretches and stops near the true minimum, at the cost of extra epochs after the best point.
Find the middle. Try 20 and compare where it halts against the visible minimum of the validation curve. Patience is a bias–variance trade of its own: too little stops on noise, too much wastes compute.
Why it works as regularisation
Early stopping restricts how far the weights can travel from their initialisation. With small initial weights and a bounded number of gradient steps, the reachable region of weight space is limited — which is a constraint on effective model capacity, not merely a stopping heuristic.
For linear models with gradient descent this can be made precise: early stopping is equivalent to L2 regularisation, with the number of epochs playing the role of 1/λ. Fewer epochs correspond to a stronger penalty. Deep networks are not linear, but the intuition transfers — which is why early stopping and weight decay are partly redundant.
What usually goes wrong
Not restoring the best weights. Stopping is only half the technique. Without restore_best_weights=True or an explicit checkpoint reload, you keep the overfit weights you trained past the minimum.
Monitoring the training loss. Training loss almost never rises, so it will never trigger a stop. Monitor validation loss, or validation accuracy if that is the deployment metric.
Using the test set as the validation set. Choosing the stopping epoch by test performance leaks the test set and makes the reported number optimistic.
Patience of 1 or 2. Guaranteed to stop on noise. Scale patience to how noisy the curve is — 10 to 20 epochs is a common default.
A validation set too small to be informative. With a few hundred samples the curve is so noisy that the stopping point is essentially random.
Key takeaway
Early stopping monitors validation loss, waits patience epochs after the best score, then halts and restores the best checkpoint. It regularises by limiting how far weights travel from initialisation — provably equivalent to L2 for linear models — and it costs nothing beyond the validation split you already have. The two things that make it fail are monitoring the wrong curve and forgetting to restore the best weights.
Why it works as a regulariser
Early stopping is not merely a convenience. For linear models it is provably equivalent to L2 regularisation of a particular strength: limiting the number of gradient steps limits how far the weights can travel from their small initial values, which is exactly what a weight penalty does.
The same intuition carries to networks. Weights grow during training, and larger weights mean sharper, more confident functions that fit finer detail — including noise. Stopping early keeps them smaller.
That also explains why early stopping and weight decay partly overlap, and why adding both usually gives less than the sum of their individual effects.
Where it fits with the other regularisers
Method
Cost
Tuning
Early stopping
Negative — saves compute
One patience value
Weight decay
Free
One coefficient
Dropout
Slows convergence
One rate per layer group
Augmentation
CPU time
Task-specific choices
Early stopping should essentially always be on. It costs nothing, it protects against the most common failure mode, and the checkpoint it keeps is the model you would have wanted anyway.
The one situation where it is set aside is very large-scale pretraining, where a fixed step budget and a fixed schedule are decided in advance and the model is not trained to convergence at all — there is no overfitting to stop, because the data is effectively unlimited.
Common mistakes
Not restoring the best checkpoint, which discards the whole benefit.
Patience of one, stopping on noise.
Monitoring training loss, which never rises and so never triggers.
Using the test set to decide when to stop, which makes the test score optimistic.
A validation set too small to give a stable signal — a few hundred examples produce metrics that swing by several percent.
Stopping before a scheduled learning-rate drop has had a chance to help.
Stop at the bottom, not at the end
Validation loss falls, turns, and rises. Early stopping keeps the weights from the turn -- and the two parameters that decide when to stop are where the subtlety lives.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
# a synthetic training curve with the shape real ones have:
# training loss falling forever, validation loss turning at some point
EPOCHS = 60
epochs = np.arange(EPOCHS)
train = 1.2 * np.exp(-epochs / 12) + 0.05
val = 1.2 * np.exp(-epochs / 12) + 0.05 + 0.010 * np.maximum(0, epochs - 18)
val = val + rng.normal(0, 0.012, EPOCHS) # real curves are noisy
print("%8s %12s %12s" % ("epoch", "train", "validation"))
for e in range(0, EPOCHS, 5):
mark = ""
print("%8d %12.4f %12.4f %s" % (e, train[e], val[e], mark))
print()
best = int(np.argmin(val))
print("training loss falls for all %d epochs and never turns." % EPOCHS)
print("validation loss bottoms out at epoch %d (%.4f) and ends at %.4f."
% (best, val[best], val[-1]))
print("training to the end costs you %.4f of validation loss for nothing."
% (val[-1] - val[best]))
print()
def run(patience, min_delta=0.0):
best_val, best_epoch, wait = np.inf, 0, 0
for e in range(EPOCHS):
if val[e] < best_val - min_delta:
best_val, best_epoch, wait = val[e], e, 0
else:
wait += 1
if wait >= patience:
return e, best_epoch, best_val
return EPOCHS - 1, best_epoch, best_val
print("patience -- how many epochs of no improvement you tolerate:")
print("%10s %12s %14s %12s %12s"
% ("patience", "stopped at", "restored to", "val loss", "epochs saved"))
for p in (1, 3, 5, 10, 20):
stop, be, bv = run(p)
print("%10d %12d %14d %12.4f %12d" % (p, stop, be, bv, EPOCHS - 1 - stop))
print()
print("patience=1 stops at epoch %d -- the curve is noisy, so a single bad"
% run(1)[0])
print("epoch looks like the end. the true minimum is at %d." % best)
print("patience=10 finds it and still saves %d epochs of compute."
% (EPOCHS - 1 - run(10)[0]))
print()
print("the second parameter is min_delta -- how much counts as improvement:")
for md in (0.0, 0.005, 0.02):
stop, be, bv = run(10, md)
print(" min_delta %.3f: stopped at %2d, best epoch %2d, val %.4f"
% (md, stop, be, bv))
print(" a min_delta larger than the noise stops you chasing it. a min_delta")
print(" larger than the real improvements stops you too early.")
print()
print("and the part that is easy to get wrong -- RESTORING the weights:")
print(" stopping at epoch %d without restoring leaves you the weights from"
% run(10)[0])
print(" epoch %d, with validation loss %.4f." % (run(10)[0], val[run(10)[0]]))
print(" restoring gives you epoch %d, at %.4f." % (best, val[best]))
print(" the difference is %.4f, which is the entire benefit. an early"
% (val[run(10)[0]] - val[best]))
print(" stopping callback that does not restore is just a compute saver.")
print()
print("what it is really doing: early stopping is a regulariser. it limits")
print("how far the weights can travel from their initialisation, which is")
print("the same thing weight decay does by a different route. that is why")
print("stacking both often helps less than you would expect.")
print()
print("two practical notes. stop on the metric you care about, not always on")
print("loss -- a model can improve F1 while its loss worsens. and never stop")
print("on the TEST set: the epoch you choose is a fitted parameter, so a test")
print("set used to choose it is no longer a test set.")
Output
Questions people ask
What patience should I use? 5–10 epochs for most work; more if the metric is noisy or the schedule is long.
Should I monitor loss or accuracy? Whichever you will report, with patience matched to its noise. Log both.
Does early stopping replace other regularisation? No. It complements weight decay and augmentation, and overlaps partly with weight decay.
Can I combine it with a learning-rate schedule? Yes, and reduce-on-plateau plus early stopping is a common pairing — cut the rate on the first plateau, stop on the second.
Should I retrain on train plus validation afterwards? Some do, for the extra data, using the epoch count found by early stopping. It gives up the ability to detect that the retrained model is worse, so it is a judgement call.
What is min_delta for? To ignore improvements too small to be meaningful, so a metric creeping by 1e-6 does not reset the patience counter for ever.
Recap in one screen
Watch validation performance, keep the best checkpoint, and stop when it stops improving.
Restore the best weights — stopping without restoring is the common mistake.
Patience of 5–10 epochs rides out normal noise; patience of 1 stops on it.
Monitor the metric you will report, since loss and accuracy can diverge.
It is provably related to L2 regularisation, costs nothing, and should almost always be enabled.
Check yourself
0 of 3
Answer without scrolling back up.
Early stopping monitors which quantity?
Training loss almost always keeps falling, so it can never signal when to stop. Validation loss is the one that turns around when generalisation starts degrading.
What does the 'patience' setting control?
Validation loss is noisy and can tick up for an epoch or two before improving again. Patience stops you from quitting on a wobble.
Patience is set to 0. What is the risk?
With no tolerance at all, one bad epoch ends the run. You get an undertrained model and a validation curve that had plenty left in it.
Cheat sheet
Early Stopping
Prevent overfitting by halting training when the model stops generalizing! The network monitors Validation Loss. If it fails to improve for a set number of epochs (Patience), training stops and the weights are restored to the Best Epoch.
DEEP LEARNING · vizlearn.in/deep_learning/early_stopping_in_neural_networks.html
Early Stopping - But When?Prechelt, Neural Networks: Tricks of the Trade, 1998
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.