Prevent overfitting by halting training when the model stops generalizing! The network monitors Validation Loss. If it fails to improve for a set number of epochs (Patience), training stops and the weights are restored to the Best Epoch.
20
Number of epochs to wait for improvement before stopping.
Training Metrics
Status
READY
Click Start to begin training simulation.
Train Loss
--
Val Loss
--
Epoch: 0
Best Epoch: 0
Train
Val
Best
Drag to Pan | Scroll to Zoom
Overview
The shape each training run has
Training loss falls more or less monotonically: the net keeps getting better at the data it can see. Validation loss falls too, up to a point — then turns and rises.
That turning point is where the net stops learning generalisable structure and starts memorising the training set. All after it makes the net worse on new data while the training curve continues to look excellent. Early stopping simply keeps the weights from the bottom of the validation curve and discards all after. The build uses a "patience" param: training continues for p epochs after the last improvement in validation loss. And if no improvement appears, it stops and restores the weights from the best epoch. Patience is needed since validation loss is noisy — a single uptick does not mean the trend has reversed. Vals between 5 and 20 are typical. More patience avoids premature stopping at the cost of longer training. The "delta" param sets the min improvement that counts: a decrease of 0.0001 after a plateau is noise, not progress. Early stopping is complementary to other regularisers: you can use it alongside dropout, weight decay. And data augmentation. In PyTorch, torch.optim.lr_scheduler.ReduceLROnPlateau reduces the learning rate when the loss plateaus. And early stopping acts as the final backstop when still a reduced learning rate cannot improve. Keras gives EarlyStopping as a callback with restore_best_weights=True. This page trains a net with and without early stopping, plots both loss curves, and shows how patience and delta affect when training halts.
Prevent overfitting by halting training when the model stops generalizing!
Stop training when validation loss stops improving. It is the cheapest regulariser available - it costs nothing, needs no extra term in the loss, and often works as well as anything else.
The shape each training run has
Training loss falls more or less monotonically: the net keeps getting better at the data it can see. Validation loss falls too, up to a point — then turns and rises.
That turning point is where the net stops learning generalisable structure and starts memorising the training set. All after it makes the net worse on new data while the training curve continues to look excellent. Early stopping simply keeps the weights from the bottom of the validation curve and discards all after. The build uses a "patience" param: training continues for p epochs after the last improvement in validation loss. And if no improvement appears, it stops and restores the weights from the best epoch. Patience is needed since validation loss is noisy — a single uptick does not mean the trend has reversed. Vals between 5 and 20 are typical. More patience avoids premature stopping at the cost of longer training. The "delta" param sets the min improvement that counts: a decrease of 0.0001 after a plateau is noise, not progress. Early stopping is complementary to other regularisers: you can use it alongside dropout, weight decay. And data augmentation. In PyTorch, torch.optim.lr_scheduler.ReduceLROnPlateau reduces the learning rate when the loss plateaus. And early stopping acts as the final backstop when still a reduced learning rate cannot improve. Keras gives EarlyStopping as a callback with restore_best_weights=True. This page trains a net with and without early stopping, plots both loss curves, and shows how patience and delta affect when training halts.
Patience, and why you cannot stop at the first uptick
Validation loss is noisy. Mini-batch sampling, dropout and a finite validation set mean it rises and falls between epochs still while the underlying trend is still improving. Stopping at the first increase would stop almost at once and almost always too early.
Patience is the count of epochs to keep training without improvement before giving up. With patience 10, the run continues 10 epochs past the best score and stops just if nothing beats it in that window.
The other half is restoring weights. Since training continued past the best point, the final weights are not the best weights — the best checkpoint must be saved when it occurs and reloaded at the end. Skipping that gives you a net that is deliberately patience epochs overfit.
Stopping at the right moment
Train a network long enough and training loss keeps falling while validation loss stops improving and then rises. That turning point is where the net stops learning the pattern and starts memorising the noise.
Early stopping watches validation performance and halts when it stops improving. It is the cheapest regulariser on hand — free in compute, since it saves time — and it needs no tuning beyond one count.
The mechanism has three parts:
Monitor a validation metric after each epoch.
Track the best value seen, and save a checkpoint whenever it improves.
Stop when it has not improved for patience epochs. And restore the best checkpoint.
That third detail is the one people miss. Stopping without restoring leaves you with the weights from patience epochs past the best point — the worst of both worlds.
Patience, and why it is not zero
Validation loss is noisy. It dips and recovers, sometimes for a few epochs, very with small validation sets or aggressive augmentation. Stopping at the first non-improvement would end most runs prematurely.
Patience
Behaviour
0–1
Stops on noise; almost always too early
5–10
The usual range
20+
For noisy metrics, or long schedules
Patience also interacts with the learning-rate schedule. With a cosine decay, the final low-rate epochs often give a distinct improvement — so patience must be long enough to survive the plateau before it. Or you will stop just short of the best result the schedule was building towards.
best, wait = float("inf"), 0
for epoch in range(max_epochs):
train_one_epoch()
val = validate()
if val < best - 1e-4: # min_delta: ignore trivial changes
best, wait = val, 0
torch.save(model.state_dict(), "best.pt")
else:
wait += 1
if wait >= patience:
break
model.load_state_dict(torch.load("best.pt")) # restore the best, not the last
What to monitor
Validation loss is the default, and it is not always the right choice.
Loss is smooth and sensitive, which makes it a good early-stopping signal. But it can rise while accuracy holds steady — a net turning overconfident on cases it now classifies correctly. Stopping there discards a net that was still improving on the thing you care about.
The task metric — accuracy, F1, IoU, AUC — is what you will report. So stopping on it aligns the decision with the goal. It is noisier, so it needs more patience.
The real answer: monitor the metric that matters for the decision, with enough patience to ride out its noise. And log both.
Note that the validation set is now part of the training procedure. Each early-stopping decision uses it. So its score is no longer an unbiased estimate of generalisation. Keep a split test set for the count you report.
Interactive Exploration Guide
Watch the curves diverge. Press Start Training and follow both lines. They fall together, then split — training keeps dropping while validation flattens and turns up. That gap is overfitting, drawn right.
Stop too eagerly. Set Patience (Epochs) to 5 and press Reset Net, then train. The run often halts in a temporary plateau, before the net has finished improving — underfitting caused by impatience.
Give it room. Set Patience (Epochs) to 60 and train again. Now the run survives noisy stretches and stops near the true min, at the cost of extra epochs after the best point.
Find the middle. Try 20 and check where it halts against the visible min of the validation curve. Patience is a bias–variance trade of its own: too little stops on noise, too much wastes compute.
Why it works as regularisation
Early stopping restricts how far the weights can travel from their initialisation. With small initial weights and a bounded count of gradient steps, the reachable region of weight space is limited — which is a constraint on effective net capacity, not merely a stopping heuristic.
For linear nets with gradient descent this can be made precise: early stopping is equivalent to L2 regularisation, with the count of epochs playing the role of 1/λ. Fewer epochs correspond to a stronger cost. Deep networks are not linear. But the intuition transfers — which is why early stopping and weight decay are partly redundant.
What often goes wrong
Not restoring the best weights. Stopping is just half the technique. Without restore_best_weights=True or an explicit checkpoint reload, you keep the overfit weights you trained past the min.
Monitoring the training loss. Training loss almost never rises. So it will never trigger a stop. Monitor validation loss, or validation accuracy if that is the deployment metric.
With the test set as the validation set. Choosing the stopping epoch by test performance leaks the test set and makes the reported count optimistic.
Patience of 1 or 2. Guaranteed to stop on noise. Scale patience to how noisy the curve is — 10 to 20 epochs is a common default.
A validation set too small to be informative. With a few hundred samples the curve is so noisy that the stopping point is in effect random.
Key takeaway
Early stopping monitors validation loss, waits patience epochs after the best score, then halts and restores the best checkpoint. It regularises by limiting how far weights travel from initialisation — provably equivalent to L2 for linear nets — and it costs nothing beyond the validation split you now have. The two things that make it fail are monitoring the wrong curve and forgetting to restore the best weights.
Why it works as a regulariser
Early stopping is not merely a convenience. For linear nets it is provably equivalent to L2 regularisation of a particular strength: limiting the count of gradient steps limits how far the weights can travel from their small initial vals. It is just what a weight cost does.
The same intuition carries to networks. Weights grow in training. And more weights mean sharper, more confident functions that fit finer detail — with noise. Stopping early keeps them less.
That also explains why early stopping and weight decay partly overlap. And why adding both often gives less than the sum of their individual effects.
Where it fits with the other regularisers
Method
Cost
Tuning
Early stopping
Negative — saves compute
One patience value
Weight decay
Free
One coefficient
Dropout
Slows convergence
One rate per layer group
Augmentation
CPU time
Task-specific choices
Early stopping should in effect always be on. It costs nothing, it protects against the most common failure mode. And the checkpoint it keeps is the net you would have wanted anyway.
The one case where it is set aside is very large-scale pretraining. There a fixed step budget and a fixed schedule are decided in advance and the net is not trained to convergence at all — there is no overfitting to stop, since the data is effectively unlimited.
Common mistakes
Not restoring the best checkpoint, which discards the whole gain.
Patience of one, stopping on noise.
Monitoring training loss, which never rises and so never triggers.
With the test set to decide when to stop, which makes the test score optimistic.
A validation set too small to give a stable signal — a few hundred cases give metrics that swing by a few percent.
Stopping before a scheduled learning-rate drop has had a chance to help.
Stop at the bottom, not at the end
Validation loss falls, turns, and rises. Early stopping keeps the weights from the turn -- and the two params that decide when to stop are where the subtlety lives.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
# a synthetic training curve with the shape real ones have:
# training loss falling forever, validation loss turning at some point
EPOCHS = 60
epochs = np.arange(EPOCHS)
train = 1.2 * np.exp(-epochs / 12) + 0.05
val = 1.2 * np.exp(-epochs / 12) + 0.05 + 0.010 * np.maximum(0, epochs - 18)
val = val + rng.normal(0, 0.012, EPOCHS) # real curves are noisy
print("%8s %12s %12s" % ("epoch", "train", "validation"))
for e in range(0, EPOCHS, 5):
mark = ""
print("%8d %12.4f %12.4f %s" % (e, train[e], val[e], mark))
print()
best = int(np.argmin(val))
print("training loss falls for all %d epochs and never turns." % EPOCHS)
print("validation loss bottoms out at epoch %d (%.4f) and ends at %.4f."
% (best, val[best], val[-1]))
print("training to the end costs you %.4f of validation loss for nothing."
% (val[-1] - val[best]))
Output
The patience param controls how many epochs of non-improvement to tolerate before stopping.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
EPOCHS = 60
epochs = np.arange(EPOCHS)
val = 1.2 * np.exp(-epochs / 12) + 0.05 + 0.010 * np.maximum(0, epochs - 18)
val = val + rng.normal(0, 0.012, EPOCHS)
best = int(np.argmin(val))
def run(patience, min_delta=0.0):
best_val, best_epoch, wait = np.inf, 0, 0
for e in range(EPOCHS):
if val[e] < best_val - min_delta:
best_val, best_epoch, wait = val[e], e, 0
else:
wait += 1
if wait >= patience:
return e, best_epoch, best_val
return EPOCHS - 1, best_epoch, best_val
print("patience -- how many epochs of no improvement you tolerate:")
print("%10s %12s %14s %12s %12s"
% ("patience", "stopped at", "restored to", "val loss", "epochs saved"))
for p in (1, 3, 5, 10, 20):
stop, be, bv = run(p)
print("%10d %12d %14d %12.4f %12d" % (p, stop, be, bv, EPOCHS - 1 - stop))
print()
print("patience=1 stops at epoch %d -- the curve is noisy, so a single bad"
% run(1)[0])
print("epoch looks like the end. the true minimum is at %d." % best)
print("patience=10 finds it and still saves %d epochs of compute."
% (EPOCHS - 1 - run(10)[0]))
Output
The min_delta param sets the threshold for what counts as real improvement. And restoring the best weights is what makes the whole technique work.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
EPOCHS = 60
epochs = np.arange(EPOCHS)
val = 1.2 * np.exp(-epochs / 12) + 0.05 + 0.010 * np.maximum(0, epochs - 18)
val = val + rng.normal(0, 0.012, EPOCHS)
best = int(np.argmin(val))
def run(patience, min_delta=0.0):
best_val, best_epoch, wait = np.inf, 0, 0
for e in range(EPOCHS):
if val[e] < best_val - min_delta:
best_val, best_epoch, wait = val[e], e, 0
else:
wait += 1
if wait >= patience:
return e, best_epoch, best_val
return EPOCHS - 1, best_epoch, best_val
print("the second parameter is min_delta -- how much counts as improvement:")
for md in (0.0, 0.005, 0.02):
stop, be, bv = run(10, md)
print(" min_delta %.3f: stopped at %2d, best epoch %2d, val %.4f"
% (md, stop, be, bv))
print(" a min_delta larger than the noise stops you chasing it. a min_delta")
print(" larger than the real improvements stops you too early.")
print()
print("and the part that is easy to get wrong -- RESTORING the weights:")
print(" stopping at epoch %d without restoring leaves you the weights from"
% run(10)[0])
print(" epoch %d, with validation loss %.4f." % (run(10)[0], val[run(10)[0]]))
print(" restoring gives you epoch %d, at %.4f." % (best, val[best]))
print(" the difference is %.4f, which is the entire benefit. an early"
% (val[run(10)[0]] - val[best]))
print(" stopping callback that does not restore is just a compute saver.")
print()
print("what it is really doing: early stopping is a regulariser. it limits")
print("how far the weights can travel from their initialisation, which is")
print("the same thing weight decay does by a different route. that is why")
print("stacking both often helps less than you would expect.")
Output
Stop on the metric you care about, not always on loss. And never stop on the test set — the epoch you choose is a fitted param. So a test set used to choose it is no longer a test set.
example_04.pyNumPy
print("stop on the metric you care about, not always on loss -- a model can")
print("improve F1 while its loss worsens. and never stop on the TEST set: the")
print("epoch you choose is a fitted parameter, so a test set used to choose")
print("it is no longer a test set.")
Output
Questions people ask
What patience should I use? 5–10 epochs for most work. More if the metric is noisy or the schedule is long.
Should I monitor loss or accuracy? Whichever you will report, with patience matched to its noise. Log both.
Does early stopping replace other regularisation? No. It complements weight decay and augmentation, and overlaps partly with weight decay.
Can I mix it with a learning-rate schedule? Yes. And reduce-on-plateau plus early stopping is a common pairing — cut the rate on the first plateau, stop on the second.
Should I retrain on train plus validation afterwards? Some do, for the extra data, with the epoch count found by early stopping. It gives up the ability to detect that the retrained net is worse. So it is a judgement call.
What is min_delta for? To ignore improvements too small to be meaningful. So a metric creeping by 1e-6 does not reset the patience counter for ever.
Recap in one screen
Watch validation performance, keep the best checkpoint. And stop when it stops improving.
Restore the best weights — stopping without restoring is the common mistake.
Patience of 5–10 epochs rides out normal noise. Patience of 1 stops on it.
Monitor the metric you will report, since loss and accuracy can diverge.
It is provably tied to L2 regularisation, costs nothing. And should almost always be enabled.
Check yourself
0 of 3
Answer without scrolling back up.
Early stopping monitors which quantity?
Training loss almost always keeps falling, so it can never signal when to stop. Validation loss is the one that turns around when generalisation starts degrading.
What does the 'patience' setting control?
Validation loss is noisy and can tick up for an epoch or two before improving again. Patience stops you from quitting on a wobble.
Patience is set to 0. What is the risk?
With no tolerance at all, one bad epoch ends the run. You get an undertrained model and a validation curve that had plenty left in it.
Cheat sheet
Early Stopping
Watch validation loss bottom out then start climbing, and see early stopping halt training at the moment generalisation peaks.
DEEP LEARNING · vizlearn.in/deep_learning/early_stopping_in_neural_networks.html
Early Stopping - But When?Prechelt, Neural Networks: Tricks of the Trade, 1998
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.