Stop asking the model for certainty it cannot have. Move a little probability off the correct answer and the confidence problem goes away.
Overview
The problem with a hard target
Cross-entropy with a one-hot target asks the model to put chance 1 on the correct class and 0 on all else.
Softmax can't give just 1. It approaches it as the correct logit runs away from the others, and reaches it only in the limit. So the target is unattainable, the grad never vanishes, and training keeps pushing the logit gap wider for if you let it.
The consequences are familiar. The model turns too sure, reporting 0.99 on cases it gets wrong. It overfits, because widening logits on training examples is always an available way to reduce the loss. And it's badly calibrated, in just the way the calibration module measures.
Label Smoothing
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
A one-hot target asks for probability 1 on the correct class. That requires an infinite logit gap, so training never stops pushing.
Label smoothing replaces the target with 1−ε+ε/K on the true class and ε/K on the rest.
The optimal logit gap becomes finite, so the model has a reason to stop becoming more confident.
The minimum achievable loss is no longer zero. A loss that plateaus above zero is expected, not a bug.
Label Smoothing
A one-line change to the target that raises calibration. The reason the loss no longer reaches zero.
The change
Move a little chance off the correct class and spread it over the others:
Drag epsilon and watch the bars. At 0 you've the one-hot target. At 0.1 the true class asks for 0.91 and each of nine others asks for about 0.011.
That is the entire technique.
Why it fixes the problem
The readout gives the best logit gap: how far apart the logits have to be to match the target just.
At epsilon 0 it's infinite. This Is the problem restated. At epsilon 0.1 it's a exact finite number, and once the model reaches it the grad is zero. The model now has a reason to *stop*.
That is the mechanism. Not noise, not smoothing in the weight-decay sense — a reachable target where before there was none.
The loss will not reach zero
The readout also gives the min achievable loss. This Is the entropy of the smoothed target.
A model matching the smoothed target just right still reports a loss around 0.5 at common settings. This surprises people who expect training loss to approach zero, and it isn't a bug: you changed what perfect means.
It also means training and validation losses are no longer close with runs that didn't use smoothing. Compare fit, or compare like with like.
What it buys, and what it costs
Better calibration. The headline gain. Confidences become closer to seen accuracies.
Better generalisation. Steady small gains across image classification and translation. It was in the Inception-v3 paper and in the first Transformer, at 0.1 in both, and 0.1 has been the default ever since.
Tighter class clusters. Forms of a class group more tightly, with more even distances between classes.
That last one has a cost. Label smoothing hurts distillation. A student learns from the teacher's full output spread, and the helpful part is the scaled sizes of the *wrong* classes. That this dog was slightly cat and not at all lorry. Smoothing on purpose flattens just that, erasing what the student was supposed to learn. If a model is going to be a distillation teacher, train it without.
The floor the loss cannot go below
Two numbers make label smoothing concrete. With epsilon = 0.1 and K = 10 classes, the target puts 1 - 0.1 + 0.1/10 = 0.91 on the correct class and 0.1/10 = 0.011 on each of the other nine. A model that matches that target *just* still records a cross-entropy equal to the target's own entropy:
-(0.91 · ln 0.91 + 9 · 0.011 · ln 0.011) ≈ 0.50
target
minimum achievable loss
one-hot
0 (but never reached — needs an infinite logit gap)
smoothed, eps = 0.1
~0.50
That is the whole mechanism, stated as math. The one-hot target has a floor of zero that softmax can only approach by driving the correct logit infinitely far from the rest — so the grad never vanishes and training keeps inflating trust. The smoothed target has a floor of 0.50 reached at a *finite* logit gap, so the model has a reason to stop, and it stops before it turns too sure. It also means a smoothed run's training loss isn't close with an unsmoothed one: 0.50 is the new zero, and checking the raw numbers across the two is checking against different floors.
The one place smoothing actively hurts is distillation. A student learns from the teacher's full output spread, and the helpful part is the scaled sizes of the *wrong* classes — that a given dog photo is a little cat-like and not at all lorry-like. That structure is just what smoothing flattens, by design, pushing each wrong class toward the same small value. So a model destined to be a distillation teacher should be trained without it, though 0.1 is the right default almost all over else. It was in the Inception-v3 paper and the first Transformer, and it has stayed the default because the calibration and generalisation gains are steady across classification and translation alike.
It's also worth remembering what smoothing isn't: it's a change to a cat target, so it has no meaning for regression, and it isn't a substitute for clear calibration. Temperature scaling on a validation set corrects trust more right, and the two compose — smoothing during training, scaling afterwards.
Where it goes wrong with Label Smoothing
Epsilon too large. Past about 0.2 you're actively teaching the model that wrong answers are plausible, and fit falls. Drag it up and watch the target flatten.
Checking losses across runs. The floor moved.
Using it on a teacher model. It drops the dark knowledge distillation depends on.
Applying it to regression. It's a change to a cat target and has no meaning without one.
Expecting it to fix a badly calibrated model on its own. It helps, but it is not a full fix on its own. Clear temperature scaling on a validation set helps more, and the two combine well together.
Questions people ask about Label Smoothing
What is label smoothing? It replaces the one-hot target with a softer one, such as 0.9 for the true class and the remaining 0.1 spread over the others.
Why does it help? Hard targets push the logits towards infinity, giving overconfident models. Smoothing caps the confidence and often improves generalisation and calibration.
What value should I use? 0.1 is the common default. Values from 0.05 to 0.2 are typical, and large values hurt accuracy.
Why does my loss never reach zero? Because the target is no longer one-hot, the minimum cross-entropy is the entropy of the smoothed target. That floor is expected.
When does it hurt? In knowledge distillation, where the teacher's soft outputs matter, and in tasks needing sharp probabilities. It can also reduce the quality of a teacher model.
How do I turn it on in PyTorch? Pass label_smoothing=0.1 to nn.CrossEntropyLoss. No change to the targets is needed.
Tell the model it might be wrong
Replacing a target of 1.0 with 0.9 changes what the loss is minimised by. This finds that optimum just, then shows the effect on trust and on the logits themselves.
example_01.pyNumPy
import numpy as np
K = 5 # classes
EPS = 0.1 # smoothing strength
hard = np.zeros(K); hard[0] = 1.0
soft = np.full(K, EPS / K); soft[0] += 1 - EPS
print("a %d-class problem, true class 0." % K)
print(" hard target :", np.round(hard, 4))
print(" smoothed 0.1 :", np.round(soft, 4), " sums to", round(soft.sum(), 6))
print(" the formula is (1 - eps) * onehot + eps / K.")
print()
def softmax(z):
e = np.exp(z - z.max())
return e / e.sum()
def ce(z, target):
return -(target * np.log(softmax(z))).sum() + 0.0
print("what each target is minimised BY. the loss against a hard target")
print("keeps falling as the true logit grows, forever:")
print("%14s %14s %16s" % ("logit gap", "hard loss", "smoothed loss"))
for gap in (1.0, 2.0, 4.0, 8.0, 16.0, 40.0):
z = np.zeros(K); z[0] = gap
print("%14.1f %14.6f %16.6f" % (gap, ce(z, hard), ce(z, soft)))
print()
print("the hard column goes to zero and never stops rewarding a bigger gap.")
print("the smoothed column has a MINIMUM. find it by search:")
gaps = np.linspace(0.1, 20, 4000)
losses = [ce(np.r_[g, np.zeros(K - 1)], soft) for g in gaps]
best = gaps[int(np.argmin(losses))]
print(" smoothed loss is lowest at a logit gap of %.4f" % best)
print(" the closed form says log((1-eps+eps/K) / (eps/K)) = %.4f"
% np.log((1 - EPS + EPS / K) / (EPS / K)))
print()
print("so smoothing does not merely 'soften' the target -- it gives the model")
print("a finite target to aim for. without it the optimum is at infinity, and")
print("the weights grow without bound trying to reach it.")
Output
The next block sweeps epsilon to show the confidence each value targets, and the cost of smoothing on a certain example.
example_02.pyNumPy
import numpy as np
K = 5
EPS = 0.1
def softmax(z):
e = np.exp(z - z.max())
return e / e.sum()
def ce(z, target):
return -(target * np.log(softmax(z))).sum() + 0.0
best = np.log((1 - EPS + EPS / K) / (EPS / K))
z_at_best = np.r_[best, np.zeros(K - 1)]
print("the confidence that corresponds to the optimal logit gap:")
print(" probabilities:", np.round(softmax(z_at_best), 4))
print(" the model's best possible answer is %.4f, not 1.0000."
% softmax(z_at_best)[0])
print()
print("smoothing strength against the confidence it targets:")
print("%10s %16s %16s" % ("eps", "optimal gap", "peak probability"))
for eps in (0.0, 0.01, 0.05, 0.1, 0.2, 0.5):
if eps == 0:
print("%10.2f %16s %16s" % (eps, "infinite", "1.0000"))
continue
gap = np.log((1 - eps + eps / K) / (eps / K))
print("%10.2f %16.4f %16.4f"
% (eps, gap, softmax(np.r_[gap, np.zeros(K - 1)])[0]))
print(" eps=0.5 targets only %.2f confidence, which is barely a decision."
% softmax(np.r_[np.log((1 - 0.5 + 0.5 / K) / (0.5 / K)), np.zeros(K - 1)])[0])
print(" 0.1 is the usual choice and 0.2 is about as far as anyone goes.")
print()
hard = np.zeros(K); hard[0] = 1.0
soft = np.full(K, EPS / K); soft[0] += 1 - EPS
print("and what it costs. a genuinely certain example is now penalised:")
z_sure = np.r_[12.0, np.zeros(K - 1)]
print(" a model that is right and certain (p=%.6f):" % softmax(z_sure)[0])
print(" hard loss %.6f" % ce(z_sure, hard))
print(" smoothed loss %.6f <- it is being pushed back down"
% ce(z_sure, soft))
print()
print("that is the trade. label smoothing buys better-calibrated confidence")
print("and slightly better accuracy, and costs you the ability to be certain")
print("about anything. measure the calibration before and after -- do not assume.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Why does a one-hot target make a model overconfident?
The target is unattainable, so widening logits is always an available way to reduce the loss. Smoothing makes the optimal gap finite and reachable.
Why does the training loss no longer approach zero?
You changed what perfect means. It also means losses are no longer comparable with runs that did not use smoothing.
Why should a distillation teacher be trained without label smoothing?
The informative part is that this dog was slightly cat and not at all lorry. Smoothing erases the dark knowledge distillation depends on.
Cheat sheet
Label Smoothing
Change the target from a hard one-hot to a soft distribution, watch the loss curve shift, and measure the calibration improvement it produces.
DEEP LEARNING · vizlearn.in/deep_learning/label_smoothing.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.