Categorical vs. Probabilistic labels. Pick a tweet to see how data representation impacts nuance.
Overview
Overview
When we train a classification model, we need to provide it with labeled data. The way we represent these labels can have a significant impact on how the model learns. This lab explores two common approaches: Hard Labelling and Soft Labelling. By comparing them side-by-side, you can build a strong intuition for why representing uncertainty can lead to more robust and nuanced models.
Select Input
Hard Label
Negative0%
Neutral0%
Positive0%
Vector: [0, 0, 0]
Soft Label
Negative0%
Neutral0%
Positive0%
Vector: [0, 0, 0]
Hard labelling picks one winner. It’s simple but rigid.
Soft labelling maps a probability distribution. It allows the model to learn uncertainty and class relationships, making it more robust against noisy or ambiguous data.
Understanding Hard vs. Soft Labelling
When we train a classification model, we need to provide it with labeled data. The way we represent these labels can have a significant impact on how the model learns. This lab explores two common approaches: Hard Labelling and Soft Labelling. By comparing them side-by-side, you can build a strong intuition for why representing uncertainty can lead to more robust and nuanced models.
Defining the Labels
Imagine you're training a model to classify the sentiment of a tweet as 'Negative', 'Neutral', or 'Positive'. How you tell the model the "correct" answer for each tweet is where labelling strategy comes in.
Hard Labelling (One-Hot Encoding)
This is the most common approach. You are 100% certain about the class. The correct class gets a value of 1, and all other classes get a 0. It's a "winner-takes-all" method. Example Vector: For a 'Positive' tweet, the hard label is [0, 0, 1]. This tells the model, "This tweet is positive, and nothing else."
Soft Labelling (Probabilistic)
This method acknowledges that the world is messy and some data points are ambiguous. Instead of picking one class, you assign a probability distribution across all classes. Example Vector: For a tweet that is mostly positive but has a hint of neutrality, the soft label might be [0.1, 0.2, 0.7]. This tells the model, "I'm pretty sure this is positive, but there's a small chance it's neutral or even slightly negative."
The two ways to say what the answer is
A hard label is a single confident claim: this image is a cat. Written as a vector for three classes, that is [0, 1, 0] — all the probability mass on one class, none anywhere else.
A soft label spreads the claim: [0.05, 0.90, 0.05], or for a genuinely ambiguous image, [0.30, 0.60, 0.10]. It says what the answer probably is and how sure anyone is.
The difference sounds cosmetic and is not, because of what it does to the loss. With cross-entropy loss and a hard label, the model is pushed to output 1.0 for the true class and 0.0 for everything else. It can never reach those values — softmax outputs are strictly between 0 and 1 — so the pressure never stops. The model keeps growing its weights to push the winning logit further from the others, long after the classification itself is correct.
That endless pressure produces a specific and well-documented failure: a network that is 99.9% confident about images it gets wrong. The confidence has stopped tracking the probability of being right.
Where soft labels come from
Soft targets are not something you invent by hand. Four sources supply them naturally:
Multiple annotators. Ask seven radiologists whether a scan shows a tumour and five say yes. The honest label is 0.71, not 1.0. Averaging the votes into a hard majority throws away exactly the information about which cases are difficult.
Label smoothing. A mechanical version: take the hard label and move a small amount ε (typically 0.1) off the true class, spread evenly across the others. [0, 1, 0] becomes [0.05, 0.90, 0.05]. One line of code, and it is standard in modern image classification training recipes.
Knowledge distillation. Train a large model, then use its full output distribution as the target for a small model. The large model's "this is a cat, but there is something dog-ish about it" carries far more information per example than the word "cat", which is why a distilled small model beats the same model trained on hard labels.
Mixup and similar augmentations. Blend two training images by 70/30 and blend their labels by 70/30 too. The soft label is the honest description of what the blended image contains.
What it buys you, and what it costs
Aspect
Hard labels
Soft labels
Confidence calibration
Typically overconfident
Much closer to honest
Robustness to label noise
Poor — one wrong label is learned absolutely
Better — error is diluted
Information per example
One class name
A full distribution over classes
Peak training accuracy
Higher
Slightly lower by design
Cost to produce
Cheap
More annotators, or a teacher model
The middle row is the practical headline. If 3% of your labels are wrong — which is normal for large annotated datasets — hard labels demand that the model memorise those mistakes perfectly. Soft labels leave room for the model to remain unconvinced.
The fourth row is the honest cost. Label smoothing usually lowers training accuracy a little, because you have explicitly told the model not to be certain. Test accuracy and calibration generally improve, which is the trade you wanted.
One caveat worth knowing: label smoothing tightens the clusters a network forms for each class, which is good for classification but has been shown to hurt when you later want to distil from that network, or use its embeddings for retrieval. Smooth the student, think twice about smoothing the teacher.
What you throw away when you take argmax
A hard label is a probability with the uncertainty deleted. Keeping the probability lets you abstain, rank, and pass calibrated confidence downstream.
example_01.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, log_loss
X, y = make_classification(n_samples=6000, n_features=12, n_informative=5,
flip_y=0.15, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.35, random_state=0)
model = LogisticRegression(max_iter=3000).fit(Xtr, ytr)
prob = model.predict_proba(Xte)[:, 1]
hard = model.predict(Xte)
print("overall accuracy %.4f -- one number for %d very different rows."
% (accuracy_score(yte, hard), len(yte)))
print()
conf = np.maximum(prob, 1 - prob)
print("accuracy, broken down by how confident the model was:")
print("%18s %8s %10s" % ("confidence band", "rows", "accuracy"))
for lo, hi in ((0.5, 0.6), (0.6, 0.7), (0.7, 0.8), (0.8, 0.9), (0.9, 1.01)):
m = (conf >= lo) & (conf < hi)
if m.sum():
print("%18s %8d %10.4f"
% ("%.1f - %.1f" % (lo, hi), m.sum(), accuracy_score(yte[m], hard[m])))
print()
print("the hard label looks the same in every one of those rows. it is not.")
print()
print("so you can abstain. send the uncertain rows to a human:")
print("%14s %10s %14s %12s" % ("send to human", "rows kept", "auto accuracy", "human load"))
for band in (0.00, 0.05, 0.10, 0.20, 0.30):
keep = conf >= 0.5 + band
print("%14s %10d %14.4f %11.1f%%"
% ("conf < %.2f" % (0.5 + band), keep.sum(),
accuracy_score(yte[keep], hard[keep]) if keep.sum() else float("nan"),
100 * (~keep).mean()))
print(" handing over the least confident %.0f%% of rows lifts the automated"
% (100 * (conf < 0.7).mean()))
print(" accuracy from %.4f to %.4f. argmax cannot offer you that trade."
% (accuracy_score(yte, hard), accuracy_score(yte[conf >= 0.7], hard[conf >= 0.7])))
print()
print("and you can rank. take the 100 rows the model is most sure are positive:")
top = np.argsort(-prob)[:100]
print(" precision in that top 100: %.4f (base rate %.4f)"
% (yte[top].mean(), yte.mean()))
print(" hard labels carry no order: all %d rows predicted positive are"
% (hard == 1).sum())
print(" indistinguishable from each other, so there is no 'top 100' to take.")
print()
print("soft labels as TARGETS are a separate idea. two annotators disagreeing")
print("60/40 is real information, and a hard label discards it:")
for soft_target, name in ((1.0, "confident yes"), (0.6, "3 of 5 said yes")):
for pred in (0.95, 0.60, 0.30):
print(" target %.1f (%s), model said %.2f -> loss %.4f"
% (soft_target, name, pred,
-(soft_target * np.log(pred) + (1 - soft_target) * np.log(1 - pred))))
print(" against a soft target of 0.6, predicting 0.60 is the MINIMUM loss.")
print(" against a hard 1.0, the model is pushed toward 1.0 regardless, which")
print(" is how a model learns to be more confident than its data supports.")
print()
print("this is also what label smoothing and knowledge distillation are doing:")
print("replacing a 1.0 with a 0.9, or with a bigger model's actual output, so")
print("the target carries some of the uncertainty the hard label deleted.")
Output
Things to try
Select different tweets from the list to see how these labelling strategies handle different scenarios.
The Clear-Cut Case: Select the second tweet: "Absolutely stunning. Every detail is perfect. Highly recommend!" This is unambiguously positive. Notice that the hard label [0, 0, 1] and the soft label [0.01, 0.04, 0.95] are very similar. In such clear cases, both methods convey almost the same information. The model is strongly encouraged to predict 'Positive'.
The Ambiguous Case: Now, select the first tweet: "The design is gorgeous, but the software is a total nightmare to navigate." This tweet contains both positive ('gorgeous design') and negative ('nightmare software') elements.
The Hard Label is forced to choose one winner. Here, it's classified as 'Negative' ([1, 0, 0]), completely ignoring the positive aspect. This forces the model to be overconfident and lose valuable information.
The Soft Label, however, captures the nuance perfectly with a vector like [0.65, 0.1, 0.25]. It tells the model: "This is mostly negative, but there's a significant positive component too."
The Neutral-Leaning Case: Select the last tweet: "Shipping was late, but the box was packed really well." Again, we see a mix of sentiments. The hard label might classify this as 'Neutral' ([0, 1, 0]), but the soft label [0.4, 0.45, 0.15] provides a much richer signal, indicating it leans neutral but has a strong negative element ('late shipping') and a minor positive one ('packed well').
Why Use Soft Labelling?
While hard labelling is simpler, soft labelling offers several key advantages, a technique often referred to as Label Smoothing:
Better Generalization: By preventing the model from becoming overconfident in its predictions for the training data, soft labelling helps it generalize better to new, unseen data. It acts as a form of regularization.
Robustness to Noise: Real-world datasets often contain mislabeled examples. Soft labels can account for this uncertainty, making the training process more stable and less susceptible to noisy data.
Capturing Inter-Class Relationships: Soft labels can implicitly teach the model about the relationships between classes. For example, it can learn that 'sarcastic' is closer to 'negative' than it is to 'positive'.
Knowledge Distillation: Soft labels are the foundation of knowledge distillation, where the probabilistic outputs of a large, complex model are used as soft targets to train a smaller, more efficient model.
Reading a model's confidence honestly
Once you care about soft labels you inevitably start caring about calibration: does a predicted 0.8 mean the model is right about 80% of the time?
The way to check is a reliability diagram. Bucket the predictions by confidence — 0.5–0.6, 0.6–0.7, and so on — and for each bucket plot average confidence against actual accuracy. A perfectly calibrated model traces the diagonal. A typical modern network traces a curve sagging below it: the 0.9 bucket is right only 75% of the time.
The single number that summarises this is expected calibration error, the average gap between confidence and accuracy across the buckets.
Two standard repairs, both applied after training on a held-out set:
Temperature scaling. Divide the logits by a single learned number T before the softmax. T > 1 softens every prediction. One parameter, does not change any prediction's ranking, and it fixes most of the problem. This is the default choice.
Isotonic regression or Platt scaling. More flexible mappings from raw score to calibrated probability, useful for non-neural models. Isotonic needs more data to avoid overfitting the calibration set itself.
Calibration matters wherever the number is used rather than just the argmax: a medical triage threshold, an expected-value calculation, or a downstream system that combines several models' probabilities.
Questions people ask
Should I always use label smoothing? For large image and text classifiers, it is close to a free improvement at ε = 0.1. For small datasets, calibrated-probability-critical tasks, or when you plan to distil from the model, test it rather than assuming.
Do soft labels need a different loss function? No. Cross-entropy already accepts a full target distribution; the hard-label case is just the special case where one entry is 1. Most frameworks accept either.
How do I combine several annotators' opinions? The simple average is a reasonable start. Better methods weight annotators by their measured reliability, which requires overlapping assignments so agreement can be estimated.
Does this apply to regression? The label-smoothing form does not, but the idea does: predicting a distribution rather than a point value — quantile regression, or a model that outputs a mean and a variance — is the regression version of admitting uncertainty.
Can soft labels hide a genuinely bad model? They can flatter the loss, so keep reporting accuracy and a proper calibration measure alongside it. Low confidence everywhere is not the same as being well calibrated.
Recap in one screen
A hard label asserts one class; a soft label gives a probability across classes.
Training on hard labels with cross-entropy pushes a model towards unwarranted certainty.
Soft targets come from multiple annotators, label smoothing, distillation, or mixup-style augmentation.
They usually cost a little training accuracy and buy calibration and noise tolerance.
Check calibration with a reliability diagram; fix it with temperature scaling.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Overview”?
When we train a classification model, we need to provide it with labeled data. The way we represent these labels can have a significant impact on how the model learns. This lab explores two common approaches: Hard Labelling and Soft Labelling . By comparing them side-by-side, you can build a strong intuition for why representing uncertainty can lead to more robust and nuanced models.
What does this module say about “Defining the Labels”?
Imagine you're training a model to classify the sentiment of a tweet as 'Negative', 'Neutral', or 'Positive'. How you tell the model the "correct" answer for each tweet is where labelling strategy comes in.
What does this module say about “Hard Labelling (One-Hot Encoding)”?
This is the most common approach. You are 100% certain about the class. The correct class gets a value of 1, and all other classes get a 0. It's a "winner-takes-all" method. Example Vector: For a 'Positive' tweet, the hard label is [0, 0, 1] . This tells the model, "This tweet is positive, and nothing else." Soft Labelling (Probabilistic)
Cheat sheet
Hard vs Soft Labelling
When we train a classification model, we need to provide it with labeled data. The way we represent these labels can have a significant impact on how the model learns. This lab explores two common approaches: Hard Labelling and Soft Labelling. By comparing them side-by-side, you can build a strong intuition for why representing uncertainty can lead to more robust and nuanced models.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.