When we train a classification model, we need to provide it with labeled data. The way we represent these labels can have a significant impact on how the model learns. This lab explores two common approaches: Hard Labelling and Soft Labelling. By comparing them side-by-side, you can build a strong intuition for why representing uncertainty can lead to more robust and nuanced models.
Defining the Labels
Imagine you're training a model to classify the sentiment of a tweet as 'Negative', 'Neutral', or 'Positive'. How you tell the model the "correct" answer for each tweet is where labelling strategy comes in.
Guided Experiments
Select different tweets from the list to see how these labelling strategies handle different scenarios.
-
The Clear-Cut Case:
Select the second tweet: "Absolutely stunning. Every detail is perfect. Highly recommend!" This is unambiguously positive. Notice that the hard label [0, 0, 1] and the soft label [0.01, 0.04, 0.95] are very similar. In such clear cases, both methods convey almost the same information. The model is strongly encouraged to predict 'Positive'.
-
The Ambiguous Case:
Now, select the first tweet: "The design is gorgeous, but the software is a total nightmare to navigate." This tweet contains both positive ('gorgeous design') and negative ('nightmare software') elements.
- The Hard Label is forced to choose one winner. Here, it's classified as 'Negative' ([1, 0, 0]), completely ignoring the positive aspect. This forces the model to be overconfident and lose valuable information.
- The Soft Label, however, captures the nuance perfectly with a vector like [0.65, 0.1, 0.25]. It tells the model: "This is mostly negative, but there's a significant positive component too."
-
The Neutral-Leaning Case:
Select the last tweet: "Shipping was late, but the box was packed really well." Again, we see a mix of sentiments. The hard label might classify this as 'Neutral' ([0, 1, 0]), but the soft label [0.4, 0.45, 0.15] provides a much richer signal, indicating it leans neutral but has a strong negative element ('late shipping') and a minor positive one ('packed well').
Why Use Soft Labelling?
While hard labelling is simpler, soft labelling offers several key advantages, a technique often referred to as Label Smoothing:
- Better Generalization: By preventing the model from becoming overconfident in its predictions for the training data, soft labelling helps it generalize better to new, unseen data. It acts as a form of regularization.
- Robustness to Noise: Real-world datasets often contain mislabeled examples. Soft labels can account for this uncertainty, making the training process more stable and less susceptible to noisy data.
- Capturing Inter-Class Relationships: Soft labels can implicitly teach the model about the relationships between classes. For example, it can learn that 'sarcastic' is closer to 'negative' than it is to 'positive'.
- Knowledge Distillation: Soft labels are the foundation of knowledge distillation, where the probabilistic outputs of a large, complex model are used as soft targets to train a smaller, more efficient model.