Quick Context
Regularization is any technique that constrains the model to prevent overfitting — learning the training data too well at the expense of new, unseen data. In neural networks, common regularization methods include L1/L2 weight penalties, dropout, data augmentation, and early stopping.
1) Why Overfitting Happens
Neural networks are extremely flexible function approximators. A network with millions of parameters can easily memorize every sample in the training set, including noise and outliers. The result: near-zero training loss but poor performance on new data. Regularization adds constraints that favor simpler, generalizable solutions.
2) L2 Regularization (Weight Decay)
Adds a penalty proportional to the squared magnitude of weights to the loss function:
L_total = L_original + λ · Σ wᵢ²
This encourages small weights, which means simpler decision boundaries. λ (lambda) controls the strength — too high and the model underfits, too low and overfitting persists. Typical range: 1e-4 to 1e-2.
3) L1 Regularization (Lasso)
Adds a penalty proportional to the absolute value of weights:
L_total = L_original + λ · Σ |wᵢ|
L1 drives many weights to exactly zero, effectively performing feature selection. Less common in deep learning than L2, but useful when sparsity is desired.
4) Comparison of Regularization Techniques
| Technique | How It Works | When to Use |
| L2 (Weight Decay) | Penalizes large weights | Default choice; always try first |
| L1 (Lasso) | Drives weights to zero | When you want sparse features |
| Dropout | Randomly zeros neurons | Fully-connected layers |
| Early Stopping | Stops before overfitting | Always; no hyperparameter beyond patience |
| Data Augmentation | Expands effective dataset | Image/text tasks |
5) Guided Experiments
- Train without any regularization — observe the train/val loss gap (overfitting).
- Add L2 regularization (weight decay) — the gap should shrink.
- Increase λ progressively — at some point the model starts underfitting (both losses increase).
- Compare L1 vs. L2 — observe how L1 drives more weights to near-zero.
6) Common Mistakes
- Too much regularization — the model can't learn anything useful (underfitting).
- Regularizing the bias terms — biases should typically be excluded from weight decay.
- Stacking too many regularization techniques — using dropout + L2 + data augmentation + early stopping may be overkill for simple problems.
- Not monitoring the train/val gap — this is how you know whether more or less regularization is needed.
7) Key Takeaways
- Regularization prevents overfitting by constraining model complexity.
- L2 (weight decay) is the default — it penalizes large weights and is built into most optimizers.
- The right amount of regularization balances the train/val loss gap without increasing both losses.
- Combine techniques thoughtfully — more is not always better.