vizlearn

Regularization Techniques

Deep LearningBy Updated

Watch how penalties affect the network parameters! Ridge (L2) smoothly shrinks weights toward zero to prevent overfitting. Lasso (L1) drops useless connections EXACTLY to zero (Sparsity). Select both for Elastic Net.

Overview

Penalising complexity

An overfitting model has found a way to fit noise, and doing that almost always requires large weights — sharp, wiggly functions need big coefficients. Regularisation exploits that by adding the size of the weights to the loss:

total loss = data loss + λ × penalty(w)

Now the optimiser has two objectives in tension: fit the data, and keep the weights small. λ sets the exchange rate. At λ = 0 there is no penalty and you are back to plain training; at very large λ the weights are crushed toward zero and the model underfits.

0.20

Model Parameters

Regularization Mode
UNREGULARIZED
No penalties. The model is likely to overfit and keep noise parameters.
Active Params
72 / 72
Avg Magnitude
2.150
Parameter Sparsity (% Active) 100%

Visual Legend

Dropped (0.0) Stable (~1.0) Large (>2.5)
Drag to Pan | Scroll to Zoom

Regularization Techniques: A Practical Guide

Add a penalty on the size of the weights and the model prefers simpler explanations. L1 drives weights to exactly zero and selects features; L2 shrinks them all smoothly and is the usual default.

L1 and L2, and why one is sparse

L2 (ridge, weight decay) penalises the sum of squared weights:

penalty = Σ wi²   →   gradient = 2wi

L1 (lasso) penalises the sum of absolute values:

penalty = Σ |wi|   →   gradient = sign(wi)

The gradients explain the difference completely. L2’s pull is proportional to the weight, so as a weight approaches zero the force pulling it there fades — weights shrink toward zero and never quite arrive. L1’s pull is constant regardless of size, so a weight near zero is pushed just as hard as a large one and is driven exactly to zero, where it stays.

That is why L1 produces sparse models and performs feature selection, while L2 keeps every feature with a smaller coefficient. Elastic net uses both, getting L1’s selection with L2’s stability when features are correlated.

Constraining the model on purpose

A network with millions of parameters can fit almost any training set exactly, including its noise. Regularisation is any deliberate constraint that makes that harder, trading a little training accuracy for better generalisation.

The methods available, and what each actually does:

MethodMechanismTypical setting
Weight decay (L2)Penalises large weights0.01–0.1 with AdamW
L1Penalises absolute weights, drives some to zeroRare in networks
DropoutRandomly zeroes activations0.1–0.5
Early stoppingStops at the validation minimumPatience of 5–20 epochs
Data augmentationExpands the effective datasetTask-specific
Batch normalisationNormalises, and adds batch noiseStandard in CNNs
Label smoothingSoftens the targetsε = 0.1
Mixup / CutMixBlends examples and labelsImages
Gradient clippingCaps the update sizemax_norm 1.0

Only some of these are regularisers in the strict sense; several are stabilisers that happen to reduce overfitting. In practice they are used together and their effects overlap.

Weight decay, and what it does geometrically

Adding a penalty on the squared size of the weights changes the objective:

loss = prediction error + λ × Σw²

The gradient of the penalty is 2λw, so every step pulls each weight slightly towards zero — hence the name "decay". Weights survive only if the data justifies them.

Two implementation notes that matter.

Use AdamW, not Adam with weight_decay. Plain Adam adds the penalty into the gradient, which then gets divided by the adaptive scaling — so parameters with small gradients are penalised more, which was never the intent. AdamW applies the decay directly to the weights, decoupled from the scaling.

Exclude biases and normalisation parameters. Decaying a batch-norm γ towards zero fights what the layer is for, and biases carry no capacity worth penalising.

decay, no_decay = [], []
for name, p in model.named_parameters():
    (no_decay if p.ndim <= 1 or "bias" in name else decay).append(p)

opt = torch.optim.AdamW([
    {"params": decay,    "weight_decay": 0.01},
    {"params": no_decay, "weight_decay": 0.0},
], lr=3e-4)

That one-line rule — parameters with one dimension or fewer are exempt — captures biases and all normalisation parameters, and is what most reference implementations use.

What to reach for first

Not everything at once. A sensible order:

  1. Early stopping — free, and it needs no tuning beyond a patience value.
  2. Weight decay — cheap, always applicable, and it should essentially always be on.
  3. Data augmentation — the largest single gain on image and audio tasks with limited data.
  4. Transfer learning — if a pretrained model exists for your domain, this outperforms every regulariser on a small dataset.
  5. Dropout — if a gap remains after the above.
  6. A smaller model — last, because it also lowers the ceiling.

Stacking regularisers has diminishing and sometimes negative returns. Batch normalisation already injects noise, so heavy dropout on top of it often hurts. Add one at a time and measure.

Six ways to stop a model memorising

L2, L1, dropout, early stopping, augmentation and more data all fight overfitting by different mechanisms. Here they are on one problem, so the differences are visible rather than described.

example_01.pyNumPy
Output

                

                

                    
Guided experiments

                    
  1. Train with no penalty. Set Penalty Strength (λ) to its minimum and press Train Network. Weights grow freely and the model fits the targets as closely as it can, noise included.
  2. Apply L2. Enable the L2 toggle, set Penalty Strength (λ) to about 0.1, press Reset & Randomize Targets and train. Every weight shrinks, and none reaches zero — that smooth shrinkage is the whole behaviour of L2.
  3. Switch to L1. Enable the L1 toggle instead at the same λ. Now some weights collapse to exactly zero while others stay large. The model has selected a subset of connections and switched the rest off.
  4. Turn it up too far. Set Penalty Strength (λ) to 0.8 and train. Nearly everything is crushed toward zero and the model can no longer fit even the real structure — regularisation causing underfitting, which is the failure mode at the far end.

                

                

                    
Weight decay is not quite L2

                    
The two are used interchangeably and are only equivalent for plain SGD. Weight decay multiplies the weights by a factor slightly below 1 at each step; L2 adds a term to the loss, so its contribution passes through the optimiser’s gradient machinery.
With Adam that difference is real. Adam divides each gradient by a running estimate of its magnitude, which also rescales the L2 term — so parameters with large gradients end up effectively less regularised. AdamW fixes this by applying the decay directly to the weights, outside the adaptive step, and it is the reason AdamW is now the default optimiser for transformers.

                

                

                    
What usually goes wrong

                    
  • Regularising a model that is underfitting. If training loss is already high, adding a penalty makes both curves worse. Confirm the model can overfit before trying to stop it.
  • Penalising the biases. Biases shift the output rather than scale the input, so shrinking them limits what the model can represent without reducing overfitting. Regularise weights only.
  • Using L2 with Adam and expecting weight decay. Use AdamW when you want true decay.
  • Failing to scale features first. The penalty treats all weights equally, so a feature on a large scale needs a small weight and is under-penalised relative to the rest. Standardise before regularising.
  • Tuning λ linearly. It spans orders of magnitude; search it on a log scale, typically 10−5 to 10−1.

                

                

                    
In one line

                    
Regularisation adds a weight-size penalty to the loss so the optimiser trades fit against simplicity, with λ setting the rate. L2 shrinks all weights proportionally and keeps every feature; L1 applies constant pressure and drives weights to exactly zero, producing a sparse, self-selecting model. Use it when the model overfits, scale the features first, and reach for AdamW if the optimiser is Adam.

                

                

                    
Knowing whether it worked

                    
Regularisation is tuned by watching two curves, not by reasoning about the settings.
Too little: training loss keeps falling while validation loss rises. The gap widens with epochs.
About right: both curves flatten near each other, and validation reaches its best value late in training.
Too much: both curves plateau high, and training loss never gets low. The model is being prevented from learning the signal along with the noise.
That last case is real and easy to miss, because "the model is not overfitting" looks like success. Check the training loss as well: if the model cannot fit its own training data, the constraint is too tight.
The other measurement worth making is the effective one: does validation performance improve? A regulariser that reduces the gap without improving validation has achieved nothing except making training worse.

                

                

                    
Implicit regularisation

                    
Several things regularise without being labelled as such, and they explain why large networks generalise better than their parameter counts suggest.
Stochastic gradient noise. Mini-batch gradients are noisy estimates, and that noise biases the optimiser towards flatter minima, which tend to generalise better than sharp ones. Very large batches reduce this noise, which is part of why they sometimes generalise slightly worse.
Early stopping is implicit L2. For linear models this is provable: stopping gradient descent early is equivalent to an L2 penalty of a particular strength.
Architecture itself. Convolution's weight sharing and locality are constraints — a strong prior that image statistics are translation-invariant. That is why CNNs beat transformers on small image datasets: the constraint is doing work that the transformer must learn from data.
Parameter sharing in recurrent and graph networks does the same along their own axes.

                

                

                    
Questions people ask

                    
Which regulariser matters most? On images, data augmentation. On tabular data, weight decay and early stopping. Across the board, more data.
What weight decay value? 0.01 with AdamW is a reasonable default; 0.1 for large transformers. Tune it on a log scale.
Do I need dropout with batch normalisation? Often not. If you use both, lower the dropout rate.
Does regularisation slow training? In epochs, usually yes. In final quality on unseen data, that is the point.
Can I regularise too much? Yes — the symptom is a training loss that will not come down.
Is a bigger model with more regularisation better than a smaller one? Frequently, yes, especially with pretrained weights. Capacity plus constraint often beats a small model outright.

                

                

                    
Recap in one screen

                    
  • Regularisation trades training accuracy for generalisation by constraining the model.
  • Weight decay is the baseline — use AdamW, and exclude biases and normalisation parameters.
  • Early stopping is free; augmentation is the biggest win on images; transfer learning beats them all on small datasets.
  • Add one at a time and judge by validation performance, not by the size of the gap alone.
  • Batch noise, early stopping and the architecture itself all regularise implicitly.

                

            


        

    




        
    



Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
  1. Without scrolling back — what is the one-line takeaway from this module?
  2. What does this module say about “Penalising complexity”?
  3. What does this module say about “L1 and L2, and why one is sparse”?





 Cheat sheet
Regularization Techniques
Watch how penalties affect the network parameters! Ridge (L2) smoothly shrinks weights toward zero to prevent overfitting. Lasso (L1) drops useless connections EXACTLY to zero (Sparsity). Select both for Elastic Net.
DEEP LEARNING · vizlearn.in/deep_learning/regularization_in_neural_networks.html
More in Deep Learning
Deep LearningEarly StoppingDeep LearningHyperparameter TuningDeep LearningData SparsityDeep LearningNeural Network for Regression
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact