Watch how penalties affect the network parameters!
Ridge (L2) smoothly shrinks weights toward zero to prevent overfitting.
Lasso (L1) drops useless connections EXACTLY to zero (Sparsity). Select both for Elastic Net.
0.20
Model Parameters
Regularization Mode
UNREGULARIZED
No penalties. The model is likely to overfit and keep noise parameters.
Active Params
72 / 72
Avg Magnitude
2.150
Parameter Sparsity (% Active)100%
Visual Legend
Dropped (0.0)Stable (~1.0)Large (>2.5)
Drag to Pan | Scroll to Zoom
Overview
Penalising complexity
An overfitting net has found a way to fit noise. And doing that almost always needs large weights — sharp, wiggly functions need big coefficients. Regularisation exploits that by adding the size of the weights to the loss:
total loss = data loss + λ × cost(w)
Now the optimiser has two objectives in tension: fit the data. And keep the weights small. λ sets the exchange rate. At λ = 0 there is no cost and you are back to plain training. At very large λ the weights are crushed toward zero and the net underfits. L2 regularisation (weight decay) penalises the sum of squared weights, pulling each weight proportionally toward zero — nothing reaches zero. But the function turns smoother. L1 regularisation penalises the sum of absolute vals, pulling with constant force no matter magnitude. It drives small weights just to zero and does feature choice. Dropout takes a other way fully: in each training step it randomly sets a fraction of neuron outputs to zero, forcing the network to distribute information across many neurons rather than relying on any one — an implicit ensemble of thinned networks. Early stopping monitors validation loss and halts training when it begins to rise, with the validation curve as a proxy for the complexity the net has absorbed. This page shows the decision edge with and without each technique, lets you adjust λ and dropout rate. And covers batch normalisation's implicit regularisation effect.
Watch how penalties affect the network parameters!
Add a penalty on the size of the weights and the model prefers simpler explanations. L1 drives weights to exactly zero and selects features; L2 shrinks them all smoothly and is the usual default.
Penalising complexity
An overfitting net has found a way to fit noise. And doing that almost always needs large weights — sharp, wiggly functions need big coefficients. Regularisation exploits that by adding the size of the weights to the loss:
total loss = data loss + λ × cost(w)
Now the optimiser has two objectives in tension: fit the data. And keep the weights small. λ sets the exchange rate. At λ = 0 there is no cost and you are back to plain training. At very large λ the weights are crushed toward zero and the net underfits. L2 regularisation (weight decay) penalises the sum of squared weights, pulling each weight proportionally toward zero — nothing reaches zero. But the function turns smoother. L1 regularisation penalises the sum of absolute vals, pulling with constant force no matter magnitude. It drives small weights just to zero and does feature choice. Dropout takes a other way fully: in each training step it randomly sets a fraction of neuron outputs to zero, forcing the network to distribute information across many neurons rather than relying on any one — an implicit ensemble of thinned networks. Early stopping monitors validation loss and halts training when it begins to rise, with the validation curve as a proxy for the complexity the net has absorbed. This page shows the decision edge with and without each technique, lets you adjust λ and dropout rate. And covers batch normalisation's implicit regularisation effect.
L1 and L2, and why one is sparse
L2 (ridge, weight decay) penalises the sum of squared weights:
cost = Σ wi² → gradient = 2wi
L1 (lasso) penalises the sum of absolute vals:
cost = Σ |wi| → gradient = sign(wi)
The gradients explain the gap fully. L2’s pull is proportional to the weight. So as a weight ways zero the force pulling it there fades — weights shrink toward zero and never quite arrive. L1’s pull is constant no matter size. So a weight near zero is pushed just as hard as a large one and is driven just to zero. There it stays.
That is why L1 gives sparse nets and does feature choice, while L2 keeps each feature with a less coefficient. Elastic net uses both, getting L1’s choice with L2’s stability when features are correlated.
Constraining the net on purpose
A network with millions of params can fit almost any training set just, with its noise. Regularisation is any deliberate constraint that makes that harder, trading a little training accuracy for better generalisation.
The ways on hand, and what each really does:
Method
Mechanism
Typical setting
Weight decay (L2)
Penalises large weights
0.01–0.1 with AdamW
L1
Penalises absolute weights, drives some to zero
Rare in networks
Dropout
Randomly zeroes activations
0.1–0.5
Early stopping
Stops at the validation minimum
Patience of 5–20 epochs
Data augmentation
Expands the effective dataset
Task-specific
Batch normalisation
Normalises, and adds batch noise
Standard in CNNs
Label smoothing
Softens the targets
ε = 0.1
Mixup / CutMix
Blends examples and labels
Images
Gradient clipping
Caps the update size
max_norm 1.0
Just some of these are regularisers in the strict sense. A few are stabilisers that happen to reduce overfitting. In practice they are used together and their effects overlap.
Weight decay, and what it does geometrically
Adding a cost on the squared size of the weights changes the objective:
loss = prediction error + λ × Σw²
The gradient of the cost is 2λw. So each step pulls each weight slightly towards zero — hence the name "decay". Weights survive just if the data justifies them.
Two build notes that matter.
Use AdamW, not Adam with weight_decay. Plain Adam adds the cost into the gradient. It then gets divided by the adaptive scaling — so params with small gradients are penalised more. It was never the intent. AdamW applies the decay right to the weights, decoupled from the scaling.
Exclude biases and normalisation params. Decaying a batch-norm γ towards zero fights what the layer is for. And biases carry no capacity worth penalising.
decay, no_decay = [], []
for name, p in model.named_parameters():
(no_decay if p.ndim <= 1 or "bias" in name else decay).append(p)
opt = torch.optim.AdamW([
{"params": decay, "weight_decay": 0.01},
{"params": no_decay, "weight_decay": 0.0},
], lr=3e-4)
That one-line rule — params with one dimension or fewer are exempt — captures biases and all normalisation params. And is what most reference implementations use.
What to reach for first
Not all at once. A sensible order:
Early stopping — free. And it needs no tuning beyond a patience value.
Weight decay — cheap, always applicable. And it should in effect always be on.
Data augmentation — the largest single gain on pic and audio tasks with limited data.
Transfer learning — if a pretrained net exists for your domain, this beats each regulariser on a small dataset.
Dropout — if a gap stays after the above.
A less net — last, since it also lowers the ceiling.
Stacking regularisers has diminishing and sometimes negative returns. Batch normalisation now injects noise. So heavy dropout on top of it often hurts. Add one at a time and measure.
Six ways to stop a net memorising
L2, L1, dropout, early stopping, augmentation and more data all fight overfitting by other mechanisms. Here they are on one problem. So the gaps are visible rather than described.
Now see how L1 differs: it drives weights to just zero and does feature choice.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
NOISE, DEG = 0.4, 18
def make(n):
x = rng.uniform(-3, 3, n)
return x, np.sin(1.5 * x) + rng.normal(0, NOISE, n)
x_tr, y_tr = make(22)
x_te, y_te = make(2000)
SCALE = np.linalg.norm(np.vander(x_tr, DEG + 1), axis=0)
def design(xv):
return np.vander(xv, DEG + 1) / SCALE
A, A_te = design(x_tr), design(x_te)
L = np.linalg.eigvalsh(2 * A.T @ A / len(A)).max()
def score(w):
return ((A @ w - y_tr) ** 2).mean(), ((A_te @ w - y_te) ** 2).mean()
def lasso(lam, steps=8000):
lr, w = 0.9 / L, np.zeros(DEG + 1)
for _ in range(steps):
w -= lr * 2 * A.T @ (A @ w - y_tr) / len(A)
w = np.sign(w) * np.maximum(np.abs(w) - lr * lam, 0)
return w
for lam in (1e-3, 1e-1):
w = lasso(lam)
tr, te = score(w)
print("L1 lambda=%.0e train=%.4f test=%.4f %d/%d weights=0"
% (lam, tr, te, (w == 0).sum(), DEG + 1))
print("\nL1 pushes weights to EXACTLY zero -- it selects features, not just shrinks them.")
Output
Finally, check dropout, early stopping and data augmentation to see a other mechanism at work.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
NOISE, DEG = 0.4, 18
def make(n):
x = rng.uniform(-3, 3, n)
return x, np.sin(1.5 * x) + rng.normal(0, NOISE, n)
x_tr, y_tr = make(22)
x_te, y_te = make(2000)
SCALE = np.linalg.norm(np.vander(x_tr, DEG + 1), axis=0)
def design(xv):
return np.vander(xv, DEG + 1) / SCALE
A, A_te = design(x_tr), design(x_te)
L = np.linalg.eigvalsh(2 * A.T @ A / len(A)).max()
def score(w):
return ((A @ w - y_tr) ** 2).mean(), ((A_te @ w - y_te) ** 2).mean()
def row(name, w, extra=""):
tr, te = score(w)
print("%-28s train=%.4f test=%.4f max|w|=%.2e %s"
% (name, tr, te, np.abs(w).max(), extra))
r = np.random.default_rng(1)
lr_d, w_d = 0.5 / L, np.zeros(DEG + 1)
for _ in range(8000):
Ad = A * ((r.random(A.shape) > 0.3) / 0.7)
w_d -= lr_d * 2 * Ad.T @ (Ad @ w_d - y_tr) / len(A)
row("dropout p=0.3", w_d)
lr, w = 0.9 / L, np.zeros(DEG + 1)
best_te, best_w, best_step = np.inf, w.copy(), 0
for step in range(1, 60001):
w -= lr * 2 * A.T @ (A @ w - y_tr) / len(A)
if step % 250 == 0:
te = ((A_te @ w - y_te) ** 2).mean()
if te < best_te:
best_te, best_w, best_step = te, w.copy(), step
row("early stopping", best_w, "step %d" % best_step)
aug_x = np.repeat(x_tr, 30) + rng.normal(0, 0.15, 22*30)
row("augmentation (30x)", np.linalg.lstsq(design(aug_x), np.repeat(y_tr, 30), rcond=None)[0])
x_m, y_m = make(400)
row("400 real points",
np.linalg.lstsq(design(x_m), y_m, rcond=None)[0])
print("\npenalties crush weight size; augmentation adds information instead.")
Output
Guided experiments
Train with no cost. Set Cost Strength (λ) to its min and press Train Network. Weights grow freely and the net fits the targets as closely as it can, noise included.
Apply L2. Enable the L2 toggle, set Cost Strength (λ) to about 0.1, press Reset & Randomize Targets and train. Each weight shrinks, and none reaches zero — that smooth shrinkage is the whole behaviour of L2.
Switch to L1. Enable the L1 toggle instead at the same λ. Now some weights collapse to just zero while others stay large. The net has selected a subset of connections and switched the rest off.
Turn it up too far. Set Cost Strength (λ) to 0.8 and train. Nearly all is crushed toward zero and the net can no longer fit still the real structure — regularisation causing underfitting. It is the failure mode at the far end.
Weight decay is not quite L2
The two are used interchangeably and are just equivalent for plain SGD. Weight decay multiplies the weights by a factor slightly below 1 at each step. L2 adds a term to the loss. So its contribution passes through the optimiser’s gradient machinery.
With Adam that gap is real. Adam divides each gradient by a running estimate of its magnitude. It also rescales the L2 term — so params with large gradients end up effectively less regularised. AdamW fixes this by applying the decay right to the weights, outside the adaptive step. And it is the reason AdamW is now the default optimiser for transformers.
What often goes wrong
Regularising a net that is underfitting. If training loss is now high, adding a cost makes both curves worse. Confirm the net can overfit before trying to stop it.
Penalising the biases. Biases shift the output rather than scale the input. So shrinking them limits what the net can represent without reducing overfitting. Regularise weights just.
With L2 with Adam and expecting weight decay. Use AdamW when you want true decay.
Failing to scale features first. The cost treats all weights equally. So a feature on a large scale needs a small weight and is under-penalised relative to the rest. Standardise before regularising.
Tuning λ linearly. It spans orders of magnitude. Search it on a log scale, often 10−5 to 10−1.
In one line
Regularisation adds a weight-size cost to the loss so the optimiser trades fit against simplicity, with λ setting the rate. L2 shrinks all weights proportionally and keeps each feature. L1 applies constant pressure and drives weights to just zero, giving a sparse, self-selecting net. Use it when the net overfits, scale the features first. And reach for AdamW if the optimiser is Adam.
Knowing whether it worked
Regularisation is tuned by watching two curves, not by reasoning about the settings.
Too little: training loss keeps falling while validation loss rises. The gap widens with epochs.
About right: both curves flatten near each other. And validation reaches its best value late in training.
Too much: both curves plateau high, and training loss never gets low. The net is being stopped from learning the signal along with the noise.
That last case is real and easy to miss, since "the net is not overfitting" looks like success. Check the training loss as well: if the net cannot fit its own training data, the constraint is too tight.
The other measurement worth making is the effective one: does validation performance improve? A regulariser that reduces the gap without improving validation has reached nothing except making training worse.
Implicit regularisation
A few things regularise without being labelled as such. And they explain why large networks generalise better than their param counts suggest.
Stochastic gradient noise. Mini-batch gradients are noisy estimates. And that noise biases the optimiser towards flatter minima. It tend to generalise better than sharp ones. Very large batches reduce this noise. It is part of why they sometimes generalise slightly worse.
Early stopping is implicit L2. For linear nets this is provable: stopping gradient descent early is equivalent to an L2 cost of a particular strength.
Design it. Convolution's weight sharing and locality are constraints — a strong prior that pic statistics are translation-invariant. That is why CNNs beat transformers on small pic datasets: the constraint is doing work that the transformer must learn from data.
Param sharing in recurrent and graph networks does the same along their own axes.
Questions people ask
Which regulariser matters most? On pics, data augmentation. On tabular data, weight decay and early stopping. Across the board, more data.
What weight decay value? 0.01 with AdamW is a reasonable default. 0.1 for large transformers. Tune it on a log scale.
Do I need dropout with batch normalisation? Often not. If you use both, less the dropout rate.
Does regularisation slow training? In epochs, often yes. In final quality on unseen data, that is the point.
Can I regularise too much? Yes — the symptom is a training loss that will not come down.
Is a bigger net with more regularisation better than a less one? Often, yes, especially with pretrained weights. Capacity plus constraint often beats a small net outright.
Recap in one screen
Regularisation trades training accuracy for generalisation by constraining the net.
Weight decay is the baseline — use AdamW. And exclude biases and normalisation params.
Early stopping is free. Augmentation is the biggest win on pics. Transfer learning beats them all on small datasets.
Add one at a time and judge by validation performance, not by the size of the gap alone.
Batch noise, early stopping and the design it all regularise implicitly.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Penalising complexity”?
What does this module say about “L1 and L2, and why one is sparse”?
Cheat sheet
Regularization Techniques
Compare L1 and L2 penalties, and watch Lasso zero out useless weights while Ridge shrinks all of them smoothly.
DEEP LEARNING · vizlearn.in/deep_learning/regularization_in_neural_networks.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.