Watch the training flow! Unscaled data causes first-layer weights to explode (Red), while subsequent deep layers vanish (Blue). Toggle Batch Normalization to dynamically rescale intermediate outputs, locking the deep gradients into stability (Green).
Overview
Quick Context
Batch Normalization (BatchNorm) was introduced by Ioffe & Szegedy in 2015 and is now a standard building block in virtually every deep network. It normalizes the inputs to each layer so that training is faster and more stable. This page lets you see what happens with and without it.
35
80k
Gradient Health
Global Status
STABLE
Waiting to start...
Average Weight Magnitude
W1 (Input → H1)1.000
W3 (H2 → H3)1.000
Color Legend
Vanish (~0)Stable (~1)Explode (>2)
Drag to Pan | Scroll to Zoom
Batch Normalization in Deep Networks: Complete Guide
Read this after trying the interactive once, then run it again with the suggested experiments.
1) The Problem BatchNorm Solves
When you feed raw features with different scales (e.g., Age 18–80 vs. Salary 20k–200k) into a network, the first-layer weights that receive the large feature grow disproportionately. During backpropagation those large weights amplify gradients in early layers (exploding), while deeper layers receive progressively smaller updates (vanishing). The result: unstable training or very slow convergence.
Normalizing inputs at the front door helps the first layer, but internal covariate shift — the distribution of each hidden layer's inputs keeps changing as weights update — still destabilizes deeper layers. BatchNorm fixes this by re-normalizing activations inside the network at every layer.
2) How Batch Normalization Works
For a mini-batch of activations z at a given layer:
# batch mean # batch variance # normalize # scale & shift (learnable) - ε is a tiny constant (e.g., 1e-5) to avoid division by zero. - γ and β are learnable parameters that let the network recover any scale/shift it needs — BatchNorm doesn't permanently force zero-mean unit-variance; it just gives the optimizer a much friendlier starting point. - At inference time, the running mean and variance computed during training are used instead of batch statistics.
3) Why This Is Worth Learning
Faster convergence — you can often use much higher learning rates.
Regularization effect — the noise from batch statistics acts as mild regularization, sometimes reducing the need for Dropout.
Enables deeper networks — gradients stay in a healthy range across 50, 100, or even 1000+ layers.
Interview staple — expect questions on BatchNorm in almost every ML/DL interview.
Normalising the activations inside the network
Feature scaling puts the inputs on a common footing. Batch normalisation does the same thing to the activations between layers, and does it continuously during training.
For each channel, over the current mini-batch:
x̂ = (x − μbatch) / √(σ²batch + ε) out = γx̂ + β
The first step standardises to mean 0 and variance 1. The second step is what makes it a layer rather than a fixed transform: γ and β are learned, so the network can rescale and shift the normalised values — and can undo the normalisation entirely if that turns out to be better.
Two parameters per channel, which is why batch normalisation is almost free in parameter count while being one of the most impactful layers available.
What it buys
Higher learning rates. The main practical benefit. Normalised activations keep gradients in a sensible range, so you can train at rates that would diverge without it — often 10× higher.
Faster convergence. Fewer epochs to the same loss, consistently.
Depth becomes trainable. Networks beyond about 20 layers were unreliable before batch normalisation and residual connections; together they made 100+ layers routine.
Mild regularisation. Each example's normalisation depends on which other examples happened to share its batch, which injects noise. That is why models with batch normalisation often need less dropout.
Less sensitivity to initialisation. A layer that normalises its inputs cares much less about the scale it was initialised with.
The original explanation was "reducing internal covariate shift". Later work argued that the real mechanism is smoothing the loss landscape. The debate is unresolved and does not change how you use it.
Train and inference behave differently
This is where the bugs are. During training, statistics come from the current batch. During inference, there may be no batch — a single image must give a deterministic answer — so the layer uses running averages accumulated during training.
That means the layer has two modes, and forgetting to switch is a genuine and common error:
model.train() # batch statistics, running averages updated
# ... training loop ...
model.eval() # running averages used, not updated
with torch.no_grad():
predictions = model(x)
Symptoms of getting it wrong: predictions that change depending on what else is in the batch, validation accuracy far below training accuracy for no other reason, or a model that behaves differently on a batch of one.
The momentum parameter controls how fast the running averages update (0.1 in PyTorch means each batch contributes 10%). If your data distribution shifts during training, the running statistics lag behind.
Where it fails
Small batches. With a batch of 2 or 4, the batch statistics are noise. This is a real constraint in detection and segmentation, where large images force small batches — and the reason GroupNorm exists.
Sequence models. Sequences vary in length and normalising across the batch mixes positions that are not comparable. LayerNorm replaced it entirely in transformers.
Batch size 1. The variance is zero and the layer cannot function.
Distribution shift between train and inference. The running averages describe the training data; if production data differs, the normalisation is wrong.
Alternative
Normalises over
Use when
BatchNorm
Batch, per channel
CNNs with reasonable batch sizes
LayerNorm
All features of one example
Transformers, RNNs
GroupNorm
Groups of channels, one example
Small batches in vision
InstanceNorm
Each channel of one example
Style transfer, generative models
RMSNorm
Scale only, no centring
Modern large language models
4) Guided Experiments with This Interactive
Baseline — Set Age ≈ 35, Salary ≈ 80k. Leave both toggles OFF. Click Train Network. Watch W1 (first layer) turn red and W3 (deep layer) turn blue. The status should say "Unstable".
Input normalization only — Turn ON "Normalize Raw Inputs", leave Batch Norm OFF. Train again. W1 calms down, but deep layers may still drift.
Batch Normalization — Turn ON "Enable Batch Norm" (keep input normalization ON too). Train again. All bars should stay green and status should read "Stable".
Extreme inputs — Set Salary to 200k with only Batch Norm ON (no input normalization). Observe how BatchNorm alone handles large feature scales internally.
Reset & reproduce — Click Reset and repeat experiment 3 to confirm the result is consistent.
5) Where BatchNorm Is Placed
There are two common conventions:
Before activation: Linear → BatchNorm → ReLU (the original paper's approach).
After activation: Linear → ReLU → BatchNorm (some practitioners prefer this).
In practice both work well; consistency within your architecture matters more than the placement choice.
6) Common Mistakes
Forgetting model.eval() — at inference, if you don't switch to eval mode, PyTorch/TensorFlow will use live batch stats instead of running averages, producing inconsistent predictions.
Batch size too small — with batch size 1 or 2, the batch mean/variance estimate is too noisy. Use Group Normalization or Layer Normalization instead.
Using BatchNorm with Dropout carelessly — both introduce stochasticity; their interaction can hurt. Test whether you need both.
Applying BatchNorm to the output layer — normalizing the final prediction layer often harms performance.
7) BatchNorm vs. Other Normalizations
Technique
Normalizes Over
Best For
Batch Norm
Batch dimension
CNNs with large batches
Layer Norm
Feature dimension
Transformers, RNNs
Group Norm
Channel groups
Small-batch CNNs
Instance Norm
Single sample, single channel
Style transfer
8) Key Takeaways
BatchNorm normalizes hidden-layer activations using batch statistics during training and running statistics during inference.
It fights internal covariate shift, enabling higher learning rates and deeper architectures.
The learnable parameters γ and β let the network undo the normalization if needed.
Always switch to eval mode at inference and be mindful of batch size.
Placement, and the redundant bias
The conventional order is Conv → BatchNorm → ReLU. Normalising before the activation keeps the pre-activations centred, so roughly half the ReLU units are active — which is what the initialisation assumed.
Two details that appear in real code:
The convolution's bias is redundant. Batch normalisation subtracts the mean, which cancels any constant the convolution added. That is why ResNet-style implementations use bias=False on convolutions followed by normalisation — it saves parameters and changes nothing.
Pre-activation ordering trains deeper. In residual blocks, BN → ReLU → Conv leaves the identity path completely clean, which was shown to train deeper networks more reliably than the original post-activation arrangement.
At inference, batch normalisation can be folded into the preceding convolution: since both are linear at that point, their weights can be combined into a single convolution. Every deployment toolkit does this, and it removes the layer's runtime cost entirely.
Practical guidance
Use it in convolutional networks unless batches are small.
Use LayerNorm in transformers and RNNs. Batch normalisation is not the right tool there.
Use GroupNorm when batch size is under about 8. It is independent of batch size and nearly as effective.
Keep it in eval mode when a pretrained backbone is frozen, or its running statistics will drift on your small dataset and quietly degrade the features.
Exclude γ and β from weight decay. Decaying them towards zero fights what the layer is for.
Shuffle your training data. Batches sorted by class give meaningless statistics.
Normalise, then let the network undo it
BatchNorm rescales each feature to zero mean and unit variance, then gives the network two parameters to change that back. Both halves matter, and the train/eval difference is where most bugs live.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
# a batch of 8 rows, 4 features on wildly different scales
x = np.column_stack([rng.normal(100, 15, 8), rng.normal(0, 0.01, 8),
rng.normal(-5, 2, 8), rng.normal(1000, 300, 8)])
print("a batch of %d rows, 4 features on very different scales:" % len(x))
print(" per-feature mean:", np.round(x.mean(axis=0), 4))
print(" per-feature sd :", np.round(x.std(axis=0), 4))
print()
mu = x.mean(axis=0)
var = x.var(axis=0)
xhat = (x - mu) / np.sqrt(var + 1e-5)
print("normalise each COLUMN across the batch:")
print(" after: mean", np.round(xhat.mean(axis=0), 10))
print(" after: sd ", np.round(xhat.std(axis=0), 6))
print(" every feature now arrives at the next layer on the same scale.")
print()
print(" except feature 1, which lands at %.4f rather than 1.0. its true sd"
% xhat.std(axis=0)[1])
print(" is %.4f -- the same order as the eps of 1e-5 we added for safety,"
% x.std(axis=0)[1])
print(" so eps is a meaningful part of the denominator rather than a rounding")
print(" guard. that is worth knowing: on near-constant features BatchNorm")
print(" quietly stops normalising, and eps is what stops it dividing by zero.")
print()
gamma = np.array([1.0, 2.0, 0.5, 1.0])
beta = np.array([0.0, 0.0, 3.0, -1.0])
out = gamma * xhat + beta
print("then two LEARNED parameters per feature put the scale back if useful:")
print(" gamma (scale) ", gamma)
print(" beta (shift) ", beta)
print(" output mean ", np.round(out.mean(axis=0), 4), " <- equals beta")
print(" output sd ", np.round(out.std(axis=0), 4),
" <- |gamma|, give or take the eps effect above")
print()
print("without gamma and beta, BatchNorm would force every layer's output to")
print("be zero-mean and unit-variance whether that helps or not. with them,")
print("the network can learn gamma=sd and beta=mean and recover the original")
print("distribution exactly -- so normalisation costs it nothing it needs.")
print()
print("the axis matters, and getting it wrong is a classic bug:")
print(" normalising over rows (correct, per feature):")
print(" ", np.round(((x - x.mean(axis=0)) / x.std(axis=0)).mean(axis=0), 6))
print(" normalising over columns (wrong -- that is LayerNorm's axis):")
wrong = (x - x.mean(axis=1, keepdims=True)) / x.std(axis=1, keepdims=True)
print(" per-feature mean is now", np.round(wrong.mean(axis=0), 4))
print(" it mixed a feature measured in thousands with one measured in")
print(" hundredths, inside a single row. that is not the same operation.")
print()
print("TRAINING vs INFERENCE -- the part that breaks in production.")
print("at training time the statistics come from the batch. at inference")
print("there may be only one row, so a running average is kept instead:")
running_mu, running_var, mom = np.zeros(4), np.ones(4), 0.1
for step in range(1, 6):
b = np.column_stack([rng.normal(100, 15, 8), rng.normal(0, 0.01, 8),
rng.normal(-5, 2, 8), rng.normal(1000, 300, 8)])
running_mu = (1 - mom) * running_mu + mom * b.mean(axis=0)
running_var = (1 - mom) * running_var + mom * b.var(axis=0)
print(" after batch %d: running mean[0] = %8.3f (batch had %8.3f)"
% (step, running_mu[0], b.mean(axis=0)[0]))
print(" true mean of feature 0 is 100. after 5 batches the estimate is %.3f"
% running_mu[0])
print(" it takes roughly 1/momentum batches to converge, which is why a")
print(" model evaluated too early can score badly for no other reason.")
print()
print("and the consequence people trip over -- at training time a prediction")
print("depends on the OTHER rows in its batch:")
one = x[:1]
print(" row 0 alone, normalised by its own batch of 1:")
print(" ", np.round(((one - one.mean(axis=0)) / np.sqrt(one.var(axis=0) + 1e-5))[0], 4))
print(" the same row inside the batch of 8:")
print(" ", np.round(xhat[0], 4))
print(" a batch of 1 has zero variance, so BatchNorm outputs zeros. that is")
print(" why batch_size=1 breaks it, why you must call model.eval(), and why")
print(" small batches make it noisy.")
print()
print("that batch dependence is exactly what LayerNorm removes, which is why")
print("transformers use LayerNorm and convolutional nets still use BatchNorm.")
Output
Questions people ask
Does batch normalisation replace input scaling? No. Scale the inputs as well; the first layer benefits from the same treatment as the rest.
Why does my model perform worse in eval mode? Usually the running statistics are poor — too few training steps, unshuffled data, or a distribution that shifted during training.
Can I use it with dropout? Yes, though the combination is often unnecessary. If you use both, dropout goes after the normalisation.
How many parameters does it add? Two per channel, plus two non-trained running statistics per channel.
Is it needed with residual connections? They solve different problems and are normally used together. Removing normalisation from a deep ResNet usually breaks training unless the initialisation is adjusted to compensate.
What is the ε for? To avoid dividing by zero when a channel's variance is tiny. The default of 1e-5 is fine.
Recap in one screen
Standardise each channel over the batch, then rescale with two learned parameters.
The main gain is that much higher learning rates become stable, so training is faster and depth is feasible.
It behaves differently in training and inference — call model.eval(), or predictions depend on the batch.
Small batches break it; use GroupNorm there, and LayerNorm for sequences and transformers.
Drop the preceding layer's bias, exclude the parameters from weight decay, and fold the layer away at deployment.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
The conventional order is Conv → BatchNorm → ReLU . Normalising before the activation keeps the pre-activations centred, so roughly half the ReLU units are active — which is what the initialisation assumed.
What does this module say about “Quick Context”?
Batch Normalization (BatchNorm) was introduced by Ioffe & Szegedy in 2015 and is now a standard building block in virtually every deep network. It normalizes the inputs to each layer so that training is faster and more stable. This page lets you see what happens with and without it.
What does this module say about “The Problem BatchNorm Solves”?
When you feed raw features with different scales (e.g., Age 18–80 vs. Salary 20k–200k) into a network, the first-layer weights that receive the large feature grow disproportionately. During backpropagation those large weights amplify gradients in early layers ( exploding ), while deeper layers receive progressively smaller updates ( vanishing ). The result: unstable training or very slow convergence.
Cheat sheet
Batch Normalization in Deep Networks
Batch Normalization (BatchNorm) was introduced by Ioffe & Szegedy in 2015 and is now a standard building block in virtually every deep network. It normalizes the inputs to each layer so that training is faster and more stable. This page lets you see what happens with and without it.
DEEP LEARNING · vizlearn.in/deep_learning/batch_normalization.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.