Watch the training flow! Unscaled data causes first-layer weights to explode (Red), while subsequent deep layers vanish (Blue). Toggle Batch Normalization to dynamically rescale intermediate outputs, locking the deep gradients into stability (Green).
35
80k
Gradient Health
Global Status
STABLE
Waiting to start...
Average Weight Magnitude
W1 (Input → H1)1.000
W3 (H2 → H3)1.000
Color Legend
Vanish (~0)Stable (~1)Explode (>2)
Drag to Pan | Scroll to Zoom
Overview
Quick Context
Batch Normalization (BatchNorm) was added by Ioffe & Szegedy in 2015. It is now a usual part of nearly each deep net. It scales the inputs to each layer so that training is faster and more stable. This page lets you see what happens with and without it.
The step is simple: for each feature in each layer, subtract the batch mean and divide by the batch std dev, then scale and shift by two learned params γ and β. The subtraction and division centre the data. γ and β let the net recover any scale and offset it needs. So BatchNorm does not force outputs to zero mean and unit spread — it just gives the solver a friendlier start.
In training, the mean and spread come from the current mini-batch. At eval time, they come from running means saved in training. That switch is what net.train() and net.eval() control in PyTorch. And forgetting to call net.eval() at test time is one of the most common BatchNorm bugs.
BatchNorm lets you use more learning rates, since the scaling stops outputs from growing or shrinking across layers. Without it, a high rate can push outputs into the flat zones of sigmoid or tanh, where gradients vanish. With it, outputs stay in the steep zone, where gradients flow.
It also acts as a mild noise source. Since each row's scaling depends on the other rows in the batch, the net sees slightly other inputs on each pass. That noise is close to dropout. It cuts overfitting. And in practice nets with BatchNorm often need less dropout or none at all.
The batch-size link is BatchNorm's main flaw. Small batches give noisy guesses of the mean and spread, which hurts training. For tasks where small batches are forced — object detection, fine-grained work, training on a single GPU — Layer Norm, Group Norm and Instance Norm compute stats over other axes and avoid the batch link.
Batch Normalization in Deep Networks: Complete Guide
Read this after trying the interactive once, then run it again with the suggested experiments.
Quick Context
Batch Normalization (BatchNorm) was added by Ioffe & Szegedy in 2015. It is now a usual part of nearly each deep net. It scales the inputs to each layer so that training is faster and more stable. This page lets you see what happens with and without it.
The step is simple: for each feature in each layer, subtract the batch mean and divide by the batch std dev, then scale and shift by two learned params γ and β. The subtraction and division centre the data. γ and β let the net recover any scale and offset it needs. So BatchNorm does not force outputs to zero mean and unit spread — it just gives the solver a friendlier start.
In training, the mean and spread come from the current mini-batch. At eval time, they come from running means saved in training. That switch is what net.train() and net.eval() control in PyTorch. And forgetting to call net.eval() at test time is one of the most common BatchNorm bugs.
BatchNorm lets you use more learning rates, since the scaling stops outputs from growing or shrinking across layers. Without it, a high rate can push outputs into the flat zones of sigmoid or tanh, where gradients vanish. With it, outputs stay in the steep zone, where gradients flow.
It also acts as a mild noise source. Since each row's scaling depends on the other rows in the batch, the net sees slightly other inputs on each pass. That noise is close to dropout. It cuts overfitting. And in practice nets with BatchNorm often need less dropout or none at all.
The batch-size link is BatchNorm's main flaw. Small batches give noisy guesses of the mean and spread, which hurts training. For tasks where small batches are forced — object detection, fine-grained work, training on a single GPU — Layer Norm, Group Norm and Instance Norm compute stats over other axes and avoid the batch link.
1) The Problem BatchNorm Solves
When you feed raw features with other scales (e.g., Age 18–80 vs. Salary 20k–200k) into a net, the weights that get the large feature grow too fast. In backprop those large weights boost gradients in early layers (exploding). And deeper layers get fewer updates (vanishing). The result: shaky training or very slow gains.
Scaling inputs at the front door helps the first layer. But internal shift — the spread of each hidden layer's inputs keeps moving as weights update — still hurts deeper layers. BatchNorm fixes this by re-scaling outputs inside the net at each layer.
2) How Batch Normalization Works
For a mini-batch of activations z at a given layer:
# batch mean # batch variance # normalize # scale & shift (learnable) - ε is a tiny constant (e.g., 1e-5) to avoid division by zero. - γ and β are learnable params that let the network recover any scale/shift it needs — BatchNorm doesn't permanently force zero-mean unit-variance. It just gives the optimizer a much friendlier starting point. - At inference time, the running mean and variance computed in training are used instead of batch statistics.
3) Why This Is Worth Learning
Faster gains — you can often use much more learning rates.
Noise effect — the noise from batch stats acts as mild smoothing, often cutting the need for Dropout.
Depth works — gradients stay in a healthy range across 50, 100. Or still 1000+ layers.
Interview staple — expect questions on BatchNorm in almost each ML/DL interview.
Normalising the activations inside the network
Feature scaling puts the inputs on a common footing. Batch norm does the same thing to the outputs between layers. And does it on each step in training.
For each channel, over the current mini-batch:
x̂ = (x − μbatch) / √(σ²batch + ε) out = γx̂. + β
The first step sets mean to 0 and spread to 1. The second step is what makes it a layer rather than a fixed transform: γ and β are learned. So the net can rescale and shift the scaled vals — and can undo the scaling fully if that turns out to be better.
Two params per channel. That is why batch norm is almost free in param count while being one of the most useful layers on hand.
What it buys
More learning rates. The main real gain. Scaled outputs keep gradients in a sane range. So you can train at rates that would blow up without it — often 10× more.
Faster gains. Fewer epochs to the same loss. Always.
Depth works. Nets beyond about 20 layers were shaky before batch norm and residual links. Together they made 100+ layers routine.
Mild noise. Each row's scaling depends on which other rows shared its batch. That injects noise. It is why nets with batch norm often need less dropout.
Less fuss with init. A layer that scales its inputs cares much less about the scale it was set up with.
The first story was "cutting internal shift". Later work argued that the real cause is smoothing the loss surface. The debate is open and does not change how you use it.
Train and inference behave differently
This is where the bugs are. In training, stats come from the current batch. At eval time, there may be no batch — a single pic must give a fixed answer — so the layer uses running means saved from training.
That means the layer has two modes. And forgetting to switch is a real and common error:
model.train() # batch stats, running means updated
# ... training loop ...
model.eval() # running means used, not updated
with torch.no_grad():
predictions = model(x)
Signs of getting it wrong: predictions that shift depending on what else is in the batch. Or val score far below train score for no clear reason. Or a net that acts differently on a batch of one.
The momentum param controls how fast the running means update (0.1 in PyTorch means each batch adds 10%). If your data spread shifts in training, the running stats lag behind.
Where it fails
Small batches. With a batch of 2 or 4, the batch stats are noise. This is a real limit in detection and fine-grained tasks. There large pics force small batches — and the reason GroupNorm exists.
Sequence nets. Sequences vary in length. Scaling across the batch mixes spots that are not alike. LayerNorm replaced it fully in transformers.
Batch size 1. The spread is zero and the layer cannot work.
Shift between train and eval. The running means describe the training data. If live data differs, the scaling is wrong.
Alternative
Normalises over
Use when
BatchNorm
Batch, per channel
CNNs with fair batch sizes
LayerNorm
All features of one row
Transformers, RNNs
GroupNorm
Groups of channels, one row
Small batches in vision
InstanceNorm
Each channel of one row
Style transfer, gen nets
RMSNorm
Scale only, no centring
Modern large language nets
4) Guided Experiments with This Interactive
Baseline — Set Age ≈ 35, Salary ≈ 80k. Leave both toggles OFF. Click Train Network. Watch W1 (first layer) turn red and W3 (deep layer) turn blue. The status should say "Unstable".
Input normalization just — Turn ON "Normalize Raw Inputs", leave Batch Norm OFF. Train again. W1 calms down, but deep layers may still drift.
Batch Normalization — Turn ON "Enable Batch Norm" (keep input normalization ON too). Train again. All bars should stay green and status should read "Stable".
Extreme inputs — Set Salary to 200k with just Batch Norm ON (no input normalization). Observe how BatchNorm alone handles large feature scales internally.
Reset & reproduce — Click Reset and repeat experiment 3 to confirm the result is consistent.
5) Where BatchNorm Is Placed
There are two common conventions:
Before activation: Linear → BatchNorm → ReLU (the first paper's way).
After activation: Linear → ReLU → BatchNorm (some practitioners prefer this).
In practice both work well. Consistency within your design matters more than the placement choice.
6) Common Mistakes
Forgetting net.eval() — at test time, if you don't switch to eval mode, the framework will use live batch stats instead of the running means, giving shaky results.
Batch size too small — with batch size 1 or 2, the batch mean/spread guess is too noisy. Use Group Norm or Layer Norm instead.
Mixing BatchNorm with Dropout — both add noise. Their mix can hurt. Test whether you need both.
Adding BatchNorm to the output layer — scaling the final layer often hurts scores.
7) BatchNorm vs. Other Normalizations
Technique
Normalizes Over
Best For
Batch Norm
Batch dimension
CNNs with large batches
Layer Norm
Feature dimension
Transformers, RNNs
Group Norm
Channel groups
Small-batch CNNs
Instance Norm
Single sample, single channel
Style transfer
8) Key Takeaways
BatchNorm scales hidden-layer outputs with batch stats in training and running means at test time.
It fights internal shift, letting you use more learning rates and deeper designs.
The learned params γ and β let the net undo the scaling if needed.
Always switch to eval mode at test time. And mind the batch size.
Placement, and the redundant bias
The usual order is Conv → BatchNorm → ReLU. Scaling before the act fn keeps the pre-outputs centred. So roughly half the ReLU units are active — which is what the init assumed.
Two details that appear in real code:
The conv bias is not needed. Batch norm subtracts the mean. It cancels any constant the conv added. That is why ResNet-style code uses bias=False on convs followed by norm — it saves params and changes nothing.
Pre-act ordering trains deeper. In residual blocks, BN → ReLU → Conv leaves the skip path fully clean. It was shown to train deeper nets more reliably.
At test time, batch norm can be folded into the conv: since both are linear at that point, their weights can be mixed into a single conv. Each deploy tool does this. And it removes the layer's run cost fully.
Real guidance
Use it in conv nets unless batches are small.
Use LayerNorm in transformers and RNNs. Batch norm is not the right tool there.
Use GroupNorm when batch size is under about 8. It is free of batch size and nearly as good.
Keep it in eval mode when a pretrained backbone is frozen, or its running stats will drift on your small data and quietly hurt the features.
Exclude γ and β from weight decay. Pulling them toward zero fights what the layer is for.
Shuffle your training data. Batches sorted by class give useless stats.
Scale it, then let the net undo it
BatchNorm rescales each feature to zero mean and unit spread, then gives the net two params to change that back. Both halves matter. And the train/eval gap is where most bugs live.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
# a batch of 8 rows, 4 features on wildly different scales
x = np.column_stack([rng.normal(100, 15, 8), rng.normal(0, 0.01, 8),
rng.normal(-5, 2, 8), rng.normal(1000, 300, 8)])
print("a batch of %d rows, 4 features on very different scales:" % len(x))
print(" per-feature mean:", np.round(x.mean(axis=0), 4))
print(" per-feature sd :", np.round(x.std(axis=0), 4))
print()
mu = x.mean(axis=0)
var = x.var(axis=0)
xhat = (x - mu) / np.sqrt(var + 1e-5)
print("normalise each COLUMN across the batch:")
print(" after: mean", np.round(xhat.mean(axis=0), 10))
print(" after: sd ", np.round(xhat.std(axis=0), 6))
print(" every feature now arrives at the next layer on the same scale.")
print()
print(" except feature 1, which lands at %.4f rather than 1.0. its true sd"
% xhat.std(axis=0)[1])
print(" is %.4f -- the same order as the eps of 1e-5 we added for safety,"
% x.std(axis=0)[1])
print(" so eps is a meaningful part of the denominator rather than a rounding")
print(" guard. that is worth knowing: on near-constant features BatchNorm")
print(" quietly stops normalising, and eps is what stops it dividing by zero.")
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
gamma = np.array([1.0, 2.0, 0.5, 1.0])
beta = np.array([0.0, 0.0, 3.0, -1.0])
out = gamma * xhat + beta
print("then two LEARNED parameters per feature put the scale back if useful:")
print(" gamma (scale) ", gamma)
print(" beta (shift) ", beta)
print(" output mean ", np.round(out.mean(axis=0), 4), " <- equals beta")
print(" output sd ", np.round(out.std(axis=0), 4),
" <- |gamma|, give or take the eps effect above")
print()
print("without gamma and beta, BatchNorm would force every layer's output to")
print("be zero-mean and unit-variance whether that helps or not. with them,")
print("the network can learn gamma=sd and beta=mean and recover the original")
print("distribution exactly -- so normalisation costs it nothing it needs.")
print()
print("the axis matters, and getting it wrong is a classic bug:")
print(" normalising over rows (correct, per feature):")
print(" ", np.round(((x - x.mean(axis=0)) / x.std(axis=0)).mean(axis=0), 6))
print(" normalising over columns (wrong -- that is LayerNorm's axis):")
wrong = (x - x.mean(axis=1, keepdims=True)) / x.std(axis=1, keepdims=True)
print(" per-feature mean is now", np.round(wrong.mean(axis=0), 4))
print(" it mixed a feature measured in thousands with one measured in")
print(" hundredths, inside a single row. that is not the same operation.")
print()
Output
The next block covers the left steps.
example_03.pyNumPy
import numpy as np
print("TRAINING vs INFERENCE -- the part that breaks in production.")
print("at training time the statistics come from the batch. at inference")
print("there may be only one row, so a running average is kept instead:")
running_mu, running_var, mom = np.zeros(4), np.ones(4), 0.1
for step in range(1, 6):
b = np.column_stack([rng.normal(100, 15, 8), rng.normal(0, 0.01, 8),
rng.normal(-5, 2, 8), rng.normal(1000, 300, 8)])
running_mu = (1 - mom) * running_mu + mom * b.mean(axis=0)
running_var = (1 - mom) * running_var + mom * b.var(axis=0)
print(" after batch %d: running mean[0] = %8.3f (batch had %8.3f)"
% (step, running_mu[0], b.mean(axis=0)[0]))
print(" true mean of feature 0 is 100. after 5 batches the estimate is %.3f"
% running_mu[0])
print(" it takes roughly 1/momentum batches to converge, which is why a")
print(" model evaluated too early can score badly for no other reason.")
print()
print("and the consequence people trip over -- at training time a prediction")
print("depends on the OTHER rows in its batch:")
one = x[:1]
print(" row 0 alone, normalised by its own batch of 1:")
print(" ", np.round(((one - one.mean(axis=0)) / np.sqrt(one.var(axis=0) + 1e-5))[0], 4))
print(" the same row inside the batch of 8:")
print(" ", np.round(xhat[0], 4))
print(" a batch of 1 has zero variance, so BatchNorm outputs zeros. that is")
print(" why batch_size=1 breaks it, why you must call model.eval(), and why")
print(" small batches make it noisy.")
print()
print("that batch dependence is exactly what LayerNorm removes, which is why")
print("transformers use LayerNorm and convolutional nets still use BatchNorm.")
Output
Questions people ask
Does batch norm replace input scaling? No. Scale the inputs as well. The first layer gains from the same care as the rest.
Why does my net do worse in eval mode? Often the running stats are poor — too few training steps, or data not shuffled. Or a spread that shifted in training.
Can I use it with dropout? Yes, though the mix is often not needed. If you use both, dropout goes after the norm.
How many params does it add? Two per channel, plus two non-trained running stats per channel.
Is it needed with residual links? They solve other problems and are often used together. Dropping norm from a deep ResNet often breaks training unless the init is tuned to make up for it.
What is the ε for? To avoid dividing by zero when a channel's spread is tiny. The default of 1e-5 is fine.
Recap in one screen
Scale each channel over the batch, then rescale with two learned params.
The main gain is that much more learning rates turn stable. Training is faster and depth works.
It acts differently in training and at test time — call net.eval(). Or results depend on the batch.
Small batches break it; use GroupNorm there. And LayerNorm for sequences and transformers.
Drop the prior layer's bias, exclude the params from weight decay. And fold the layer away at deploy time.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Quick Context”?
What does this module say about “The Problem BatchNorm Solves”?
Cheat sheet
Batch Normalization in Deep Networks
Toggle batch normalisation on a deep network and watch it rescale layer outputs to keep gradients stable as they flow back.
DEEP LEARNING · vizlearn.in/deep_learning/batch_normalization.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.