Layer Normalization

What LayerNorm changes and why it matters

BatchNorm normalizes each feature — each column — with the mean and standard deviation computed across each sample now in the batch. That is powerful, and it has one structural weakness: its statistics depend on which other cases happen to be in the batch with you. It turns a real problem at batch size 1 and in designs like transformers where "the batch" is not a stable, meaningful group.

LayerNorm normalizes each sample against it — the mean and standard deviation are computed across that one row's own features. So no other sample's presence or absence changes the answer.

That independence from the batch is the whole point. It is why each transformer uses LayerNorm and not BatchNorm. And it is why LayerNorm behaves identically at training and inference time — there are no running statistics to track. This page checks the two axis choices side by side, shows the formula and the learned affine params. And covers the Pre-LN versus Post-LN placement that finds whether training is stable at large depth. The code blocks below apply both normalisation axes to the same tensor so you can see just which statistics change and which stay the same.

The formula, row instead of column

LayerNorm: y_i = (x_i − μ_row) / σ_row

μ_row and σ_row are the mean and standard deviation of that one sample's own feature vector. Check to BatchNorm's μ_col, σ_col — computed down a column, across samples. Same shape of formula, other axis.

Normalising across features, not across the batch

Batch normalisation standardises each feature with statistics computed across the cases in a batch. Layer normalisation does the opposite: it standardises each case with statistics computed across its own features.

x̂ = (x − μ_case) / √(σ²_case + ε)    out = γx̂ + β

That change of axis has one enormous real consequence: the computation does not depend on the batch at all. Each case is normalised in isolation.

Why transformers use it

Three reasons, all next from batch independence.

Sequences vary in length. Batch normalisation would compute statistics across positions in a batch, mixing a sentence's third token with another's third token — and with padding, mixing real tokens with padding tokens. Layer normalisation normalises each token's own feature vector, which is a meaningful step.

Inference is often batch size 1. Making text one request at a time gives no batch to compute statistics from. Layer normalisation needs none.

Training and inference behave identically. There are no running averages to accumulate, no train()/eval() distinction for this layer. And no risk of a train/serve mismatch.

Together those made layer normalisation the obvious choice for sequence nets. And it has been standard since the first transformer.

Pre-norm versus post-norm

Where the normalisation sits relative to the residual connection turns out to matter a great deal.

Post-norm — the first transformer — is x + sublayer(x) followed by normalisation. It works, and beyond about 12 layers it needs careful warm-up to train at all.

Pre-norm is x + sublayer(norm(x)). The normalisation is inside the residual branch. So the identity path from input to output is fully clean — no normalisation touches it. Gradients flow back through that path unchanged. And very deep transformers turn trainable.

Each large language net uses pre-norm. And it is one of the small architectural details that made scaling to 100 layers real.

RMSNorm is the more simplification now common in large nets: it divides by the root mean square of the features and skips the mean subtraction fully. One fewer statistic to compute, no accuracy cost in practice. And measurably faster at scale.

The same idea, along the other axis

LayerNorm normalises across the features of one row instead of across the batch. That one change makes it independent of batch size. It is why each transformer uses it.

import numpy as np

rng = np.random.default_rng(0)
x = rng.normal(0, 1, (4, 6)) * np.array([1.0, 5.0, 0.2, 3.0, 1.0, 0.5]) + 2.0
np.set_printoptions(precision=4, suppress=True)
print("4 rows, 6 features:")
print(x)
print()

def batchnorm(a, eps=1e-5):
    return (a - a.mean(axis=0)) / np.sqrt(a.var(axis=0) + eps)

def layernorm(a, eps=1e-5):
    return (a - a.mean(axis=1, keepdims=True)) / np.sqrt(a.var(axis=1, keepdims=True) + eps)

bn, ln = batchnorm(x), layernorm(x)
print("BatchNorm -- each COLUMN gets mean 0, sd 1:")
print("  column means:", np.round(bn.mean(axis=0), 8))
print("  row means   :", np.round(bn.mean(axis=1), 4), " <- not controlled")
print()
print("LayerNorm -- each ROW gets mean 0, sd 1:")
print("  row means   :", np.round(ln.mean(axis=1), 8))
print("  column means:", np.round(ln.mean(axis=0), 4), " <- not controlled")
print()
print("same formula, different axis. that is the entire difference.")
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

print("and here is what it buys. process the SAME row in three batch sizes:")
row = x[:1]
for k in (1, 2, 4):
    b = x[:k]
    print("  batch of %d:" % k)
    print("     BatchNorm gives", np.round(batchnorm(b)[0], 4))
    print("     LayerNorm gives", np.round(layernorm(b)[0], 4))
print("  LayerNorm returns the same numbers every time. BatchNorm does not,")
print("  and at batch size 1 it collapses to zeros because a single row has")
print("  no variance to divide by.")
print()

print("that independence is why LayerNorm won in sequence models:")
print("  - sequences have variable length, so a 'batch' is ragged")
print("  - inference is often batch size 1, and the answer must not change")
print("  - there is no running mean to keep, so train and eval are identical")
print("  - it works unchanged when the batch is split across 8 GPUs")
print()

print("it has gamma and beta too, one pair per feature:")
gamma = np.array([1.0, 1.0, 2.0, 1.0, 0.5, 1.0])
beta = np.array([0.0, 0.0, 0.0, 1.0, 0.0, 0.0])
out = gamma * ln + beta
print("  out[0] =", np.round(out[0], 4))
print("  note gamma and beta are per-FEATURE even though the normalisation")
print("  was per-row. the statistics are shared along one axis, the learned")
print("  parameters along the other.")
print()

The next block covers the left steps.

import numpy as np

print("RMSNorm, which newer models use, drops the mean subtraction entirely:")
def rmsnorm(a, eps=1e-6):
    return a / np.sqrt((a ** 2).mean(axis=1, keepdims=True) + eps)
print("  LayerNorm(x)[0] =", np.round(ln[0], 4))
print("  RMSNorm(x)[0]   =", np.round(rmsnorm(x)[0], 4))
print("  it only rescales, never recentres. one less reduction per call,")
print("  and in practice the mean subtraction turned out not to matter much.")
print()

print("where the norm sits relative to the residual is its own decision:")
print("  post-norm: x + Sublayer(x), then normalise. the original transformer.")
print("  pre-norm:  x + Sublayer(norm(x)). the skip path never passes through")
print("             a norm, so the gradient reaches layer 1 untouched --")
print("             which is what made very deep transformers trainable")
print("             without a long warmup.")

Guided experiments

1. Start at None. Four samples with deliberately other total magnitudes — Sample B in particular is roughly 500x the scale of Sample C. Per-column stats are dominated by whichever sample happens to be largest.
2. Switch to LayerNorm. Each row now has mean 0 and standard deviation 1, no matter its first scale — the per-row stats confirm it. And the wildly other starting magnitudes stop mattering to whatever layer reads this next.
3. Switch to BatchNorm instead. Now the per-column stats read 0/1. But the rows keep their very other scales — this is the complementary normalization, useful when the batch is a meaningful, stable group.
4. Shrink the batch to one sample. Tick Batch Size = 1. Under BatchNorm, a single sample has no variation to normalize against — its standard deviation is 0. And the column stats break down. Under LayerNorm nothing changes at all, since it never depended on the other samples in the first place.

Summing up

BatchNorm and LayerNorm are the same normalization idea applied to other axes: BatchNorm down a column, across the batch. LayerNorm across a row, within one sample. That single gap in axis is why LayerNorm works identically at any batch size, with one, while BatchNorm's statistics depend on the batch it happens to see — which is why transformers and RNNs. It often run with small or var batches, use LayerNorm almost universally.

The other members of the family

GroupNorm is the one to know for computer vision. It splits channels into groups (32 is the usual default) and normalises within each group, per case. Being batch-independent, it works at batch size 2 — which is just the case in detection and segmentation. There large pics force small batches. It is the standard replacement for batch normalisation there.

Note that GroupNorm with one group is LayerNorm. And with one channel per group is InstanceNorm. They are points on the same axis.

In code, and the shape that trips people up

import torch.nn as nn

norm = nn.LayerNorm(512)              # normalise the last dimension, size 512
x = torch.randn(8, 100, 512)          # (batch, sequence, features)
out = norm(x)                         # each of the 800 tokens normalised separately

nn.LayerNorm(512) normalises over the last dimension just. It for a transformer is the feature dimension — correct. You can pass a tuple to normalise over a few trailing dimensions. It is what pic applications sometimes need.

The mistake to avoid is normalising over the wrong axis. nn.LayerNorm([100, 512]) on the tensor above would normalise across sequence positions as well as features, mixing tokens together — not what a transformer wants. And it gives a net that trains poorly for no obvious reason.

Param count is 2 per normalised feature: 1,024 for a 512-dimensional layer. Negligible, and like batch normalisation's params they should be excluded from weight decay.

Questions people ask

Which should I use? LayerNorm for anything sequential or transformer-shaped. BatchNorm for CNNs with reasonable batch sizes. GroupNorm for CNNs with small batches.

Does LayerNorm need a net.eval() switch? Not for its own behaviour — it computes the same thing in both modes. You still need eval() for dropout and any batch normalisation elsewhere.

Why does it help at all? It keeps activations in a stable range through depth. It keeps gradients in a usable range and lets more learning rates. The precise theoretical account is still debated, just as with batch normalisation.

Is pre-norm always better? For deep networks, yes in practice. Post-norm sometimes reaches slightly better final quality on shallow nets with careful tuning.

What is RMSNorm's advantage? One fewer statistic. So slightly faster, with no measurable loss in quality. Standard in recent large language nets.

Can I use both BatchNorm and LayerNorm? In one net, technically yes. And it is unusual. Pick the one that matches the data's structure.

Recap in one screen

• LayerNorm standardises each case across its own features. So it is fully independent of the batch.
• That makes it correct for var-length sequences and for batch size 1. And identical in training and inference.
• Transformers use it, in the pre-norm arrangement. It keeps the residual path clean and makes deep stacks trainable.
• GroupNorm is the batch-independent choice for vision with small batches.
• RMSNorm drops the mean subtraction and is now common in large language nets.