Layer Normalization

The formula, row instead of column

LayerNorm: y_i = (x_i − μ_row) / σ_row

μ_row and σ_row are the mean and standard deviation of that one sample's own feature vector. Compare to BatchNorm's μ_col, σ_col — computed down a column, across samples. Same shape of formula, different axis.

Normalising across features, not across the batch

Batch normalisation standardises each feature using statistics computed across the examples in a batch. Layer normalisation does the opposite: it standardises each example using statistics computed across its own features.

x̂ = (x − μ_example) / √(σ²_example + ε)    out = γx̂ + β

That change of axis has one enormous practical consequence: the computation does not depend on the batch at all. Each example is normalised in isolation.

Why transformers use it

Three reasons, all following from batch independence.

Sequences vary in length. Batch normalisation would compute statistics across positions in a batch, mixing a sentence's third token with another's third token — and with padding, mixing real tokens with padding tokens. Layer normalisation normalises each token's own feature vector, which is a meaningful operation.

Inference is often batch size 1. Generating text one request at a time gives no batch to compute statistics from. Layer normalisation needs none.

Training and inference behave identically. There are no running averages to accumulate, no train()/eval() distinction for this layer, and no risk of a train/serve mismatch.

Together those made layer normalisation the obvious choice for sequence models, and it has been standard since the original transformer.

Pre-norm versus post-norm

Where the normalisation sits relative to the residual connection turns out to matter a great deal.

Post-norm — the original transformer — is x + sublayer(x) followed by normalisation. It works, and beyond about 12 layers it needs careful warm-up to train at all.

Pre-norm is x + sublayer(norm(x)). The normalisation is inside the residual branch, so the identity path from input to output is completely clean — no normalisation touches it. Gradients flow back through that path unchanged, and very deep transformers become trainable.

Every large language model uses pre-norm, and it is one of the small architectural details that made scaling to 100 layers practical.

RMSNorm is the further simplification now common in large models: it divides by the root mean square of the features and skips the mean subtraction entirely. One fewer statistic to compute, no accuracy cost in practice, and measurably faster at scale.

The same idea, along the other axis

LayerNorm normalises across the features of one row instead of across the batch. That one change makes it independent of batch size, which is why every transformer uses it.

import numpy as np

rng = np.random.default_rng(0)
x = rng.normal(0, 1, (4, 6)) * np.array([1.0, 5.0, 0.2, 3.0, 1.0, 0.5]) + 2.0
np.set_printoptions(precision=4, suppress=True)
print("4 rows, 6 features:")
print(x)
print()

def batchnorm(a, eps=1e-5):
    return (a - a.mean(axis=0)) / np.sqrt(a.var(axis=0) + eps)

def layernorm(a, eps=1e-5):
    return (a - a.mean(axis=1, keepdims=True)) / np.sqrt(a.var(axis=1, keepdims=True) + eps)

bn, ln = batchnorm(x), layernorm(x)
print("BatchNorm -- each COLUMN gets mean 0, sd 1:")
print("  column means:", np.round(bn.mean(axis=0), 8))
print("  row means   :", np.round(bn.mean(axis=1), 4), " <- not controlled")
print()
print("LayerNorm -- each ROW gets mean 0, sd 1:")
print("  row means   :", np.round(ln.mean(axis=1), 8))
print("  column means:", np.round(ln.mean(axis=0), 4), " <- not controlled")
print()
print("same formula, different axis. that is the entire difference.")
print()

print("and here is what it buys. process the SAME row in three batch sizes:")
row = x[:1]
for k in (1, 2, 4):
    b = x[:k]
    print("  batch of %d:" % k)
    print("     BatchNorm gives", np.round(batchnorm(b)[0], 4))
    print("     LayerNorm gives", np.round(layernorm(b)[0], 4))
print("  LayerNorm returns the same numbers every time. BatchNorm does not,")
print("  and at batch size 1 it collapses to zeros because a single row has")
print("  no variance to divide by.")
print()

print("that independence is why LayerNorm won in sequence models:")
print("  - sequences have variable length, so a 'batch' is ragged")
print("  - inference is often batch size 1, and the answer must not change")
print("  - there is no running mean to keep, so train and eval are identical")
print("  - it works unchanged when the batch is split across 8 GPUs")
print()

print("it has gamma and beta too, one pair per feature:")
gamma = np.array([1.0, 1.0, 2.0, 1.0, 0.5, 1.0])
beta = np.array([0.0, 0.0, 0.0, 1.0, 0.0, 0.0])
out = gamma * ln + beta
print("  out[0] =", np.round(out[0], 4))
print("  note gamma and beta are per-FEATURE even though the normalisation")
print("  was per-row. the statistics are shared along one axis, the learned")
print("  parameters along the other.")
print()

print("RMSNorm, which newer models use, drops the mean subtraction entirely:")
def rmsnorm(a, eps=1e-6):
    return a / np.sqrt((a ** 2).mean(axis=1, keepdims=True) + eps)
print("  LayerNorm(x)[0] =", np.round(ln[0], 4))
print("  RMSNorm(x)[0]   =", np.round(rmsnorm(x)[0], 4))
print("  it only rescales, never recentres. one less reduction per call,")
print("  and in practice the mean subtraction turned out not to matter much.")
print()

print("where the norm sits relative to the residual is its own decision:")
print("  post-norm: x + Sublayer(x), then normalise. the original transformer.")
print("  pre-norm:  x + Sublayer(norm(x)). the skip path never passes through")
print("             a norm, so the gradient reaches layer 1 untouched --")
print("             which is what made very deep transformers trainable")
print("             without a long warmup.")

Guided experiments

1. Start at None. Four samples with deliberately different overall magnitudes — Sample B in particular is roughly 500x the scale of Sample C. Per-column stats are dominated by whichever sample happens to be largest.
2. Switch to LayerNorm. Every row now has mean 0 and standard deviation 1, regardless of its original scale — the per-row stats confirm it, and the wildly different starting magnitudes stop mattering to whatever layer reads this next.
3. Switch to BatchNorm instead. Now the per-column stats read 0/1, but the rows keep their very different scales — this is the complementary normalization, useful when the batch is a meaningful, stable group.
4. Shrink the batch to one sample. Tick Batch Size = 1. Under BatchNorm, a single sample has no variation to normalize against — its standard deviation is 0, and the column stats break down. Under LayerNorm nothing changes at all, because it never depended on the other samples in the first place.

Summing up

BatchNorm and LayerNorm are the same normalization idea applied to different axes: BatchNorm down a column, across the batch; LayerNorm across a row, within one sample. That single difference in axis is why LayerNorm works identically at any batch size, including one, while BatchNorm's statistics depend on the batch it happens to see — which is why transformers and RNNs, which often run with small or variable batches, use LayerNorm almost universally.

The other members of the family

GroupNorm is the one to know for computer vision. It splits channels into groups (32 is the usual default) and normalises within each group, per example. Being batch-independent, it works at batch size 2 — which is exactly the situation in detection and segmentation, where large images force small batches. It is the standard replacement for batch normalisation there.

Note that GroupNorm with one group is LayerNorm, and with one channel per group is InstanceNorm. They are points on the same axis.

In code, and the shape that trips people up

import torch.nn as nn

norm = nn.LayerNorm(512)              # normalise the last dimension, size 512
x = torch.randn(8, 100, 512)          # (batch, sequence, features)
out = norm(x)                         # each of the 800 tokens normalised separately

nn.LayerNorm(512) normalises over the last dimension only, which for a transformer is the feature dimension — correct. You can pass a tuple to normalise over several trailing dimensions, which is what image applications sometimes need.

The mistake to avoid is normalising over the wrong axis. nn.LayerNorm([100, 512]) on the tensor above would normalise across sequence positions as well as features, mixing tokens together — not what a transformer wants, and it produces a model that trains poorly for no obvious reason.

Parameter count is 2 per normalised feature: 1,024 for a 512-dimensional layer. Negligible, and like batch normalisation's parameters they should be excluded from weight decay.

Questions people ask

Which should I use? LayerNorm for anything sequential or transformer-shaped; BatchNorm for CNNs with reasonable batch sizes; GroupNorm for CNNs with small batches.

Does LayerNorm need a model.eval() switch? Not for its own behaviour — it computes the same thing in both modes. You still need eval() for dropout and any batch normalisation elsewhere.

Why does it help at all? It keeps activations in a stable range through depth, which keeps gradients in a usable range and allows higher learning rates. The precise theoretical account is still debated, exactly as with batch normalisation.

Is pre-norm always better? For deep networks, yes in practice. Post-norm sometimes reaches slightly better final quality on shallow models with careful tuning.

What is RMSNorm's advantage? One fewer statistic, so slightly faster, with no measurable loss in quality. Standard in recent large language models.

Can I use both BatchNorm and LayerNorm? In one model, technically yes, and it is unusual. Pick the one that matches the data's structure.

Recap in one screen

• LayerNorm standardises each example across its own features, so it is completely independent of the batch.
• That makes it correct for variable-length sequences and for batch size 1, and identical in training and inference.
• Transformers use it, in the pre-norm arrangement, which keeps the residual path clean and makes deep stacks trainable.
• GroupNorm is the batch-independent choice for vision with small batches.
• RMSNorm drops the mean subtraction and is now common in large language models.