Networks train best when inputs live on a small, consistent scale. Compare raw, Min-Max, and Z-Score views of the same series — then throw in an outlier.
Overview
Normalising over the wrong axis
Every normalisation layer does the same arithmetic — subtract a mean, divide by a standard deviation, then apply a learned scale and shift. What differs is which values are pooled to compute that mean and deviation.
Batch normalisation computes statistics per feature, across the batch: for feature 3, average over all samples in the batch. Layer normalisation computes statistics per sample, across the features: for sample 3, average over all of its features.
On images batch norm is excellent. On sequences it breaks, for three separate reasons.
Technique
Replaces one sensor reading with a huge spike (420) and re-normalizes — watch what each technique does to the rest of the curve.
Series Statistics
Min
-
Max
-
Mean μ
-
Std σ
-
Formula
x' = x
Sensor Readings Over Time
RANGE: -
Raw values — note the y-axis scale. Large, drifting magnitudes make gradient updates uneven across features and slow training down.
Normalization Techniques for Sequential Data: A Practical Guide
Batch normalisation works beautifully on images and badly on sequences. Layer normalisation is what replaced it, and the reason is entirely about which axis the statistics are computed over.
Why batch norm fails on sequences
Variable length. Sequences in a batch have different lengths, so timestep 50 might have 32 real values in one batch and 3 in another, with the rest padding. Statistics computed over that are unstable, and computed over padding they are simply wrong.Per-timestep statistics. A recurrent network applies the same cell at every step, but batch norm would need separate running statistics for each timestep — and at inference on a sequence longer than anything seen in training, there are no statistics for those positions at all.Batch dependence. A sample’s normalised representation depends on the other samples it happens to be batched with. For generation, where inference often runs one sequence at a time, the batch statistics are meaningless.
Layer norm sidesteps all three. It uses only the sample’s own features, so it is independent of batch size, identical at training and inference, and unaffected by how many other sequences are present or how long they are.
Two different meanings of "normalisation"
The word covers two unrelated operations in sequence work, and conflating them causes confusion.
Data normalisation scales the input values — putting a temperature series and a price series on comparable footing before the model sees them.
Layer normalisation is an architectural component that standardises activations inside the network at every step.
Both matter for sequences, and for different reasons. This covers both.
Scaling sequence inputs
The requirement is the same as for tabular data: a feature ranging over thousands will dominate the first layer's weighted sum, and gradients for the other features become negligible.
The rule that is specific to sequences is fit the scaler on the training period only:
from sklearn.preprocessing import StandardScaler
split = int(len(series) * 0.7)
scaler = StandardScaler().fit(series[:split].reshape(-1, 1)) # past only
scaled = scaler.transform(series.reshape(-1, 1))
Computing the mean and standard deviation over the whole series uses future values to scale past ones, which is a leak. It inflates results and cannot be reproduced in production, where the future is unavailable by definition.
Method
Suits
Standardisation
The default for most sequences
Min-max
Bounded signals such as audio or normalised sensors
Robust (median / IQR)
Series with spikes and outliers
Log transform
Multiplicative series — prices, counts, traffic
Differencing
Series with a trend, especially for tree models
Differencing deserves a note: tree-based models cannot extrapolate beyond their training range, so on a trending series they will never predict a new high. Modelling the change rather than the level fixes it, and this is one of the most valuable transformations in applied forecasting.
Non-stationarity, the sequence-specific problem
A series whose statistics change over time is non-stationary, and a single scaler fitted once is wrong for the later part of it.
A stock price in 2015 and in 2025 differ in level and volatility; a website's traffic in its first month and its fifth year are different distributions. Scaling both with one mean and standard deviation puts the recent data far from zero.
Three responses:
Difference the series so the model works on changes, which are usually far more stationary than levels.
Use rolling normalisation — scale each window by its own statistics, so each example is locally normalised. This is standard in financial modelling, and it discards absolute level, which is sometimes exactly what you want and sometimes the signal.
Refit periodically, as part of a retraining schedule, and monitor for drift.
Note that a log transform handles a specific and common case: when a series grows multiplicatively, differencing the logarithm gives percentage changes, which are usually stationary where absolute changes are not.
Interactive Exploration Guide
Look at the unnormalised distribution. With the outlier toggle off, note the spread of the raw values — features on different scales pull the layer’s activations around.
Introduce an outlier. Enable the outlier toggle. Watch how a single extreme value shifts the computed mean and inflates the standard deviation, dragging every other value with it.
Compare the axes. Note which values get pooled. Normalising across the batch means one sample’s statistics depend on its neighbours; normalising across features means each sample is self-contained.
Toggle back and forth. The sensitivity to a single outlier is the practical argument for robust preprocessing before the layer ever sees the data.
The variants worth knowing
Batch norm — per feature, across the batch. Convolutional networks with reasonable batch sizes.
Layer norm — per sample, across features. Transformers and recurrent networks, essentially universally.
Group norm — per sample, across groups of channels. Vision with small batches, where batch norm’s estimates go noisy.
RMSNorm — layer norm without the mean subtraction, dividing by the root mean square only. Slightly cheaper, works as well, and is what most recent large language models use.
The placement matters too. Original transformers put layer norm after the residual addition (post-norm); modern ones put it before the sublayer (pre-norm), which makes deep stacks far easier to train and often removes the need for a learning-rate warmup.
Where this goes wrong
Batch norm with a batch size of 1 or 2. The variance estimate is meaningless. Use layer or group norm.
Forgetting eval mode. Batch norm uses batch statistics in training and running averages at inference. Leaving the model in training mode makes predictions depend on whatever else is in the batch.
Normalising over padded positions. Pollutes the statistics with zeros that carry no information.
Adding a bias before a normalisation layer. The mean subtraction cancels it exactly, so those parameters do nothing. Disable the bias on a layer feeding into a norm.
The short version
All normalisation layers subtract a mean and divide by a deviation; they differ only in which values are pooled. Batch norm pools across the batch, which makes it unusable for sequences of varying length and inference on single examples. Layer norm pools across each sample’s own features, so it is independent of batch size and identical at training and inference — which is why every transformer uses it.
Layer normalisation inside the network
For recurrent and transformer models, the internal normalisation choice is not the same as for convolutional networks, and the reason is structural.
Batch normalisation does not suit sequences. It computes statistics across the batch for each feature, which means mixing position 3 of one sentence with position 3 of another — and with padding, mixing real tokens with filler. It also needs a reasonable batch size, and it behaves differently at inference, which matters when serving one request at a time.
Layer normalisation standardises each example across its own features:
It is completely independent of the batch, works at batch size 1, handles variable lengths, and behaves identically in training and inference. That is why every transformer uses it.
Normalisation
Statistics over
Sequence-safe?
BatchNorm
The batch, per feature
No
LayerNorm
Features of one example
Yes
RMSNorm
Scale only, one example
Yes
GroupNorm
Channel groups, one example
For vision
RMSNorm is the simplification now common in large language models: divide by the root mean square and skip the mean subtraction. One fewer statistic, measurably faster at scale, no quality cost in practice.
Placement matters
In transformers, where the normalisation sits relative to the residual connection is a real architectural decision.
Post-norm — LayerNorm(x + sublayer(x)) — was the original arrangement. It works, and beyond roughly 12 layers it requires careful warm-up to train at all.
Pre-norm — x + sublayer(LayerNorm(x)) — puts the normalisation inside the residual branch, leaving the identity path completely clean. Gradients flow back through it unchanged, and very deep stacks become trainable.
Every large model uses pre-norm. It is one of the small details that made scaling to 80 or more layers practical.
For recurrent cells, layer normalisation applied to the gate pre-activations stabilises training and helps with the exploding cell states that otherwise saturate the output tanh.
Why BatchNorm loses to LayerNorm on sequences
Sequences are ragged, arrive one at a time, and vary in length. Each of those breaks BatchNorm in a specific way, and this measures all three on padded batches.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
B, T, D = 4, 6, 5
lengths = [6, 3, 5, 2]
x = rng.normal(0, 1.0, (B, T, D)) * 2 + 1
mask = np.zeros((B, T))
for i, L in enumerate(lengths):
mask[i, :L] = 1
x[i, L:] = 0.0 # padding
print("a batch of %d sequences, padded to length %d." % (B, T))
print(" real lengths: %s -> %d of %d positions are padding (%.0f%%)"
% (lengths, int((1 - mask).sum()), B * T,
100 * (1 - mask).sum() / (B * T)))
print()
def batchnorm(a, eps=1e-5):
# normalise over batch AND time, per feature -- the usual sequence variant
mu = a.mean(axis=(0, 1), keepdims=True)
sd = a.std(axis=(0, 1), keepdims=True)
return (a - mu) / (sd + eps)
def layernorm(a, eps=1e-5):
mu = a.mean(axis=-1, keepdims=True)
sd = a.std(axis=-1, keepdims=True)
return (a - mu) / (sd + eps)
print("PROBLEM 1 -- PADDING POLLUTES THE STATISTICS. BatchNorm averages over")
print("the batch, and the padding is in the batch:")
real = x[mask.astype(bool)]
print(" mean over REAL positions only : %s" % np.round(real.mean(0), 4))
print(" mean over everything incl. pad: %s" % np.round(x.mean(axis=(0, 1)), 4))
print(" sd over REAL positions only : %s" % np.round(real.std(0), 4))
print(" sd over everything incl. pad: %s" % np.round(x.std(axis=(0, 1)), 4))
print(" the padding is %d of %d rows of zeros, and it drags both toward 0."
% (int((1 - mask).sum()), B * T))
print(" LayerNorm never sees it -- it normalises within a position, so a")
print(" padded position is normalised on its own and then masked away.")
print()
print("PROBLEM 2 -- THE STATISTICS DEPEND ON THE BATCH. take sequence 0 and")
print("run it in three different batches:")
target = x[0:1]
for k in (1, 2, 4):
batch = x[:k]
bn = batchnorm(batch)[0, 0]
ln = layernorm(batch)[0, 0]
print(" batch of %d: BatchNorm -> %s" % (k, np.round(bn[:4], 4)))
print(" %-11s LayerNorm -> %s" % ("", np.round(ln[:4], 4)))
print(" LayerNorm gives the same answer every time. BatchNorm does not,")
print(" so the model's output for one sentence depends on which other")
print(" sentences happened to be batched with it.")
print()
print("PROBLEM 3 -- LENGTH. BatchNorm keeps running statistics per feature,")
print("but a sequence model applies the same layer at every timestep, and")
print("the distribution at step 0 is not the distribution at step 50:")
h = np.zeros(8)
W = rng.normal(0, 0.5, (8, 8))
print("%10s %14s %14s" % ("timestep", "state mean", "state sd"))
for t in range(1, 51):
h = np.tanh(h @ W + rng.normal(0, 0.5, 8))
if t in (1, 2, 5, 20, 50):
print("%10d %14.4f %14.4f" % (t, h.mean(), h.std()))
print(" keeping one running mean per feature across all timesteps averages")
print(" over distributions that are genuinely different. keeping one PER")
print(" TIMESTEP means you cannot handle a sequence longer than the")
print(" longest one you trained on.")
print()
print("THE COMPARISON:")
rows = [("normalises over", "batch and time", "features of one position"),
("depends on batch size", "yes", "no"),
("train / eval differ", "yes -- running stats", "no"),
("padding pollutes it", "yes", "no"),
("works at batch size 1", "no", "yes"),
("variable length", "awkward", "fine")]
print("%24s %24s %28s" % ("", "BatchNorm", "LayerNorm"))
for r in rows:
print("%24s %24s %28s" % r)
print()
print("AND THE VARIANTS WORTH KNOWING:")
print(" RMSNorm -- LayerNorm without the mean subtraction. one fewer")
print(" reduction, and in practice the centring turned out")
print(" not to matter:")
def rmsnorm(a, eps=1e-6):
return a / np.sqrt((a ** 2).mean(-1, keepdims=True) + eps)
row = x[0, 0]
print(" LayerNorm(x) = %s" % np.round(layernorm(x)[0, 0][:4], 4))
print(" RMSNorm(x) = %s" % np.round(rmsnorm(x)[0, 0][:4], 4))
print()
print(" pre-norm vs post-norm -- where the norm sits relative to the")
print(" residual. pre-norm (normalise the sublayer's INPUT) leaves the")
print(" skip path clean, so the gradient reaches layer 1 unattenuated.")
print(" that is what made very deep transformers trainable without a")
print(" long learning-rate warmup, and it is why almost everything now")
print(" uses it.")
print()
print(" and always mask. whatever you normalise with, the padded positions")
print(" must be excluded from the loss -- a model scored on predicting")
print(" padding will happily learn to predict padding.")
Output
Questions people ask
Why not batch normalisation for sequences? It mixes statistics across positions and padding, needs a decent batch size, and differs between training and inference.
Do I still need to scale the inputs if I use LayerNorm? Yes. The first layer receives the raw input, and normalisation only affects activations after it.
Should I difference my series? If it trends, and especially if the model is tree-based, yes. Check stationarity rather than assuming.
What is rolling normalisation? Scaling each window by its own statistics, so every example is locally normalised. Standard in finance, and it discards absolute level.
Can I scale on the whole series if I am careful? No — that is the leak. Training-period statistics only.
Pre-norm or post-norm? Pre-norm for anything deep. It is what all current large models use.
Recap in one screen
Two meanings: scaling the input data, and normalising activations inside the network.
Fit input scalers on the training period only — using the whole series leaks the future.
Non-stationary series need differencing, log transforms, rolling normalisation or periodic refitting.
Batch normalisation is wrong for sequences; LayerNorm and RMSNorm are batch-independent and correct.
Place normalisation before each sublayer (pre-norm) so the residual path stays clean.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Normalising over the wrong axis”?
Every normalisation layer does the same arithmetic — subtract a mean, divide by a standard deviation, then apply a learned scale and shift. What differs is which values are pooled to compute that mean and deviation.
What does this module say about “Why batch norm fails on sequences”?
Variable length. Sequences in a batch have different lengths, so timestep 50 might have 32 real values in one batch and 3 in another, with the rest padding. Statistics computed over that are unstable, and computed over padding they are simply wrong. Per-timestep statistics.
What does this module say about “Two different meanings of "normalisation"”?
The word covers two unrelated operations in sequence work, and conflating them causes confusion.
Cheat sheet
Normalization Techniques for Sequential Data
Networks train best when inputs live on a small, consistent scale. Compare raw, Min-Max, and Z-Score views of the same series — then throw in an outlier.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.