Networks train best when inputs live on a small, consistent scale. Compare raw, Min-Max, and Z-Score views of the same series — then throw in an outlier.
Technique
Replaces one sensor reading with a huge spike (420) and re-normalizes — watch what each technique does to the rest of the curve.
Series Statistics
Min
-
Max
-
Mean μ
-
Std σ
-
Formula
x' = x
Sensor Readings Over Time
RANGE: -
Raw values — note the y-axis scale. Large, drifting magnitudes make gradient updates uneven across features and slow training down.
Overview
Normalising over the wrong axis
Each normalisation layer does the same arithmetic — subtract a mean, divide by a standard deviation. Then apply a learned scale and shift. What differs is which values are pooled to compute that mean and deviation.
Batch normalisation computes stats per feature, across the batch. For feature 3, average over all samples in the batch. Layer normalisation computes stats per sample, across the features. For sample 3, average over all of its features.
On images batch norm is excellent. On sequences it breaks, for three separate reasons. Variable-length inputs make per-timestep stats unstable, the running averages have no values for spots longer than the training data. And a single sample at inference gives meaningless batch stats. Layer normalisation sidesteps all three by computing stats per sample rather than per batch. This is why it is the default in transformers and recurrent networks. RMSNorm cuts down layer norm further by dropping the mean subtraction, and group norm covers the vision cases where batch sizes are too small for batch norm. The choice of which axis to normalise over is not a minor implementation detail. It decides whether the layer works at all in the target setting, and switching from batch norm to layer norm is often the single change that makes a sequence model train stably. This page computes batch norm and layer norm on the same tensor, prints the per-feature and per-sample stats side by side, and covers the failure cases for each.
Networks train best when inputs live on a small, consistent scale.
Normalization Techniques for Sequential Data: A Practical Guide
Batch normalisation works beautifully on images and badly on sequences. Layer normalisation is what replaced it, and the reason is entirely about which axis the statistics are computed over.
Normalising over the wrong axis
Each normalisation layer does the same arithmetic — subtract a mean, divide by a standard deviation. Then apply a learned scale and shift. What differs is which values are pooled to compute that mean and deviation.
Batch normalisation computes stats per feature, across the batch. For feature 3, average over all samples in the batch. Layer normalisation computes stats per sample, across the features. For sample 3, average over all of its features.
On images batch norm is excellent. On sequences it breaks, for three separate reasons. Variable-length inputs make per-timestep stats unstable, the running averages have no values for spots longer than the training data. And a single sample at inference gives meaningless batch stats. Layer normalisation sidesteps all three by computing stats per sample rather than per batch. This is why it is the default in transformers and recurrent networks. RMSNorm cuts down layer norm further by dropping the mean subtraction, and group norm covers the vision cases where batch sizes are too small for batch norm. The choice of which axis to normalise over is not a minor implementation detail. It decides whether the layer works at all in the target setting, and switching from batch norm to layer norm is often the single change that makes a sequence model train stably. This page computes batch norm and layer norm on the same tensor, prints the per-feature and per-sample stats side by side, and covers the failure cases for each.
Why batch norm fails on sequences
Variable length. Sequences in a batch have different lengths, so timestep 50 might have 32 real values in one batch and 3 in one more, with the rest padding. Stats computed over that are unstable, and computed over padding they are simply wrong.Per-timestep stats. A recurrent network applies the same cell at each step, but batch norm would need separate running stats for each timestep. And at inference on a sequence longer than a thing seen in training, there are no stats for those spots at all.Batch dependence. A sample’s normalised form depends on the other samples it happens to be batched with. For generation, where inference often runs one sequence at a time, the batch stats are meaningless.
Layer norm sidesteps all three. It uses only the sample’s own features, so it is free of batch size, the same at training and inference, and unaffected by how many other sequences are present or how long they are.
RMSNorm, used in LLaMA and other recent models, cuts down layer norm further by dropping the mean subtraction and normalising by the root mean square alone. One fewer statistic to compute per token, with no measurable loss in grade. Group norm (used in vision when batch sizes are small) and instance norm (used in style transfer) are two other axis choices that solve different problems. The choice of normalisation axis is not a detail. It decides whether the layer works at all in the target setting, and switching from batch norm to layer norm is often the single change that makes a sequence model train stably. This page computes batch norm and layer norm on the same tensor, prints the per-feature and per-sample stats side by side, and covers the failure cases for each.
Two different meanings of "normalisation"
The word covers two unrelated steps in sequence work, and conflating them causes confusion.
Data normalisation scales the input values — putting a temperature series and a price series on comparable footing before the model sees them.
Layer normalisation is an architectural part that standardises triggers inside the network at each step.
Both matter for sequences, and for different reasons. This covers both.
Scaling sequence inputs
The requirement is the same as for tabular data. A feature ranging over thousands will dominate the first layer's weighted sum, and gradients for the other features turn tiny.
The rule that is set to sequences is fit the scaler on the training period only:
from sklearn.preprocessing import StandardScaler
split = int(len(series) * 0.7)
scaler = StandardScaler().fit(series[:split].reshape(-1, 1)) # past only
scaled = scaler.transform(series.reshape(-1, 1))
Computing the mean and standard deviation over the whole series uses future values to scale past ones, which is a leak. It inflates results and cannot be reproduced in production, where the future is unavailable by definition.
Method
Suits
Standardisation
The default for most sequences
Min-max
Bounded signals such as audio or normalised sensors
Reliable (median / IQR)
Series with spikes and outliers
Log transform
Multiplicative series — prices, counts, traffic
Differencing
Series with a trend, especially for tree models
Differencing deserves a note: tree-based models cannot extrapolate beyond their training range. So on a trending series they will never predict a new high. Modelling the change rather than the level fixes it. And this is one of the most valuable transformations in applied forecasting.
Non-stationarity, the sequence-set problem
A series whose stats change over time is non-fixed. And a single scaler fitted once is wrong for the later part of it.
A stock price in 2015 and in 2025 differ in level and volatility; a website's traffic in its first month and its fifth year are different distributions. Scaling both with one mean and standard deviation puts the recent data far from zero.
Three responses:
Gap the series so the model works on changes, which are often far more fixed than levels.
Use rolling normalisation — scale each window by its own stats. So each case is locally normalised. This is standard in financial modelling, and it discards absolute level. This is sometimes just what you want and sometimes the sign.
Refit at times, as part of a retraining schedule, and monitor for drift.
Note that a log transform handles a set and common case. When a series grows multiplicatively, differencing the logarithm gives percentage changes. This are often fixed where absolute changes are not.
Interactive Exploration Guide
Look at the unnormalised distribution. With the outlier toggle off, note the spread of the raw values. Features on different scales pull the layer’s triggers around.
Add an outlier. Enable the outlier toggle. Watch how a single extreme value shifts the computed mean and inflates the standard deviation, dragging each other value with it.
Compare the axes. Note which values get pooled. Normalising across the batch means one sample’s stats depend on its neighbours; normalising across features means each sample is self-held.
Toggle back and forth. The response to a single outlier is the real input for reliable preprocessing before the layer ever sees the data.
The variants worth knowing
Batch norm — per feature, across the batch. Convolutional networks with fair batch sizes.
Layer norm — per sample, across features. Transformers and recurrent networks, in effect universally.
Group norm — per sample, across groups of channels. Vision with small batches, where batch norm’s estimates go noisy.
RMSNorm — layer norm without the mean subtraction, dividing by the root mean square only. Slightly cheaper, works as well, and is what most recent large language models use.
The placement matters too. First transformers put layer norm after the residual addition (post-norm). Modern ones put it before the sublayer (pre-norm), which makes deep stacks far easier to train and often removes the need for a learning-rate warmup.
Where this goes wrong
Batch norm with a batch size of 1 or 2. The variance estimate is meaningless. Use layer or group norm.
Forgetting eval mode. Batch norm uses batch stats in training and running averages at inference. Leaving the model in training mode makes predictions depend on any else is in the batch.
Normalising over padded spots. Pollutes the stats with zeros that carry no data.
Adding a bias before a normalisation layer. The mean subtraction cancels it just. So those parameters do nothing. Disable the bias on a layer feeding into a norm.
The short version
All normalisation layers subtract a mean and divide by a deviation; they differ only in which values are pooled. Batch norm pools across the batch, which makes it unusable for sequences of varying length and inference on single cases. Layer norm pools across each sample’s own features, so it is free of batch size and the same at training and inference —. This is why each transformer uses it.
Layer normalisation inside the network
For recurrent and transformer models, the internal normalisation choice is not the same as for convolutional networks. And the reason is structural.
Batch normalisation does not suit sequences. It computes stats across the batch for each feature. This means mixing spot 3 of one sentence with spot 3 of one more. And with padding, mixing real tokens with filler. It also needs a fair batch size, and it behaves differently at inference. This matters when serving one request at a time.
Layer normalisation standardises each case across its own features:
x̂ = (x − μcase) / √(σ²case + ε) out = γx̂ + β
It is fully free of the batch, works at batch size 1, handles variable lengths, and behaves identically in training and inference. That is why each transformer uses it.
Normalisation
Statistics over
Sequence-safe?
BatchNorm
The batch, per feature
No
LayerNorm
Features of one example
Yes
RMSNorm
Scale only, one example
Yes
GroupNorm
Channel groups, one example
For vision
RMSNorm is the simplification now common in large language models. Divide by the root mean square and skip the mean subtraction. One fewer statistic, measurably faster at scale, no grade cost in practice.
Placement matters
In transformers, where the normalisation sits next to the residual connection is a real architectural decision.
Post-norm — LayerNorm(x + sublayer(x)) — was the first arrangement. It works, and beyond roughly 12 layers it needs careful warm-up to train at all.
Pre-norm — x + sublayer(LayerNorm(x)) — puts the normalisation inside the residual branch. This leaves the identity path fully clean. Gradients flow back through it unchanged, and very deep stacks turn trainable.
Each large model uses pre-norm. It is one of the small details that made scaling to 80 or more layers real.
For recurrent cells, layer normalisation applied to the gate pre-triggers stabilises training and helps with the exploding cell states that if not saturate the output tanh.
Why BatchNorm loses to LayerNorm on sequences
Sequences are ragged, arrive one at a time, and vary in length. Each of those breaks BatchNorm in a set way. And this measures all three on padded batches.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
B, T, D = 4, 6, 5
lengths = [6, 3, 5, 2]
x = rng.normal(0, 1.0, (B, T, D)) * 2 + 1
mask = np.zeros((B, T))
for i, L in enumerate(lengths):
mask[i, :L] = 1
x[i, L:] = 0.0 # padding
print("a batch of %d sequences, padded to length %d." % (B, T))
print(" real lengths: %s -> %d of %d positions are padding (%.0f%%)"
% (lengths, int((1 - mask).sum()), B * T,
100 * (1 - mask).sum() / (B * T)))
print()
def batchnorm(a, eps=1e-5):
# normalise over batch AND time, per feature -- the usual sequence variant
mu = a.mean(axis=(0, 1), keepdims=True)
sd = a.std(axis=(0, 1), keepdims=True)
return (a - mu) / (sd + eps)
def layernorm(a, eps=1e-5):
mu = a.mean(axis=-1, keepdims=True)
sd = a.std(axis=-1, keepdims=True)
return (a - mu) / (sd + eps)
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
print("PROBLEM 1 -- PADDING POLLUTES THE STATISTICS. BatchNorm averages over")
print("the batch, and the padding is in the batch:")
real = x[mask.astype(bool)]
print(" mean over REAL positions only : %s" % np.round(real.mean(0), 4))
print(" mean over everything incl. pad: %s" % np.round(x.mean(axis=(0, 1)), 4))
print(" sd over REAL positions only : %s" % np.round(real.std(0), 4))
print(" sd over everything incl. pad: %s" % np.round(x.std(axis=(0, 1)), 4))
print(" the padding is %d of %d rows of zeros, and it drags both toward 0."
% (int((1 - mask).sum()), B * T))
print(" LayerNorm never sees it -- it normalises within a position, so a")
print(" padded position is normalised on its own and then masked away.")
print()
print("PROBLEM 2 -- THE STATISTICS DEPEND ON THE BATCH. take sequence 0 and")
print("run it in three different batches:")
target = x[0:1]
for k in (1, 2, 4):
batch = x[:k]
bn = batchnorm(batch)[0, 0]
ln = layernorm(batch)[0, 0]
print(" batch of %d: BatchNorm -> %s" % (k, np.round(bn[:4], 4)))
print(" %-11s LayerNorm -> %s" % ("", np.round(ln[:4], 4)))
print(" LayerNorm gives the same answer every time. BatchNorm does not,")
print(" so the model's output for one sentence depends on which other")
print(" sentences happened to be batched with it.")
print()
Output
The below block covers the left steps.
example_03.pyNumPy
import numpy as np
print("PROBLEM 3 -- LENGTH. BatchNorm keeps running statistics per feature,")
print("but a sequence model applies the same layer at every timestep, and")
print("the distribution at step 0 is not the distribution at step 50:")
h = np.zeros(8)
W = rng.normal(0, 0.5, (8, 8))
print("%10s %14s %14s" % ("timestep", "state mean", "state sd"))
for t in range(1, 51):
h = np.tanh(h @ W + rng.normal(0, 0.5, 8))
if t in (1, 2, 5, 20, 50):
print("%10d %14.4f %14.4f" % (t, h.mean(), h.std()))
print(" keeping one running mean per feature across all timesteps averages")
print(" over distributions that are genuinely different. keeping one PER")
print(" TIMESTEP means you cannot handle a sequence longer than the")
print(" longest one you trained on.")
print()
print("THE COMPARISON:")
rows = [("normalises over", "batch and time", "features of one position"),
("depends on batch size", "yes", "no"),
("train / eval differ", "yes -- running stats", "no"),
("padding pollutes it", "yes", "no"),
("works at batch size 1", "no", "yes"),
("variable length", "awkward", "fine")]
Output
The next block goes on with the left demonstrations.
example_04.pyNumPy
print("%24s %24s %28s" % ("", "BatchNorm", "LayerNorm"))
for r in rows:
print("%24s %24s %28s" % r)
print()
print("AND THE VARIANTS WORTH KNOWING:")
print(" RMSNorm -- LayerNorm without the mean subtraction. one fewer")
print(" reduction, and in practice the centring turned out")
print(" not to matter:")
def rmsnorm(a, eps=1e-6):
return a / np.sqrt((a ** 2).mean(-1, keepdims=True) + eps)
row = x[0, 0]
print(" LayerNorm(x) = %s" % np.round(layernorm(x)[0, 0][:4], 4))
print(" RMSNorm(x) = %s" % np.round(rmsnorm(x)[0, 0][:4], 4))
print()
print(" pre-norm vs post-norm -- where the norm sits relative to the")
print(" residual. pre-norm (normalise the sublayer's INPUT) leaves the")
print(" skip path clean, so the gradient reaches layer 1 unattenuated.")
print(" that is what made very deep transformers trainable without a")
print(" long learning-rate warmup, and it is why almost everything now")
print(" uses it.")
print()
print(" and always mask. whatever you normalise with, the padded positions")
print(" must be excluded from the loss -- a model scored on predicting")
print(" padding will happily learn to predict padding.")
Output
Questions people ask
Why not batch normalisation for sequences? It mixes stats across spots and padding, needs a decent batch size, and differs between training and inference.
Do I still need to scale the inputs if I use LayerNorm? Yes. The first layer receives the raw input, and normalisation only affects triggers after it.
Should I gap my series? If it trends, and especially if the model is tree-based, yes. Check stationarity rather than assuming.
What is rolling normalisation? Scaling each window by its own stats. So each case is locally normalised. Standard in finance, and it discards absolute level.
Can I scale on the whole series if I am careful? No — that is the leak. Training-period stats only.
Pre-norm or post-norm? Pre-norm for a thing deep. It is what all current large models use.
Recap in one screen
Two meanings: scaling the input data, and normalising triggers inside the network.
Fit input scalers on the training period only. Using the whole series leaks the future.
Non-fixed series need differencing, log transforms, rolling normalisation or periodic refitting.
Batch normalisation is wrong for sequences; LayerNorm and RMSNorm are batch-free and correct.
Place normalisation before each sublayer (pre-norm) so the residual path stays clean.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Normalising over the wrong axis”?
What does this module say about “Why batch norm fails on sequences”?
What does this module say about “Two different meanings of "normalisation"”?
Cheat sheet
Normalization Techniques for Sequential Data
See why batch normalisation breaks on variable-length sequences and how layer normalisation fixes it by computing stats per sample.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.