Watch how initialization impacts training flow! Poor choices cause deep networks to die/vanish (Blue) or explode (Red). Select a method below (like He Normal) or use Batch Normalization to lock gradients into a stable variance (Green) even with bad initial weights.
Overview
Why not zero, and why not all-equal
Initialising every weight to zero seems harmless and completely breaks the network. If all weights in a layer are identical, every neuron in that layer computes the same output, receives the same gradient, and applies the same update — so they stay identical forever. A layer of 512 such neurons has the expressive power of one.
This is the symmetry breaking problem, and it is why initialisation must be random. Biases can safely start at zero, because the weights already break the symmetry.
Stability Metrics
Global Status
STABLE (HE NORMAL)
Weights maintain an optimal variance (~1.0).
Average Weight Magnitude
W1 (Input → H1)1.000
W3 (H2 → H3)1.000
Color Mapping
Vanish (~0)Stable (~1)Explode (>2)
Drag to Pan | Scroll to Zoom
Weight Initialization Methods: A Practical Guide
The starting values of the weights decide whether a deep network trains at all. Too small and the signal dies; too large and it explodes; all the same and the network never becomes more than one neuron wide.
The variance is what actually matters
Randomness alone is not enough; the scale of the random values decides whether signal survives depth. Each layer multiplies its input by a weight matrix, so the variance of the activations is multiplied layer by layer.
If that per-layer factor is below 1, activations shrink geometrically and the deepest layers see almost nothing. If it is above 1, they grow geometrically and saturate or overflow. The target is a factor of about 1, so activations keep roughly constant scale from the first layer to the last — and the same for gradients on the way back.
That is the whole design goal, and the named schemes are just different solutions to it:
Xavier / Glorot — variance 2 / (nin + nout). Derived for activations symmetric about zero, so it suits tanh and sigmoid.
He / Kaiming — variance 2 / nin. ReLU zeroes negative inputs and therefore roughly halves the variance passing through, so He compensates with the extra factor of 2. This is the right default for any ReLU network.
LeCun — variance 1 / nin, used with SELU and self-normalising networks.
Note that all of them scale with layer width. A 1024-unit layer needs smaller initial weights than a 64-unit one, because it sums many more terms.
Why the starting point matters
Training begins from random weights, and the choice of "random" is not arbitrary. Two failure modes bracket the sensible range.
All zeros. Every unit in a layer computes the same thing, receives the same gradient, and updates identically. The layer never differentiates and behaves as a single unit forever. Symmetry must be broken, which is why the weights are random at all.
Too large or too small. Activations either grow or shrink as they pass through layers. After twenty layers, activations that shrink by 0.8 per layer have shrunk to 1% of their original scale, and the gradients follow. The network is dead before the first update.
The goal is a scale that keeps the variance of activations roughly constant through depth — and that scale depends on the layer's width and its activation function.
The two schemes worth knowing
Xavier (Glorot) initialisation, for tanh and sigmoid:
Var(W) = 2 / (fan_in + fan_out)
He (Kaiming) initialisation, for ReLU:
Var(W) = 2 / fan_in
The difference is the factor of 2, and the reason is direct: ReLU zeroes half its inputs, so it halves the variance passing through. Doubling the initial variance compensates exactly.
Activation
Use
ReLU, Leaky ReLU
He (kaiming_normal_)
Tanh, sigmoid
Xavier (xavier_uniform_)
GELU, SiLU
He, or the framework default
Linear output layer
Xavier, or small values
Using Xavier with ReLU is a real if subtle mistake: activations shrink through depth by a factor of √2 per layer, and a 30-layer network arrives at the loss with almost no signal.
import torch.nn as nn
for m in model.modules():
if isinstance(m, (nn.Conv2d, nn.Linear)):
nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
if m.bias is not None:
nn.init.zeros_(m.bias)
Biases are initialised to zero — there is no symmetry problem for them, since the weights already differ.
Special cases in modern architectures
Batch normalisation's γ and β start at 1 and 0, so the layer initially passes its normalised input through unchanged.
The last block's normalisation in a residual network is sometimes initialised to zero (γ = 0), which makes each residual block start as an exact identity. The network begins as a shallow one and deepens itself as training proceeds — a trick that stabilises very deep and very large models.
Transformers often scale the initialisation of the output projections by 1/√(2 × layers), because the residual stream accumulates contributions from every block and would otherwise grow with depth.
Embeddings are typically initialised from a normal distribution with a small standard deviation (0.02 is the common choice in language models).
Orthogonal initialisation for recurrent weight matrices keeps the largest singular value at 1, which is exactly the condition that prevents the hidden state exploding or vanishing over time steps.
Exploration guide
Train from a reasonable start. Press Reset Weights and then Train Network. Activations stay in a usable range across the layers and the loss falls steadily.
Watch the layer-to-layer scale. During training, compare the activation magnitudes at the first and last layers. When initialisation is right they are comparable; a systematic shrink or growth across depth is the failure this whole topic exists to prevent.
Add normalisation. Enable the batch-norm toggle and press Reset Weights, then train again. Convergence becomes markedly less sensitive to where the weights started — normalisation rescales each layer regardless.
Reset repeatedly. Press Reset Weights several times and train each time. Runs differ, sometimes noticeably. The initial draw is a genuine source of run-to-run variance, which is why single-run comparisons are unreliable.
Traps worth knowing
Zero initialisation. Symmetry never breaks and the layer collapses to a single effective neuron. Zero is fine for biases and wrong for weights.
A fixed standard deviation such as 0.01 everywhere. Ignores layer width, so wide layers explode and deep stacks vanish. This was standard before 2010 and is a large part of why deep networks did not train.
Xavier with ReLU. Understates the variance by a factor of 2, since Xavier does not account for ReLU discarding half the distribution. Use He for ReLU.
Assuming batch norm makes it irrelevant. Normalisation reduces the sensitivity but does not remove it, particularly in very deep networks and in the layers before the first normalisation.
Reinitialising a pretrained layer. When fine-tuning, only the new head should be initialised; overwriting the pretrained weights discards everything transfer learning was for.
What to remember
Initialisation must be random to break symmetry, and scaled to layer width so activation and gradient variance stay roughly constant with depth. He initialisation is the default for ReLU networks because ReLU halves the variance and He’s factor of 2 restores it; Xavier suits tanh and sigmoid. Get this wrong and a deep network either learns nothing or diverges — before the optimiser has had any say in the matter.
How much does it still matter?
Less than it did, and more than nothing.
Batch normalisation and residual connections both reduce sensitivity to initialisation, which is part of why they were such a breakthrough — they made deep networks trainable even from an imperfect starting point. Pretrained weights remove the question entirely for transfer learning.
But it still matters in three situations: training from scratch without normalisation layers, very deep or very large models where small per-layer errors compound, and recurrent networks where the same matrix is applied many times.
And the framework defaults are not always right. PyTorch's default for nn.Linear is a uniform distribution based on fan-in that predates He initialisation, and it is not optimal for ReLU networks. Setting it explicitly costs three lines.
Diagnosing a bad initialisation
Before training, do a single forward pass and print the standard deviation of the activations at each layer:
acts = {}
for name, m in model.named_modules():
if isinstance(m, nn.ReLU):
m.register_forward_hook(
lambda mod, i, o, n=name: acts.__setitem__(n, o.std().item()))
model(sample_batch)
for k, v in acts.items():
print(f"{k:<40} std {v:.4f}")
Healthy: roughly similar values throughout. Shrinking towards zero with depth means the initialisation is too small (or Xavier was used with ReLU). Growing means it is too large, and the first gradient step will be violent.
The same check on the initial loss is worth doing: with C balanced classes it should be about ln(C). A much larger value means the initial predictions are confidently wrong, which usually means the output layer's weights are too large.
Zero, too big, too small, and right
Initialisation decides whether a deep network trains at all. Four schemes on the same architecture, measured by what happens to the activations after ten layers.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
WIDTH, DEPTH, N = 128, 10, 512
def propagate(scale_fn, act, name):
a = rng.normal(size=(N, WIDTH))
stds = []
for _ in range(DEPTH):
W = scale_fn(WIDTH)
a = act(a @ W)
stds.append(a.std())
return name, stds
relu = lambda z: np.maximum(0, z)
tanh = np.tanh
print("activation standard deviation after each of %d layers." % DEPTH)
print("healthy means it stays roughly constant. it should not trend.")
print()
print("%-32s %9s %9s %9s %9s"
% ("scheme", "layer 1", "layer 4", "layer 7", "layer 10"))
for fn, act, name in (
(lambda w: rng.normal(0, 0.01, (w, w)), relu, "gaussian sd 0.01 (too small)"),
(lambda w: rng.normal(0, 1.00, (w, w)), relu, "gaussian sd 1.00 (too big)"),
(lambda w: rng.normal(0, np.sqrt(1 / w), (w, w)), tanh, "Xavier/Glorot, tanh"),
(lambda w: rng.normal(0, np.sqrt(2 / w), (w, w)), relu, "He, relu"),
):
name, stds = propagate(fn, act, name)
print("%-32s %9.3e %9.3e %9.3e %9.3e"
% (name, stds[0], stds[3], stds[6], stds[9]))
print()
print("sd 0.01 collapses toward zero -- by layer 10 every activation is")
print("effectively 0, so no information reaches the output and no gradient")
print("comes back. sd 1.00 does the opposite and saturates.")
print()
print("the two that work stay on the same order of magnitude. He is flat.")
print("Xavier drifts down gently, because tanh compresses whatever it is")
print("given -- but compare a drift from 0.63 to 0.22 against a collapse to")
print("1e-11. that is the difference between a network that trains slowly")
print("and one that does not train at all.")
print()
print("the rules themselves:")
print(" Xavier: variance 1/n_in, for tanh and sigmoid.")
print(" He: variance 2/n_in, for relu -- the 2 compensates for relu")
print(" discarding half its inputs.")
print()
print("here is where the 2 comes from. relu zeroes the negative half, and the")
print("quantity that propagates through a layer is the mean SQUARE:")
z = rng.normal(0, 1.0, 400_000)
print(" E[z^2] for the input %.4f" % (z ** 2).mean())
print(" E[relu(z)^2] after the relu %.4f <- exactly half"
% (relu(z) ** 2).mean())
print(" half the values became 0 and the other half were untouched, so the")
print(" mean of the squares halves. every layer would halve it again.")
z2 = rng.normal(0, np.sqrt(2.0), 400_000)
print(" with weights scaled by sqrt(2): E[relu(z)^2] = %.4f <- restored"
% (relu(z2) ** 2).mean())
print(" that is the whole derivation of He initialisation.")
print()
print("and the failure that is not about scale at all -- all zeros:")
W1 = np.zeros((4, 3)); W2 = np.zeros((3, 1))
x = rng.normal(size=(8, 4)); y = rng.normal(size=(8, 1))
for step in range(50):
h = np.tanh(x @ W1)
out = h @ W2
d_out = 2 * (out - y) / len(x)
dW2 = h.T @ d_out
dW1 = x.T @ ((d_out @ W2.T) * (1 - h ** 2))
W1 -= 0.1 * dW1; W2 -= 0.1 * dW2
print(" after 50 steps, the hidden layer's weights:")
print(np.round(W1, 6))
print(" every column is identical. all three hidden units computed the same")
print(" thing, received the same gradient, and updated identically.")
print(" a layer of n units with the same values is a layer of 1 unit.")
print()
print("that is why initialisation must be RANDOM, not merely small. the")
print("randomness is what breaks the symmetry between units, and no amount")
print("of training will break it later.")
print()
print("biases are the exception -- zero is the standard choice for them,")
print("because the weights have already broken the symmetry.")
Output
Questions people ask
Can I initialise all weights to the same non-zero value? No — the symmetry problem is identical to using zeros. Units must start different.
Should biases be zero? Yes, almost always. One exception: setting the output bias of a binary classifier to the log-odds of the positive class speeds up early training on imbalanced data.
Normal or uniform? Barely matters — the variance is what counts. Both are provided for both schemes.
What about fine-tuning? The pretrained weights are the initialisation. Only a newly added head needs initialising, and it should be small so it does not disturb the pretrained features on the first step.
Does initialisation affect the final result? With normalisation and residuals, usually only mildly — different seeds land at similar quality. Without them, it can decide whether the model trains at all.
Why does my loss start enormous? Output-layer weights too large, or a label/logit mismatch. Compare against ln(number of classes).
Recap in one screen
Zeros break the network by symmetry; wrong scales make activations grow or vanish through depth.
He initialisation (variance 2/fan_in) for ReLU; Xavier (2/(fan_in+fan_out)) for tanh and sigmoid.
The factor of 2 in He exists because ReLU discards half the signal.
Biases start at zero; residual blocks are sometimes started as exact identities.
Batch normalisation and residuals reduce sensitivity, but do not remove the need to get it roughly right.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Initialisation must be random to break symmetry, and scaled to layer width so activation and gradient variance stay roughly constant with depth. He initialisation is the default for ReLU networks because ReLU halves the variance and He’s factor of 2 restores it; Xavier suits tanh and sigmoid. Get this wrong and a deep network either learns nothing or diverges — before the optimiser has had any say in the matter.
What does this module say about “Why not zero, and why not all-equal”?
Initialising every weight to zero seems harmless and completely breaks the network. If all weights in a layer are identical, every neuron in that layer computes the same output, receives the same gradient, and applies the same update — so they stay identical forever. A layer of 512 such neurons has the expressive power of one.
What does this module say about “The variance is what actually matters”?
Randomness alone is not enough; the scale of the random values decides whether signal survives depth. Each layer multiplies its input by a weight matrix, so the variance of the activations is multiplied layer by layer.
Cheat sheet
Weight Initialization Methods
Watch how initialization impacts training flow! Poor choices cause deep networks to die/vanish (Blue) or explode (Red). Select a method below (like He Normal) or use Batch Normalization to lock gradients into a stable variance (Green) even with bad initial weights.
DEEP LEARNING · vizlearn.in/deep_learning/weight_initialization.html
Understanding the difficulty of training deep feedforward neural networks (Xavier initialisation)Glorot & Bengio, AISTATS 2010
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.