vizlearn

Weight Initialization Methods

Deep LearningBy Updated

Watch how initialization impacts training flow! Poor choices cause deep networks to die/vanish (Blue) or explode (Red). Select a method below (like He Normal) or use Batch Normalization to lock gradients into a stable variance (Green) even with bad initial weights.

Stability Metrics

Global Status
STABLE (HE NORMAL)
Weights maintain an optimal variance (~1.0).

Average Weight Magnitude

W1 (Input → H1) 1.000
W3 (H2 → H3) 1.000

Color Mapping

Vanish (~0) Stable (~1) Explode (>2)
Drag to Pan | Scroll to Zoom

Overview

Watch how initialization impacts training flow!

In this guide
  1. Why not zero, and why not all-equal
  2. The variance is what really matters
  3. Why the starting point matters
  4. The two schemes worth knowing
  5. Special cases in modern designs
  6. Exploration guide
  7. Traps worth knowing
  8. What to remember
  9. How much does it still matter?
  10. Diagnosing a bad setup
  11. Zero, too big, too small, and right
  12. Questions people ask
  13. Recap in one screen

Weight Initialization Methods: A Practical Guide

The starting values of the weights decide whether a deep network trains at all. Too small and the signal dies; too large and it explodes; all the same and the network never becomes more than one neuron wide.

Why not zero, and why not all-equal

Initialising each weight to zero seems harmless and fully breaks the network. If all weights in a layer are identical, each neuron in that layer computes the same output, gets the same gradient. And applies the same update — so they stay identical forever. A layer of 512 such neurons has the expressive power of one.

This is the symmetry breaking problem. And it is why setup must be random. Biases can safely start at zero, since the weights now break the symmetry. The scale of the random vals matters as much as their randomness. Each layer multiplies its input by a weight matrix. So output variance is multiplied layer by layer. If the per-layer factor is below 1, outputs shrink geometrically and deep layers see almost nothing. If above 1, they grow and saturate or overflow. Xavier (Glorot) setup sets the variance to 2 / (fan_in + fan_out). It keeps variance constant through layers with tanh or sigmoid outputs. He (Kaiming) setup sets it to 2 / fan_in. It accounts for the fact that ReLU zeros out half its inputs and would otherwise halve the variance at each layer. With Xavier with ReLU or He with sigmoid are common mistakes that quietly destabilise training. The network trains but converges to a worse solution. And the symptom is indistinguishable from a bad learning rate. This page shows output spreads at each layer under zero, Xavier. And He setup. And lets you switch output functions to see which pairing works.

The variance is what really matters

Randomness alone is not enough. The scale of the random vals decides whether signal survives depth. Each layer multiplies its input by a weight matrix. So the variance of the outputs is multiplied layer by layer.

If that per-layer factor is below 1, outputs shrink geometrically and the deepest layers see almost nothing. If it is above 1, they grow geometrically and saturate or overflow. The target is a factor of about 1. So outputs keep roughly constant scale from the first layer to the last. And the same for gradients on the way back.

That is the whole design goal. And the named schemes are just other solutions to it:

  • Xavier / Glorot — variance 2 / (nin + nout). Derived for outputs symmetric about zero. So it suits tanh and sigmoid.
  • He / Kaiming — variance 2 / nin. ReLU zeroes negative inputs and therefore roughly halves the variance passing through. So He compensates with the extra factor of 2. This is the right default for any ReLU network.
  • LeCun — variance 1 / nin, used with SELU and self-scaling networks.

Note that all of them scale with layer width. A 1024-unit layer needs less initial weights than a 64-unit one, since it sums many more terms.

Why the starting point matters

Training begins from random weights. And the choice of "random" is not arbitrary. Two failure modes bracket the sensible range.

All zeros. Each unit in a layer computes the same thing, gets the same gradient. And updates identically. The layer never differentiates and behaves as a single unit forever. Symmetry must be broken. It is why the weights are random at all.

Too large or too small. Outputs either grow or shrink as they pass through layers. After twenty layers, outputs that shrink by 0.8 per layer have shrunk to 1% of their first scale. And the gradients follow. The network is dead before the first update.

The goal is a scale that keeps the variance of outputs roughly constant through depth. And that scale depends on the layer's width and its output function.

The two schemes worth knowing

Xavier (Glorot) setup, for tanh and sigmoid:

Var(W) = 2 / (fan_in + fan_out)

He (Kaiming) setup, for ReLU:

Var(W) = 2 / fan_in

The gap is the factor of 2. And the reason is direct: ReLU zeroes half its inputs. So it halves the variance passing through. Doubling the initial variance compensates just.

ActivationUse
ReLU, Leaky ReLUHe (kaiming_normal_)
Tanh, sigmoidXavier (xavier_uniform_)
GELU, SiLUHe, or the framework default
Linear output layerXavier, or small values

With Xavier with ReLU is a real if subtle mistake: outputs shrink through depth by a factor of √2 per layer. And a 30-layer network arrives at the loss with almost no signal.

import torch.nn as nn

for m in model.modules():
    if isinstance(m, (nn.Conv2d, nn.Linear)):
        nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
        if m.bias is not None:
            nn.init.zeros_(m.bias)

Biases are set up to zero. There is no symmetry problem for them, since the weights now differ.

Special cases in modern designs

Batch normalisation's γ and β start at 1 and 0. So the layer initially passes its scaled input through unchanged.

The last block's normalisation in a residual network is sometimes set up to zero (γ = 0). It makes each residual block start as an exact identity. The network begins as a shallow one and deepens it as training proceeds. A trick that steadies very deep and very large nets.

Transformers often scale the setup of the output projections by 1/√(2 × layers), since the residual stream builds up contributions from each block and would otherwise grow with depth.

Embeddings are often set up from a normal spread with a small standard deviation (0.02 is the common choice in language nets).

Orthogonal setup for recurrent weight matrices keeps the largest singular value at 1. It is just the condition that stops the hidden state exploding or vanishing over time steps.

Exploration guide

  1. Train from a reasonable start. Press Reset Weights and then Train Network. Outputs stay in a usable range across the layers and the loss falls steadily.
  2. Watch the layer-to-layer scale. In training, check the output magnitudes at the first and last layers. When setup is right they are comparable. A systematic shrink or growth across depth is the failure this whole topic exists to stop.
  3. Add normalisation. Enable the batch-norm toggle and press Reset Weights, then train again. Convergence turns markedly less sensitive to where the weights started. Normalisation rescales each layer regardless.
  4. Reset repeatedly. Press Reset Weights a few times and train each time. Runs differ, sometimes noticeably. The initial draw is a genuine source of run-to-run variance. It is why single-run comparisons are unreliable.

Traps worth knowing

  • Zero setup. Symmetry never breaks and the layer collapses to a single effective neuron. Zero is fine for biases and wrong for weights.
  • A fixed standard deviation such as 0.01 all over. Ignores layer width. So wide layers explode and deep stacks vanish. This was standard before 2010 and is a large part of why deep networks did not train.
  • Xavier with ReLU. Understates the variance by a factor of 2, since Xavier does not account for ReLU discarding half the spread. Use He for ReLU.
  • Assuming batch norm makes it irrelevant. Normalisation reduces the response but does not remove it, very in very deep networks and in the layers before the first normalisation.
  • Reinitialising a pretrained layer. When fine-tuning, just the new head should be set up. Overwriting the pretrained weights discards all transfer learning was for.

What to remember

Setup must be random to break symmetry. And scaled to layer width so output and gradient variance stay roughly constant with depth. He setup is the default for ReLU networks since ReLU halves the variance and He’s factor of 2 restores it. Xavier suits tanh and sigmoid. Get this wrong and a deep network either learns nothing or diverges. Before the solver has had any say in the matter.

How much does it still matter?

Less than it did, and more than nothing.

Batch normalisation and residual connections both reduce response to setup. It is part of why they were such a breakthrough. They made deep networks trainable still from an imperfect starting point. Pretrained weights remove the question fully for transfer learning.

But it still matters in three cases: training from scratch without normalisation layers, very deep or very large nets where small per-layer errors compound. And recurrent networks where the same matrix is applied many times.

And the framework defaults are not always right. PyTorch's default for nn.Linear is a uniform spread based on fan-in that predates He setup. And it is not best for ReLU networks. Setting it clearly costs three lines.

Diagnosing a bad setup

Before training, do a single forward pass and print the standard deviation of the outputs at each layer:

acts = {}
for name, m in model.named_modules():
    if isinstance(m, nn.ReLU):
        m.register_forward_hook(
            lambda mod, i, o, n=name: acts.__setitem__(n, o.std().item()))

model(sample_batch)
for k, v in acts.items():
    print(f"{k:<40} std {v:.4f}")

Healthy: roughly close vals throughout. Shrinking towards zero with depth means the setup is too small (or Xavier was used with ReLU). Growing means it is too large. And the first gradient step will be violent.

The same check on the initial loss is worth doing: with C balanced classes it should be about ln(C). A much more value means the initial predictions are confidently wrong. It often means the output layer's weights are too large.

Zero, too big, too small, and right

Setup decides whether a deep network trains at all. Four schemes on the same design, measured by what happens to the outputs after ten layers.

example_01.pyNumPy
Output
The next block shows where the factor of 2 in He setup comes from.
example_02.pyNumPy
Output
Finally, the failure that is not about scale at all: all-zero weights and the symmetry problem.
example_03.pyNumPy
Output

                

                

                    
Questions people ask

                    
Can I set up all weights to the same non-zero value? No. The symmetry problem is identical to with zeros. Units must start other.
Should biases be zero? Yes, almost always. One exception: setting the output bias of a binary sorter to the log-odds of the positive class speeds up early training on imbalanced data.
Normal or uniform? Barely matters — the variance is what counts. Both are provided for both schemes.
What about fine-tuning? The pretrained weights are the setup. Just a newly added head needs initialising. And it should be small so it does not disturb the pretrained features on the first step.
Does setup affect the final result? With normalisation and residuals, often just mildly. Other seeds land at close quality. Without them, it can decide whether the net trains at all.
Why does my loss start enormous? Output-layer weights too large. Or a label/logit mismatch. Check against ln(count of classes).

                

                

                    
Recap in one screen

                    
  • Zeros break the network by symmetry. Wrong scales make outputs grow or vanish through depth.
  • He setup (variance 2/fan_in) for ReLU. Xavier (2/(fan_in+fan_out)) for tanh and sigmoid.
  • The factor of 2 in He exists since ReLU discards half the signal.
  • Biases start at zero. Residual blocks are sometimes started as exact identities.
  • Batch normalisation and residuals reduce response. But do not remove the need to get it roughly right.

                

            


        

    




        
    



Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
  1. Without scrolling back — what is the one-line takeaway from this module?
  2. What does this module say about “Why not zero, and why not all-equal”?
  3. What does this module say about “The variance is what really matters”?





 Cheat sheet
Weight Initialization Methods
Compare He, Xavier, random and zero initialisation, and watch poor starting weights kill a deep network before it learns.
DEEP LEARNING · vizlearn.in/deep_learning/weight_initialization.html
More in Deep Learning
Deep LearningVanishing & Exploding GradientsDeep LearningExploding and Vanishing GradientsDeep LearningGradient ClippingDeep LearningResidual and Skip Connections
Further reading
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact