Quick Context
Neural network weights must be initialized before training begins. The initialization strategy determines the scale and distribution of starting weights, which directly affects whether gradients flow properly through the network or vanish/explode in the first few iterations.
1) Why Not Initialize to Zero?
If all weights are zero (or any constant), every neuron in a layer computes the exact same output. During backpropagation, they all receive the same gradient and update identically. The network never breaks this symmetry — it's equivalent to having a single neuron per layer.
Random initialization breaks symmetry, ensuring each neuron learns a different feature. But the scale of that randomness matters enormously.
2) Common Initialization Methods
- Random Normal / Uniform — Draw from N(0, σ²) or U(-a, a). Simple but the choice of σ or a is ad-hoc and often wrong for deep networks.
- Xavier/Glorot (2010) — Designed for sigmoid/tanh. Sets variance to 2/(fan_in + fan_out) where fan_in and fan_out are the number of input and output neurons. Keeps activation variance roughly constant across layers.
- He/Kaiming (2015) — Designed for ReLU. Sets variance to 2/fan_in, accounting for the fact that ReLU zeros out half the activations. Standard for modern deep networks.
- Orthogonal Initialization — Initializes weight matrices as orthogonal matrices. Preserves gradient norms exactly, popular in RNNs.
3) The Math Behind Xavier and He
Xavier (Glorot):
W ~ N(0, 2 / (fan_in + fan_out))
or U(-√(6/(fan_in+fan_out)), √(6/(fan_in+fan_out)))
He (Kaiming):
W ~ N(0, 2 / fan_in)
or U(-√(6/fan_in), √(6/fan_in))
The key insight: if the variance of activations changes as data flows forward, gradients will either vanish or explode as they flow backward. These formulas keep the variance ≈ 1 at every layer.
4) Matching Initialization to Activation
| Activation | Recommended Init | Why |
| ReLU, Leaky ReLU | He (Kaiming) | Accounts for ReLU zeroing half the outputs |
| Sigmoid, Tanh | Xavier (Glorot) | Keeps outputs in the linear region |
| SELU | LeCun Normal | Variance = 1/fan_in for self-normalizing |
5) Guided Experiments
- Try zero initialization — notice all neurons converge to the same value (symmetry problem).
- Try large random init (high variance) — observe gradients exploding in early layers.
- Try small random init (low variance) — observe gradients vanishing in deep layers.
- Switch to He init with ReLU — all layers should maintain stable, green gradient health.
6) Common Mistakes
- Using Xavier init with ReLU — the variance computation doesn't account for ReLU, so activations shrink over layers.
- Initializing biases to large values — biases should almost always start at zero.
- Using the same initialization for every layer type — conv layers, dense layers, and embedding layers may have different optimal strategies.
7) Key Takeaways
- Never initialize all weights to zero — it causes a symmetry problem.
- Use He init for ReLU networks, Xavier for sigmoid/tanh.
- The goal: keep activation and gradient variance ≈ 1 across all layers.
- Modern frameworks default to good initializations, but understanding why matters for debugging.