Quick Context
During backpropagation, gradients are multiplied through each layer via the chain rule. If these multiplied factors are consistently < 1, gradients shrink to near-zero (vanishing). If > 1, they blow up (exploding). Both make training impossible for deep architectures.
1) The Chain Rule Problem
For a network with L layers, the gradient of the loss w.r.t. the first layer's weights involves a product of L partial derivatives:
∂L/∂W₁ = ∂L/∂aₗ · ∂aₗ/∂aₗ₋₁ · ... · ∂a₂/∂a₁ · ∂a₁/∂W₁
If each factor |∂aᵢ/∂aᵢ₋₁| ≈ 0.5, then after 10 layers: 0.5¹⁰ ≈ 0.001 — the gradient practically disappears. If each factor ≈ 2, then 2¹⁰ ≈ 1024 — the gradient explodes.
2) What Causes Vanishing Gradients
- Sigmoid/tanh activations — their derivatives are always < 1 (sigmoid max derivative is 0.25). Stacking many layers multiplies these small values.
- Poor weight initialization — weights that are too small shrink activations at each layer.
- Very deep networks — more layers = more multiplicative factors < 1.
Symptoms: early layers stop learning while later layers train normally; loss plateaus very early.
3) What Causes Exploding Gradients
- Large weight initialization — weights > 1 amplify signals at each layer.
- No normalization on large-scale inputs — the first layer's weights grow disproportionately.
- Recurrent networks (RNNs) — unrolling over many time steps has the same multiplicative effect.
Symptoms: loss becomes NaN, weights overflow, updates are wildly large.
4) Solutions
- ReLU activation — derivative is 1 for positive inputs, avoiding the < 1 multiplication problem.
- Proper weight initialization — He init for ReLU, Xavier/Glorot for tanh/sigmoid.
- Batch Normalization — rescales activations at each layer to maintain healthy gradient flow.
- Gradient clipping — caps gradient magnitude to prevent explosions (essential for RNNs).
- Residual connections (skip connections) — provide a direct path for gradients, used in ResNets.
- LSTM/GRU gates — in RNNs, gating mechanisms control gradient flow over long sequences.
5) Guided Experiments
- Train with default settings — observe which layers go red (exploding) or blue (vanishing).
- Toggle different initialization strategies and observe how gradient health changes across layers.
- Enable Batch Normalization and watch all layers stabilize to green.
- Try gradient clipping and see how it caps the maximum gradient magnitude.
6) Common Mistakes
- Using sigmoid in deep networks (20+ layers) — almost guaranteed vanishing gradients.
- Not monitoring gradient norms during training — you won't know there's a problem until the model fails to converge.
- Setting gradient clip value too low — clips useful gradient information too.
7) Key Takeaways
- Vanishing and exploding gradients are caused by repeated multiplication through layers via the chain rule.
- ReLU, proper initialization, BatchNorm, and skip connections are the modern toolkit to combat this.
- Always monitor gradient norms — healthy gradients stay within a consistent range across layers.