Watch the training flow! Unscaled data causes first-layer weights to explode (Red), while poor activations cause deep gradients to vanish (Blue). Toggle the solutions below to see the network converge to stability (Green).
Overview
Where both problems come from
Backpropagation computes the gradient at an early layer by multiplying together the local derivatives of every layer above it. For a network of depth L that is a product of L terms.
Products of many numbers behave exponentially. If each term averages 0.5, then after 10 layers the gradient is scaled by 0.510 ≈ 0.001, and after 30 layers by 10−9 — effectively zero. If each term averages 1.5, after 30 layers the factor is around 191,000, and weights update so violently the loss becomes NaN.
So vanishing and exploding gradients are not two problems. They are one problem — repeated multiplication — falling either side of a knife edge at 1.
35
80k
Gradient Health
Global Status
STABLE
Waiting to start...
Average Weight Magnitude
W1 (Input → H1)1.000
W3 (H2 → H3)1.000
Color Legend
Vanish (~0)Stable (~1)Explode (>2)
Drag to Pan | Scroll to Zoom
Vanishing & Exploding Gradients: A Practical Guide
Backpropagation multiplies gradients layer by layer. Multiply enough numbers below one and the signal disappears; enough above one and it blows up. Both failures come from the same chain rule.
Why sigmoid made it worse
The derivative of the sigmoid is σ(x)(1 − σ(x)), which peaks at 0.25 when x = 0 and falls toward zero for inputs of large magnitude.
Its maximum is 0.25, so every sigmoid layer multiplies the gradient by at most a quarter. Ten layers means a factor of 0.2510 ≈ 10−6 in the best case, and far worse once units saturate. This is the concrete reason deep networks were considered untrainable before roughly 2010: the early layers received no usable learning signal at all.
ReLU changed that. Its derivative is exactly 1 for positive inputs, so the gradient passes through unattenuated however many layers it crosses. That single property, more than any other, is what made depth practical.
Multiplication through depth
Backpropagation multiplies one factor per layer. That single fact explains both failures.
If each layer contributes a factor of about 0.25 — the maximum slope of the sigmoid — then after ten layers the gradient reaching the first layer is:
0.2510 ≈ 0.00000095
The early layers receive essentially nothing and stop learning. That is the vanishing gradient.
If each factor is about 2 instead, ten layers give 2¹⁰ = 1,024, and thirty layers give a billion. Weights take an enormous step, the loss becomes NaN, and training is over in one update. That is the exploding gradient.
Depth
Factor 0.25
Factor 0.9
Factor 1.1
Factor 2
5
0.001
0.59
1.6
32
10
1e-6
0.35
2.6
1,024
50
8e-31
0.005
117
1e15
Only factors very close to 1 survive depth. Every technique below is a way of keeping them there.
Telling the two apart
Symptom
Vanishing
Exploding
Loss
Falls very slowly, or not at all
Spikes, then NaN
Early-layer gradients
Orders of magnitude smaller than late ones
Enormous
Weights
Early layers barely change
Blow up, then become NaN
Onset
Gradual, from the start
Sudden, often within a few steps
The diagnostic is the same for both, and it costs almost nothing:
for name, p in model.named_parameters():
if p.grad is not None:
print(f"{name:<40} {p.grad.norm().item():.3e}")
A healthy network shows similar magnitudes across layers. A vanishing one shows a steady decay from output to input, often several orders of magnitude. An exploding one shows values in the thousands or inf.
The fixes for vanishing gradients
ReLU instead of sigmoid or tanh. Its derivative is exactly 1 for positive inputs, so active units contribute a factor of 1 rather than at most 0.25. This was the change that made deep networks trainable.
Residual connections.output = F(x) + x has derivative F'(x) + 1. That +1 is a path along which the gradient flows back unchanged however many blocks it crosses. This is why 100-layer networks train at all, and it is the single most effective fix.
Normalisation layers. Batch, layer or group normalisation keeps activations in a range where derivatives stay near 1 rather than in a saturated region.
Careful initialisation. He initialisation scales the initial weights by √(2/fan_in), which keeps the variance of activations roughly constant through depth for ReLU networks. Xavier does the same for tanh.
LSTM and GRU gates solve the sequential version: their additive cell-state path is the recurrent equivalent of a residual connection, which is why they handle far longer sequences than a plain RNN.
Exploration guide
Train without help. Leave the normalisation and ReLU toggles off and press Train Network. Watch the gradient magnitudes at the early layers — they are orders of magnitude smaller than at the output, so the first layers barely move.
Switch the activation. Enable the ReLU toggle, press Reset, and train again. Gradients now reach the early layers at a usable size, because each layer multiplies by 1 rather than by at most 0.25.
Create a scale mismatch. Set Salary near 200000 while Age stays around 30, with normalisation off. One input is thousands of times larger, so its gradients dominate and destabilise training.
Normalise and repeat. Enable Normalize Data and train again with the same inputs. Both features now contribute comparable gradients, and the run is stable — feature scaling is a gradient problem, not a cosmetic one.
The fixes that are actually used
ReLU and its variants. A derivative of 1 in the positive region removes the systematic shrinkage. Leaky ReLU and GELU additionally avoid the dead-unit problem of a hard zero.
Careful initialisation. He initialisation for ReLU and Xavier for tanh set the initial weight variance so that activations and gradients keep roughly constant scale across layers — deliberately placing the product near 1.
Batch normalisation. Renormalising each layer’s inputs during training keeps activations out of saturation and gradients in a usable range.
Residual connections. A skip connection gives the gradient an additive path around each block, so it reaches early layers without passing through every intermediate multiplication. This is what allows networks hundreds of layers deep.
Gradient clipping. For exploding gradients specifically, capping the gradient norm at a threshold is a blunt fix that works reliably — standard practice in recurrent networks.
What trips people up
Diagnosing vanishing gradients as a learning-rate problem. Raising the learning rate to compensate destabilises the layers that were training. Fix the activation and initialisation instead.
Using sigmoid or tanh in deep hidden layers. Reasonable for a shallow network or an output; a poor default in anything deep.
Ignoring a loss that suddenly becomes NaN. That is almost always an exploding gradient. Clip the norm and check for unscaled inputs before touching the architecture.
Leaving inputs unscaled. A feature measured in the tens of thousands injects huge gradients at the first layer regardless of how well the rest is designed.
What to remember
Backpropagation multiplies per-layer derivatives together, so anything consistently below 1 shrinks the gradient exponentially with depth and anything above 1 amplifies it. Sigmoid’s maximum derivative of 0.25 made deep networks untrainable; ReLU’s derivative of 1 fixed it, and initialisation, normalisation and residual connections keep the product near 1 by design. When training silently stalls suspect vanishing; when the loss becomes NaN suspect exploding, and clip.
The fixes for exploding gradients
Gradient clipping is the direct and standard answer. Compute the norm of the whole gradient and, if it exceeds a threshold, scale it down proportionally:
Clipping by norm preserves the gradient's direction and only limits its length, which is why it is preferred over clipping each value independently. A threshold of 1.0 is the usual default in transformers and recurrent models.
A lower learning rate reduces the size of the step even when the gradient is large. Exploding loss within the first few dozen steps is almost always a learning rate problem.
Warm-up ramps the learning rate up over the first few hundred steps, avoiding a large early step when the model is badly initialised and Adam's variance estimate is unreliable. Effectively mandatory for transformers.
Weight decay keeps weights small, which keeps the Jacobian factors smaller.
Careful initialisation again — weights initialised too large produce factors above 1 from the very first step.
Why depth was hard before 2015
The history is worth knowing because it explains the standard architecture of everything today.
Networks deeper than a handful of layers did not train reliably. Sigmoid activations vanished gradients; poor initialisation made it worse; recurrent networks could not learn dependencies more than a few steps apart.
Four changes fixed it, all responses to the multiplication problem: ReLU (2011), better initialisation (2010–2015), batch normalisation (2015) and residual connections (2015). Within a year, 152-layer networks were state of the art.
Transformers inherited all four. Every transformer block is x + sublayer(norm(x)) — residual plus normalisation — with GELU activations and gradient clipping during training. The architecture is a direct answer to this topic.
Watch the signal die on its way back
Gradients are a product of per-layer terms, so a repeated multiplication either collapses to zero or blows up. This measures both on a 20-layer network and then fixes them.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
def run(depth, scale, act, name, n=256, width=64):
# forward a batch, then propagate a gradient back and log its size
x = rng.normal(size=(n, width))
Ws = [rng.normal(0, scale, (width, width)) for _ in range(depth)]
acts, zs = [x], []
a = x
for W in Ws:
z = a @ W
zs.append(z)
a = act[0](z)
acts.append(a)
g = rng.normal(size=a.shape) / np.sqrt(n)
sizes = []
for i in range(depth - 1, -1, -1):
g = g * act[1](zs[i])
sizes.append(np.abs(g).mean())
g = g @ Ws[i].T
return name, sizes[::-1], np.abs(acts[-1]).mean()
sigmoid = (lambda z: 1 / (1 + np.exp(-z)),
lambda z: (1 / (1 + np.exp(-z))) * (1 - 1 / (1 + np.exp(-z))))
relu = (lambda z: np.maximum(0, z), lambda z: (z > 0).astype(float))
DEPTH = 20
print("a %d-layer network, 64 units wide. tracking the gradient's average" % DEPTH)
print("magnitude as it travels from the output back to layer 1.")
print()
print("%-34s %12s %12s %12s"
% ("setup", "at layer 20", "at layer 10", "at layer 1"))
for args in (
(DEPTH, 0.10, sigmoid, "sigmoid, small weights (sd 0.10)"),
(DEPTH, 1.00, sigmoid, "sigmoid, large weights (sd 1.00)"),
(DEPTH, 0.50, relu, "relu, weights sd 0.50"),
(DEPTH, np.sqrt(2 / 64), relu, "relu, He init (sd sqrt(2/64))"),
):
name, sizes, out = run(*args)
print("%-34s %12.3e %12.3e %12.3e" % (name, sizes[-1], sizes[9], sizes[0]))
print()
name, sizes, _ = run(DEPTH, 0.10, sigmoid, "x")
print("the sigmoid + small weights row, layer by layer:")
for i in range(0, DEPTH, 4):
print(" layer %2d: %.3e" % (DEPTH - i, sizes[DEPTH - 1 - i]))
print(" layer 1: %.3e" % sizes[0])
print(" it shrinks by a roughly constant factor per layer, which makes the")
print(" total an exponential. after %d layers it is %.1e of what it started"
% (DEPTH, sizes[0] / sizes[-1]))
print(" as -- the first layer receives no usable signal and never learns.")
print()
print("why it is exponential, in one line. each backward step multiplies by")
print("W.T and by the activation's derivative:")
for s, label in ((0.25, "sigmoid's best case (0.25)"), (0.9, "0.9"), (1.1, "1.1")):
print(" factor %-28s after 20 layers: %.3e" % (label, s ** 20))
print(" anything below 1 vanishes, anything above 1 explodes, and only")
print(" exactly 1 survives. that is a knife edge, and the whole of modern")
print(" architecture design is about staying near it.")
print()
print("the four things that actually fix it:")
print()
print("1. initialisation -- choose the weight scale so the factor starts at 1:")
for width in (64, 256, 1024):
print(" width %4d: He std = sqrt(2/%d) = %.4f" % (width, width, np.sqrt(2 / width)))
print()
print("2. activations with gradient 1 -- relu, not sigmoid (see the table).")
print()
print("3. residual connections -- measured, not asserted. same 20 layers and")
print(" the same undersized weights (sd 0.06), with and without a skip")
print(" path around each block:")
def stack(residual, depth=20, width=64, n=128):
r2 = np.random.default_rng(1)
x = r2.normal(size=(n, width))
Ws = [r2.normal(0, 0.06, (width, width)) for _ in range(depth)]
zs, xs = [], [x]
a = x
for W in Ws:
z = a @ W
zs.append(z)
a = np.maximum(0, z) + (a if residual else 0.0)
xs.append(a)
g = r2.normal(size=a.shape) / np.sqrt(n)
for i in range(depth - 1, -1, -1):
gz = g * (zs[i] > 0) # through the relu branch
skip = g if residual else 0.0 # and straight through the skip
g = gz @ Ws[i].T + skip
return np.abs(g).mean()
plain, res = stack(False), stack(True)
print(" plain stack : gradient at layer 1 = %.3e" % plain)
print(" residual stack : gradient at layer 1 = %.3e" % res)
print(" %.0e times more signal reaches layer 1, from the same weights."
% (res / plain))
print(" the '+ skip' term is an unmultiplied copy of the gradient, so no")
print(" matter what the weights do there is always a path back with a")
print(" factor of exactly 1. that is why ResNet could train 152 layers")
print(" when 20 was previously hard.")
print()
print("4. normalisation -- rescale activations at every layer so the factor")
print(" cannot drift far from 1 in the first place.")
print()
print("exploding is the easier failure: it is loud, you see NaN, and clipping")
print("the gradient norm fixes it in one line. vanishing is silent -- the loss")
print("simply stops falling and the early layers stay at their initial values.")
Output
Questions people ask
Which is more common today? Exploding, in transformers and recurrent models, which is why clipping is standard. Vanishing is largely solved by ReLU and residuals in feed-forward networks.
Does gradient clipping hurt training? Slightly, if the threshold is too low — you are discarding real signal. Log how often clipping triggers; occasionally is healthy, constantly means the threshold is too tight or the learning rate too high.
Can normalisation alone fix vanishing gradients? It helps considerably and is not sufficient on its own for very deep networks. Residual connections are what make 100+ layers work.
Why does my RNN fail on long sequences? The recurrent weight matrix's largest eigenvalue governs stability: above 1 the state explodes, below 1 it vanishes. Use an LSTM or GRU, or a transformer.
Is NaN always an exploding gradient? Not always — log(0), division by zero and sqrt of a negative also produce it. Check the loss computation before blaming the gradients.
Should I clip by value or by norm? By norm. Clipping values independently distorts the gradient's direction.
Recap in one screen
Backpropagation multiplies one factor per layer, so anything not close to 1 compounds catastrophically.
Factors below 1 vanish the gradient and freeze the early layers; factors above 1 explode it into NaN.
Log per-layer gradient norms — a steady decay means vanishing, huge values mean exploding.
ReLU, residual connections, normalisation and He initialisation address vanishing.
Gradient clipping, a lower learning rate and warm-up address exploding.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Backpropagation multiplies per-layer derivatives together, so anything consistently below 1 shrinks the gradient exponentially with depth and anything above 1 amplifies it. Sigmoid’s maximum derivative of 0.25 made deep networks untrainable; ReLU’s derivative of 1 fixed it, and initialisation, normalisation and residual connections keep the product near 1 by design.
What does this module say about “Where both problems come from”?
Backpropagation computes the gradient at an early layer by multiplying together the local derivatives of every layer above it. For a network of depth L that is a product of L terms.
What does this module say about “Why sigmoid made it worse”?
The derivative of the sigmoid is σ(x)(1 − σ(x)) , which peaks at 0.25 when x = 0 and falls toward zero for inputs of large magnitude.
Cheat sheet
Vanishing & Exploding Gradients
Backpropagation computes the gradient at an early layer by multiplying together the local derivatives of every layer above it. For a network of depth L that is a product of L terms.
DEEP LEARNING · vizlearn.in/deep_learning/vanishing_vs_exploding_gradient.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.