Watch the training flow! Unscaled data causes first-layer weights to explode (Red), while poor activations cause deep gradients to vanish (Blue). Toggle the solutions below to see the network converge to stability (Green).
35
80k
Gradient Health
Global Status
STABLE
Waiting to start...
Average Weight Magnitude
W1 (Input → H1)1.000
W3 (H2 → H3)1.000
Color Legend
Vanish (~0)Stable (~1)Explode (>2)
Drag to Pan | Scroll to Zoom
Overview
Where both problems come from
Backpropagation computes the gradient at an early layer by multiplying together the local derivatives of each layer above it. For a network of depth L that is a product of L terms.
Products of many counts behave exponentially. If each term averages 0.5, then after 10 layers the gradient is scaled by 0.510 ≈ 0.001. And after 30 layers by 10−9 — effectively zero. If each term averages 1.5, after 30 layers the factor is around 191,000. And weights update so violently the loss turns NaN.
So vanishing and exploding gradients are not two problems. They are one problem — repeated multiplication. Falling either side of a knife edge at 1.
The solutions all aim to keep the per-layer factor close to 1. ReLU replaced sigmoid as the default output since its derivative is just 1 for positive inputs, passing gradients through without shrinking them. He and Xavier setup set weight variance so that the output variance of each layer matches its input, stopping the product from drifting. Batch normalisation re-centres outputs between layers. Residual connections add a skip path that carries the gradient past a block unchanged. And gradient clipping caps the norm of the gradient vector when it grows too large, stopping the single catastrophic update that turns the loss to NaN. This page shows the multiplication chain with concrete counts, checks sigmoid and ReLU derivatives side by side, shows the effect of depth on gradient magnitude. And lets you adjust output function and setup in the visualisation to watch the gradient signal survive or die across layers.
Vanishing & Exploding Gradients: A Practical Guide
Backpropagation multiplies gradients layer by layer. Multiply enough numbers below one and the signal disappears; enough above one and it blows up. Both failures come from the same chain rule.
Where both problems come from
Backpropagation computes the gradient at an early layer by multiplying together the local derivatives of each layer above it. For a network of depth L that is a product of L terms.
Products of many counts behave exponentially. If each term averages 0.5, then after 10 layers the gradient is scaled by 0.510 ≈ 0.001. And after 30 layers by 10−9 — effectively zero. If each term averages 1.5, after 30 layers the factor is around 191,000. And weights update so violently the loss turns NaN.
So vanishing and exploding gradients are not two problems. They are one problem — repeated multiplication. Falling either side of a knife edge at 1.
The solutions all aim to keep the per-layer factor close to 1. ReLU replaced sigmoid as the default output since its derivative is just 1 for positive inputs, passing gradients through without shrinking them. He and Xavier setup set weight variance so that the output variance of each layer matches its input, stopping the product from drifting. Batch normalisation re-centres outputs between layers. Residual connections add a skip path that carries the gradient past a block unchanged. And gradient clipping caps the norm of the gradient vector when it grows too large, stopping the single catastrophic update that turns the loss to NaN. This page shows the multiplication chain with concrete counts, checks sigmoid and ReLU derivatives side by side, shows the effect of depth on gradient magnitude. And lets you adjust output function and setup in the visualisation to watch the gradient signal survive or die across layers.
Why sigmoid made it worse
The derivative of the sigmoid is σ(x)(1 − σ(x)). It peaks at 0.25 when x = 0 and falls toward zero for inputs of large magnitude.
Its max is 0.25. So each sigmoid layer multiplies the gradient by at most a quarter. Ten layers means a factor of 0.2510 ≈ 10−6 in the best case. And far worse once units saturate. This is the concrete reason deep networks were considered untrainable before roughly 2010: the early layers got no usable learning signal at all.
ReLU changed that. Its derivative is just 1 for positive inputs. So the gradient passes through unattenuated however many layers it crosses. That single property, more than any other, is what made depth real.
Multiplication through depth
Backpropagation multiplies one factor per layer. That single fact explains both failures.
If each layer adds a factor of about 0.25. The max slope of the sigmoid. Then after ten layers the gradient reaching the first layer is:
0.2510 ≈ 0.00000095
The early layers get in effect nothing and stop learning. That is the vanishing gradient.
If each factor is about 2 instead, ten layers give 2¹⁰ = 1,024. And thirty layers give a billion. Weights take an enormous step, the loss turns NaN. And training is over in one update. That is the exploding gradient.
Depth
Factor 0.25
Factor 0.9
Factor 1.1
Factor 2
5
0.001
0.59
1.6
32
10
1e-6
0.35
2.6
1,024
50
8e-31
0.005
117
1e15
Just factors very close to 1 survive depth. Each technique below is a way of keeping them there.
Telling the two apart
Symptom
Vanishing
Exploding
Loss
Falls very slowly, or not at all
Spikes, then NaN
Early-layer gradients
Orders of magnitude smaller than late ones
Enormous
Weights
Early layers barely change
Blow up, then become NaN
Onset
Gradual, from the start
Sudden, often within a few steps
The diagnostic is the same for both. And it costs almost nothing:
for name, p in model.named_parameters():
if p.grad is not None:
print(f"{name:<40} {p.grad.norm().item():.3e}")
A healthy network shows close magnitudes across layers. A vanishing one shows a steady decay from output to input, often a few orders of magnitude. An exploding one shows vals in the thousands or inf.
The fixes for vanishing gradients
ReLU instead of sigmoid or tanh. Its derivative is just 1 for positive inputs. So active units add a factor of 1 rather than at most 0.25. This was the change that made deep networks trainable.
Residual connections.output = F(x) + x has derivative F'(x) + 1. That +1 is a path along which the gradient flows back unchanged however many blocks it crosses. This is why 100-layer networks train at all. And it is the single most effective fix.
Normalisation layers. Batch, layer or group normalisation keeps outputs in a range where derivatives stay near 1 rather than in a saturated region.
Careful setup. He setup scales the initial weights by √(2/fan_in). It keeps the variance of outputs roughly constant through depth for ReLU networks. Xavier does the same for tanh.
LSTM and GRU gates solve the sequential version: their additive cell-state path is the recurrent equivalent of a residual connection. It is why they handle far longer sequences than a plain RNN.
Exploration guide
Train without help. Leave the normalisation and ReLU toggles off and press Train Network. Watch the gradient magnitudes at the early layers. They are orders of magnitude less than at the output. So the first layers barely move.
Switch the output. Enable the ReLU toggle, press Reset. And train again. Gradients now reach the early layers at a usable size, since each layer multiplies by 1 rather than by at most 0.25.
Create a scale mismatch. Set Salary near 200000 while Age stays around 30, with normalisation off. One input is thousands of times more. So its gradients dominate and destabilise training.
Scale and repeat. Enable Normalize Data and train again with the same inputs. Both features now add comparable gradients. And the run is stable. Feature scaling is a gradient problem, not a cosmetic one.
The fixes that are really used
ReLU and its variants. A derivative of 1 in the positive region removes the systematic shrinkage. Leaky ReLU and GELU also avoid the dead-unit problem of a hard zero.
Careful setup. He setup for ReLU and Xavier for tanh set the initial weight variance so that outputs and gradients keep roughly constant scale across layers. Deliberately placing the product near 1.
Batch normalisation. Renormalising each layer’s inputs in training keeps outputs out of saturation and gradients in a usable range.
Residual connections. A skip connection gives the gradient an additive path around each block. So it reaches early layers without passing through each intermediate multiplication. This is what lets networks hundreds of layers deep.
Gradient clipping. For exploding gradients namely, capping the gradient norm at a threshold is a blunt fix that works reliably. Standard practice in recurrent networks.
What trips people up
Diagnosing vanishing gradients as a learning-rate problem. Raising the learning rate to compensate destabilises the layers that were training. Fix the output and setup instead.
With sigmoid or tanh in deep hidden layers. Reasonable for a shallow network or an output. A poor default in anything deep.
Ignoring a loss that suddenly turns NaN. That is almost always an exploding gradient. Clip the norm and check for unscaled inputs before touching the design.
Leaving inputs unscaled. A feature measured in the tens of thousands injects huge gradients at the first layer no matter how well the rest is designed.
What to remember
Backpropagation multiplies per-layer derivatives together. So anything always below 1 shrinks the gradient exponentially with depth and anything above 1 amplifies it. Sigmoid’s max derivative of 0.25 made deep networks untrainable. ReLU’s derivative of 1 fixed it. And setup, normalisation and residual connections keep the product near 1 by design. When training silently stalls suspect vanishing. When the loss turns NaN suspect exploding, and clip.
The fixes for exploding gradients
Gradient clipping is the direct and standard answer. Compute the norm of the whole gradient and, if it exceeds a threshold, scale it down proportionally:
Clipping by norm preserves the gradient's way and just limits its length. It is why it is preferred over clipping each value on its own. A threshold of 1.0 is the usual default in transformers and recurrent nets.
A less learning rate reduces the size of the step still when the gradient is large. Exploding loss within the first few dozen steps is almost always a learning rate problem.
Warm-up ramps the learning rate up over the first few hundred steps, avoiding a large early step when the net is badly set up and Adam's variance estimate is unreliable. Effectively mandatory for transformers.
Weight decay keeps weights small, which keeps the Jacobian factors less.
Careful setup again — weights set up too large give factors above 1 from the very first step.
Why depth was hard before 2015
The history is worth knowing since it explains the standard design of all today.
Networks deeper than a handful of layers did not train reliably. Sigmoid outputs vanished gradients. Poor setup made it worse. Recurrent networks could not learn dependencies more than a few steps apart.
Four changes fixed it, all replies to the multiplication problem: ReLU (2011), better setup (2010–2015), batch normalisation (2015) and residual connections (2015). Within a year, 152-layer networks were state of the art.
Transformers inherited all four. Each transformer block is x + sublayer(norm(x)). Residual plus normalisation — with GELU outputs and gradient clipping in training. The design is a direct answer to this topic.
Watch the signal die on its way back
Gradients are a product of per-layer terms. So a repeated multiplication either collapses to zero or blows up. This measures both on a 20-layer network and then fixes them.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
def run(depth, scale, act, name, n=256, width=64):
x = rng.normal(size=(n, width))
Ws = [rng.normal(0, scale, (width, width)) for _ in range(depth)]
acts, zs = [x], []
a = x
for W in Ws:
z = a @ W
zs.append(z)
a = act[0](z)
acts.append(a)
g = rng.normal(size=a.shape) / np.sqrt(n)
sizes = []
for i in range(depth - 1, -1, -1):
g = g * act[1](zs[i])
sizes.append(np.abs(g).mean())
g = g @ Ws[i].T
return name, sizes[::-1], np.abs(acts[-1]).mean()
sigmoid = (lambda z: 1 / (1 + np.exp(-z)),
lambda z: (1 / (1 + np.exp(-z))) * (1 - 1 / (1 + np.exp(-z))))
relu = (lambda z: np.maximum(0, z), lambda z: (z > 0).astype(float))
DEPTH = 20
print("a %d-layer network, 64 units wide. tracking the gradient's average" % DEPTH)
print("magnitude as it travels from the output back to layer 1.")
print()
print("%-34s %12s %12s %12s"
% ("setup", "at layer 20", "at layer 10", "at layer 1"))
for args in (
(DEPTH, 0.10, sigmoid, "sigmoid, small weights (sd 0.10)"),
(DEPTH, 1.00, sigmoid, "sigmoid, large weights (sd 1.00)"),
(DEPTH, 0.50, relu, "relu, weights sd 0.50"),
(DEPTH, np.sqrt(2 / 64), relu, "relu, He init (sd sqrt(2/64))"),
):
name, sizes, out = run(*args)
print("%-34s %12.3e %12.3e %12.3e" % (name, sizes[-1], sizes[9], sizes[0]))
Output
The gradient shrinks by a constant factor at each layer. It makes the total loss exponential in depth. One multiplication per layer is all it takes.
example_02.pyNumPy
import numpy as np
print("why it is exponential, in one line. each backward step multiplies by")
print("W.T and by the activation's derivative:")
for s, label in ((0.25, "sigmoid's best case (0.25)"), (0.9, "0.9"), (1.1, "1.1")):
print(" factor %-28s after 20 layers: %.3e" % (label, s ** 20))
print(" anything below 1 vanishes, anything above 1 explodes, and only")
print(" exactly 1 survives. that is a knife edge, and the whole of modern")
print(" architecture design is about staying near it.")
print()
print("the four things that actually fix it:")
print()
print("1. initialisation -- choose the weight scale so the factor starts at 1:")
for width in (64, 256, 1024):
print(" width %4d: He std = sqrt(2/%d) = %.4f" % (width, width, np.sqrt(2 / width)))
print()
print("2. activations with gradient 1 -- relu, not sigmoid (see the table).")
Output
Residual connections give the gradient a shortcut that bypasses the shrinking chain fully. It is what let networks grow beyond 20 layers.
example_03.pyNumPy
import numpy as np
def stack(residual, depth=20, width=64, n=128):
r2 = np.random.default_rng(1)
x = r2.normal(size=(n, width))
Ws = [r2.normal(0, 0.06, (width, width)) for _ in range(depth)]
zs, xs = [], [x]
a = x
for W in Ws:
z = a @ W
zs.append(z)
a = np.maximum(0, z) + (a if residual else 0.0)
xs.append(a)
g = r2.normal(size=a.shape) / np.sqrt(n)
for i in range(depth - 1, -1, -1):
gz = g * (zs[i] > 0)
skip = g if residual else 0.0
g = gz @ Ws[i].T + skip
return np.abs(g).mean()
plain, res = stack(False), stack(True)
print("3. residual connections -- same 20 layers, same small weights:")
print(" plain stack : gradient at layer 1 = %.3e" % plain)
print(" residual stack : gradient at layer 1 = %.3e" % res)
print(" %.0e times more signal reaches layer 1, from the same weights."
% (res / plain))
print(" the '+ skip' term is an unmultiplied copy of the gradient, so no")
print(" matter what the weights do there is always a path back with a")
print(" factor of exactly 1. that is why ResNet could train 152 layers")
print(" when 20 was previously hard.")
print()
print("4. normalisation -- rescale activations at every layer so the factor")
print(" cannot drift far from 1 in the first place.")
print()
print("exploding is the easier failure: it is loud, you see NaN, and clipping")
print("the gradient norm fixes it in one line. vanishing is silent -- the loss")
print("simply stops falling and the early layers stay at their initial values.")
Output
Questions people ask
Which is more common today? Exploding, in transformers and recurrent nets. It is why clipping is standard. Vanishing is largely solved by ReLU and residuals in feed-forward networks.
Does gradient clipping hurt training? Slightly, if the threshold is too low. You are discarding real signal. Log how often clipping triggers. At times is healthy, constantly means the threshold is too tight or the learning rate too high.
Can normalisation alone fix vanishing gradients? It helps much and is not enough on its own for very deep networks. Residual connections are what make 100+ layers work.
Why does my RNN fail on long sequences? The recurrent weight matrix's largest eigenvalue governs stability: above 1 the state explodes, below 1 it vanishes. Use an LSTM or GRU, or a transformer.
Is NaN always an exploding gradient? Not always. log(0), division by zero and sqrt of a negative also give it. Check the loss computation before blaming the gradients.
Should I clip by value or by norm? By norm. Clipping vals on its own distorts the gradient's way.
Recap in one screen
Backpropagation multiplies one factor per layer. So anything not close to 1 compounds catastrophically.
Factors below 1 vanish the gradient and freeze the early layers. Factors above 1 explode it into NaN.
Log per-layer gradient norms — a steady decay means vanishing, huge vals mean exploding.
ReLU, residual connections, normalisation and He setup address vanishing.
Gradient clipping, a less learning rate and warm-up address exploding.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Where both problems come from”?
What does this module say about “Why sigmoid made it worse”?
Cheat sheet
Vanishing & Exploding Gradients
Watch gradients shrink to nothing or blow up as they multiply back through layers, and see what keeps them in range.
DEEP LEARNING · vizlearn.in/deep_learning/vanishing_vs_exploding_gradient.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.