Home / Natural Language Processing

Vanishing Gradient Problem in RNN

Natural Language ProcessingBy Ashish JangraUpdated 30 September 2026

The gradient shrinks (or explodes) geometrically with every timestep it travels back. Drag the sliders and watch w^T do its damage.

Gradient Dynamics

Recurrent factor |∂hₜ/∂hₜ₋₁| 0.70

0.1 (vanish)1.01.6 (explode)

Sequence length T 20

Log scale

Diagnosis

Gradient surviving the full trip (w^T)

Why It Matters

If the gradient from the end of a sentence cannot reach its beginning, the network cannot learn long-range dependencies — like matching "The cat ... was" across 30 words.

This is the problem that LSTM and GRU cells were invented to solve, using gates that let gradients flow undiminished.

Gradient Magnitude vs Distance Travelled Back

← t = 1 (start of sequence) t = T (loss is here) →

Overview

The gradient shrinks (or explodes) geometrically with every timestep it travels back.

In this guide

The Arithmetic of Forgetting
Vanishing and Exploding: Two Sides of One Coin
The Way Out: Gates
Product through time
What it looks like in practice
The mirror problem
The five fixes, measured against each other
Guided tests
What to remember
How gating fixes it
The full set of countermeasures
Questions people ask
Recap in one screen

The Vanishing Gradient Problem: Why RNNs Forget

A geometric series is all it takes to erase long-term memory — and to motivate LSTMs.

The Arithmetic of Forgetting

BPTT multiplies the gradient by ∂hₜ/∂hₜ₋₁ once per timestep. Call that factor's typical size w. After travelling back T steps the gradient is scaled by w^T — a geometric series. At w = 0.7 and T = 30, that's 0.7³⁰ ≈ 0.00002. The beginning of the sentence receives two hundred-thousandths of the learning signal. Exploding gradients are the mirror case: when w > 1, the geometric series grows without bound. And the weight update becomes catastrophically large. Gradient clipping (capping the gradient norm before the update) is the standard fix for exploding gradients, but there is no even fix for vanishing. You cannot amplify a signal that has already been times to zero. The LSTM solves this architecturally: its cell state flows through an additive path (C_t = f_t * C_{t-1} + i_t * candidate) rather than a multiplicative one. So the gradient along the cell state does not pass through a weight matrix at every step. The forget gate controls how much of the old state survives. And when it is near 1, the gradient flows through almost undiminished —. This is exactly why biasing the forget gate toward 1 at initialisation is so important. GRUs achieve the same effect with fewer parameters. This page computes the gradient magnitude at each timestep, shows how it decays geometrically, and covers the LSTM's additive path as the architectural solution.

Vanishing and Exploding: Two Sides of One Coin

w < 1 → vanishing: gradients decay fast; early timesteps stop learning; the network only captures short-range patterns.
w > 1 → exploding: gradients grow fast; updates become huge and training destabilizes into NaNs. (The blunt fix — gradient clipping — just caps the magnitude.)
w = 1 exactly: stable — but nothing keeps a trained weight matrix pinned there. The knife-edge is why vanilla RNNs are at its core fragile on long sequences.

The Way Out: Gates

LSTM and GRU cells attack the product itself: they add a cell state with additive updates and learned gates. This creates a path where the effective factor stays near 1 for if the gates choose. Gradients ride this "constant error carousel" across hundreds of steps —. This is why LSTMs dominated sequence learning for two decades, until attention offered an even more direct shortcut.

Product through time

Training a recurrent network means unrolling it. A 50-step sequence becomes a 50-layer network sharing one weight matrix, and backpropagation multiplies one Jacobian factor per step.

The gradient reaching step 1 from a loss at step 50 is a product of 49 factors. If they average below 1, the product collapses:

Steps back	Factor 0.5	Factor 0.9	Factor 1.1
10	0.001	0.35	2.6
25	3e-8	0.07	10.8
50	9e-16	0.005	117

At 50 steps with a factor of 0.5 the gradient is one part in a quadrillion. Indistinguishable from zero in floating point. The network cannot learn that step 1 mattered, so it cannot learn long-range dependencies at all.

Two things make factors below 1 the default. Tanh's derivative is at most 1 and usually well below it. And the recurrent weight matrix's largest singular value must be near 1 for the product to be stable, which nothing enforces.

What it looks like in practice

The symptoms are specific enough to diagnose:

The model learns local patterns and ignores distant ones. It predicts the next word well from the previous two and fails on agreement across a clause.
Early-layer or early-timestep gradients are orders of magnitude smaller than late ones.
Loss falls and then plateaus at a level that local context alone can explain.
Increasing the sequence length does not improve results, because the extra context is unreachable.

The diagnostic is to log gradient norms by time step or by layer:

loss.backward()
for name, p in model.named_parameters():
    if p.grad is not None:
        print(f"{name:<30} {p.grad.norm().item():.3e}")

A steady decay from output to input is the signature. Similar magnitudes throughout is healthy.

The mirror problem

The same product explodes when the factors exceed 1. A factor of 1.1 over 50 steps gives 117; a factor of 2 gives 10¹⁵.

The result is one enormous weight update that pushes the model far outside any sensible region, the next forward pass gives inf, the loss becomes NaN. And the run is over. Nothing recovers from NaN weights.

Exploding gradients are, in one sense, the easier problem. They are loud and at once visible, and gradient clipping fixes them fully. Vanishing gradients are silent — the model trains, converges, and simply fails to learn what you wanted.

	Vanishing	Exploding
Symptom	Slow or no learning of long range	`NaN` loss
Visibility	Silent	Immediate
Fix	Gating, architecture	Gradient clipping
Onset	From the start	Often within a few steps

The five fixes, measured against each other

The problem is a product of terms that is almost never exactly 1. Each standard remedy attacks a different factor in that product. And this measures how much each one buys.

example_01.pyNumPy

import numpy as np

rng = np.random.default_rng(0)
H, T = 16, 60

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -30, 30)))

print("the gradient from step %d back to step 0 is a product of %d terms."
      % (T, T))
print("each term is  W.T  times the activation's derivative:")
print("    dh_t/dh_(t-1) = W.T * f'(pre_t)")
print()
print("so the whole thing is governed by two numbers -- the weight matrix's")
print("spectral radius and the activation's slope -- multiplied %d times."
      % T)
print()
print("%14s %20s %20s" % ("per-step factor", "after 20 steps", "after 60 steps"))
for f in (0.5, 0.9, 0.99, 1.0, 1.01, 1.1):
    print("%14.2f %20.3e %20.3e" % (f, f ** 20, f ** 60))
print("   only 1.00 survives. everything below vanishes and everything above")
print("   explodes, and the gap between 0.99 and 1.01 is the whole design")
print("   space.")
print()

Output

The next block goes on with the left demonstrations.

example_02.pyNumPy

import numpy as np

def measure(mode, depth=T, width=H, scale=None):
    r = np.random.default_rng(1)
    # every non-orthogonal mode uses `scale`, so runs are comparable
    if mode == "orthogonal":
        A = r.normal(size=(width, width))
        W, _ = np.linalg.qr(A)                       # spectral radius exactly 1
    else:
        W = r.normal(0, scale if scale else 1.0 / np.sqrt(width), (width, width))
    h = r.normal(0, 0.5, width)
    J = np.eye(width)
    sizes = []
    for t in range(depth):
        pre = h @ W
        if mode in ("tanh", "orthogonal", "small", "large"):
            h = np.tanh(pre)
            D = np.diag(1 - h ** 2)
        elif mode == "relu":
            h = np.maximum(0, pre)
            D = np.diag((pre > 0).astype(float))
        elif mode == "clipped":
            h = np.tanh(pre)
            D = np.diag(1 - h ** 2)
        else:
            raise ValueError(mode)
        J = J @ (W @ D)
        n = np.abs(J).max()
        if mode == "clipped" and n > 1.0:            # renormalise, as clipping does
            J = J / n
            n = 1.0
        sizes.append(n)
    return sizes
print("done")

Output

The next block goes on with the left demonstrations.

example_03.pyNumPy

import numpy as np

print("MEASURED. gradient magnitude reaching step 0, over %d steps:" % T)
setups = [
    ("tanh, sd 1/sqrt(n)", dict(mode="tanh")),
    ("tanh, sd 0.5 (small)", dict(mode="small", scale=0.5 / np.sqrt(H))),
    ("tanh, sd 2.0 (large)", dict(mode="large", scale=2.0 / np.sqrt(H))),
    ("relu, sd 1/sqrt(n)", dict(mode="relu")),
    ("orthogonal W, tanh", dict(mode="orthogonal")),
]
print("%24s %14s %14s %14s" % ("setup", "step 40", "step 20", "step 0"))
for name, kw in setups:
    s = measure(**kw)
    print("%24s %14.3e %14.3e %14.3e"
          % (name, s[T - 40], s[T - 20], s[-1]))
print()
print("   read the last column. the orthogonal initialisation is the only")
print("   one that keeps the gradient anywhere near its starting size, and")
print("   that is exactly what it was designed for: a spectral radius of")
print("   precisely 1, so the W part of the product contributes nothing.")
print("   tanh's derivative still shrinks it, which is why orthogonal init")
print("   helps and does not solve.")
print()
print("   and note the relu row -- it vanishes as badly as anything else.")
print("   relu's derivative is exactly 1 where a unit is active, but it is")
print("   exactly 0 where the unit is not, and roughly half of them are not")
print("   at any step. so the product still collapses. relu fixes the")
print("   vanishing gradient in a FEED-FORWARD stack, where each layer has")
print("   its own weights; in a recurrent one, where the same W is applied")
print("   %d times, it is not enough on its own." % T)
print()

Output

The next block goes on with the left demonstrations.

example_04.pyNumPy

import numpy as np


print("THE FIVE FIXES, and which factor each attacks:")
rows = [
    ("orthogonal init", "the W factor", "sets spectral radius to 1"),
    ("relu / leaky relu", "the f' factor", "derivative is 1 where active --"),
    ("", "", "but see the caveat below"),
    ("gradient clipping", "explosion only", "caps the norm, keeps direction"),
    ("gating (LSTM/GRU)", "replaces the product", "additive state, gradient"),
    ("", "", "passes through a forget gate"),
    ("skip connections", "adds a path of 1", "gradient has a route that"),
    ("", "", "is never multiplied"),
]
print("%22s %22s %s" % ("fix", "what it attacks", "how"))
for a, b, c in rows:
    print("%22s %22s %s" % (a, b, c))
print()

Output

The next block goes on with the left demonstrations.

example_05.pyNumPy

import numpy as np

print("GATING IS THE ONE THAT CHANGED THINGS, because it replaces the")
print("matrix product with a product of scalars the model chooses:")
print("%18s %18s %18s" % ("mechanism", "per-step factor", "after 60 steps"))
print("%18s %18s %18.3e" % ("plain RNN", "W.T * tanh'", measure(mode="tanh")[-1]))
for f in (0.9, 0.99, 0.999):
    print("%18s %18.3f %18.3e" % ("LSTM forget gate", f, f ** T))
print("   the LSTM's number is a single scalar per unit per step, learned,")
print("   and nothing stops it sitting at 0.999. the RNN's is an eigenvalue")
print("   it does not control.")
print()

print("EXPLOSION IS THE EASY HALF. it is loud, and one line fixes it:")
big = measure(mode="large", scale=2.0 / np.sqrt(H))
print("   unclipped, large weights: gradient reaches %.3e" % big[-1])
clipped = measure(mode="clipped", scale=2.0 / np.sqrt(H))
print("   with the norm clipped to 1 at every step: %.3e" % clipped[-1])
print("   clipping bounds how far one update can move the weights, whatever")
print("   the data does. there is no equivalent trick for vanishing --")
print("   you cannot scale a gradient of 1e-30 back up, because the")
print("   direction information in it is already gone to rounding.")
print()
print("that asymmetry is why the architectures changed. exploding gradients")
print("were patched; vanishing ones needed the LSTM, and then attention.")

Output

Guided tests

Start at w = 0.70, T = 20. The bars die out about a third of the way back — most of the sentence is unreachable by the learning signal.
Enable log scale. The decay becomes a straight line — the visual signature of an rapid. Now drag T to 40 and read the survival value: it falls off a cliff.
Push w to 1.3. The diagnosis flips to EXPLODING and the bars blow through the top of the chart. Then park w exactly at 1.0 and notice how artificial that stability feels — one nudge either way and it's gone.

What to remember

The vanishing gradient problem is not a bug or a tuning issue — it is the inescapable arithmetic of multiplying T numbers that aren't exactly 1. It caps how far back a vanilla RNN can learn. And it is the direct reason LSTMs, GRUs, and in the end attention-based Transformers exist.

How gating fixes it

The LSTM's answer is an additive path. Its cell state updates as:

cᵗ = fᵗ ⊙ cᵗ₋₁ + iᵗ ⊙ gᵗ

Differentiating about cₜ₋₁ gives the forget gate fₜ. If the network has learned to keep this data, fₜ is near 1. And the gradient passes back through that step in effect unchanged.

Contrast with a plain RNN, where the factor is the derivative of a tanh times a weight matrix. A quantity nothing keeps near 1.

The GRU achieves the same with its update gate: hₜ includes (1 − zₜ) ⊙ hₜ₋₁. So the derivative includes (1 − zₜ), near 1 when the state is being preserved.

This is structurally the same trick as a residual connection: an additive path whose derivative is 1. This gives gradients a route that product cannot crush. The transformer's x + sublayer(x), the LSTM's cell state and the GRU's interpolation are three expressions of one idea.

Gating reduces the problem rather than solving it. LSTMs handle hundreds of steps reliably and thousands unreliably, which is part of why attention. Where any two positions are one step apart no matter distance. In time displaced them.

The full set of countermeasures

For vanishing:

Use an LSTM or GRU rather than a plain RNN. This is the main fix, and it is free.
Orthogonal initialisation of the recurrent matrix keeps its singular values at 1. So the product neither grows nor shrinks at first.
Shorter sequences, or truncated backpropagation through time, which bounds how far gradients must travel.
Skip connections across time steps in deeper recurrent stacks.
Attention, which removes the distance dependence fully.

For exploding:

Gradient clipping by norm. Practically mandatory for recurrent models; max_norm=1.0 to 5.0 is typical.
A lower learning rate.
Layer normalisation inside the recurrent cell.

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)
optimizer.step()

Questions people ask

Why does tanh cause vanishing gradients? Its derivative is at most 1 and approaches 0 when saturated. So the per-step factor is usually below 1.

Would ReLU fix it in an RNN? It removes the saturation problem and makes explosion far more likely. Since the derivative is exactly 1 on the positive side and the same matrix is applied repeatedly. Gating is the better answer.

Do LSTMs remove the problem? They greatly reduce it. Very long dependencies remain hard, and clipping is still needed for the exploding direction.

How do transformers avoid it? Any two positions are one attention step apart. So there is no long product of factors to decay.

What is truncated BPTT? Backpropagating only k steps back rather than to the beginning. It bounds memory and compute, and prevents learning dependencies longer than k.

Is this the same problem as in deep feed-forward networks? Structurally the same. A product of many factors. And the fixes rhyme: ReLU and residual connections there, gating and clipping here.

Recap in one screen

Backpropagation through time multiplies one factor per step, so anything not near 1 compounds.
Factors below 1 vanish the gradient and make long-range learning impossible — silently.
Factors above 1 explode it into NaN — loudly, and clipping fixes it fully.
Gating creates an additive path whose derivative is the gate, near 1 when data is preserved.
That is the same idea as a residual connection, and attention removes the distance dependence altogether.

Recall check

0 of 4

Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.

What is meant by “Use an LSTM or GRU” here?
What is meant by “Orthogonal initialisation” here?
What is meant by “Shorter sequences,” here?
What is meant by “Attention,” here?

Cheat sheet