Home / Natural Language Processing

Vanishing Gradient Problem in RNN

The gradient shrinks (or explodes) geometrically with every timestep it travels back. Drag the sliders and watch w^T do its damage.

Gradient Dynamics

Recurrent factor |∂hₜ/∂hₜ₋₁| 0.70

0.1 (vanish)1.01.6 (explode)

Sequence length T 20

Log scale

Diagnosis

Gradient surviving the full trip (w^T)

Why It Matters

If the gradient from the end of a sentence cannot reach its beginning, the network cannot learn long-range dependencies — like matching "The cat ... was" across 30 words.

This is the problem that LSTM and GRU cells were invented to solve, using gates that let gradients flow undiminished.

Gradient Magnitude vs Distance Travelled Back

← t = 1 (start of sequence) t = T (loss is here) →

The Vanishing Gradient Problem: Why RNNs Forget

A geometric series is all it takes to erase long-term memory — and to motivate LSTMs.

The Arithmetic of Forgetting

BPTT multiplies the gradient by ∂hₜ/∂hₜ₋₁ once per timestep. Call that factor's typical size w. After travelling back T steps the gradient is scaled by w^T — a geometric series. At w = 0.7 and T = 30, that's 0.7³⁰ ≈ 0.00002: the beginning of the sentence receives two hundred-thousandths of the learning signal.

Vanishing and Exploding: Two Sides of One Coin

w < 1 → vanishing: gradients decay exponentially; early timesteps stop learning; the network only captures short-range patterns.
w > 1 → exploding: gradients grow exponentially; updates become huge and training destabilizes into NaNs. (The blunt fix — gradient clipping — just caps the magnitude.)
w = 1 exactly: stable — but nothing keeps a trained weight matrix pinned there. The knife-edge is why vanilla RNNs are fundamentally fragile on long sequences.

The Way Out: Gates

LSTM and GRU cells attack the product itself: they add a cell state with additive updates and learned gates, creating a path where the effective factor stays near 1 for as long as the gates choose. Gradients ride this "constant error carousel" across hundreds of steps — which is why LSTMs dominated sequence learning for two decades, until attention offered an even more direct shortcut.

Interactive Exploration Guide

Start at w = 0.70, T = 20. The bars die out about a third of the way back — most of the sentence is unreachable by the learning signal.
Enable log scale. The decay becomes a straight line — the visual signature of an exponential. Now drag T to 40 and read the survival value: it falls off a cliff.
Push w to 1.3. The diagnosis flips to EXPLODING and the bars blow through the top of the chart. Then park w exactly at 1.0 and notice how artificial that stability feels — one nudge either way and it's gone.

Key Takeaway

The vanishing gradient problem is not a bug or a tuning issue — it is the inescapable arithmetic of multiplying T numbers that aren't exactly 1. It caps how far back a vanilla RNN can learn, and it is the direct reason LSTMs, GRUs, and ultimately attention-based Transformers exist.

Cheat sheet