vizlearn
Home / Natural Language Processing

Vanishing Gradient Problem in RNN

Natural Language ProcessingBy Updated

The gradient shrinks (or explodes) geometrically with every timestep it travels back. Drag the sliders and watch w^T do its damage.

Overview

The Arithmetic of Forgetting

BPTT multiplies the gradient by ∂hₜ/∂hₜ₋₁ once per timestep. Call that factor's typical size w. After travelling back T steps the gradient is scaled by wT — a geometric series. At w = 0.7 and T = 30, that's 0.7³⁰ ≈ 0.00002: the beginning of the sentence receives two hundred-thousandths of the learning signal.

Gradient Dynamics

Recurrent factor |∂hₜ/∂hₜ₋₁| 0.70
0.1 (vanish)1.01.6 (explode)
Sequence length T 20

Diagnosis

Gradient surviving the full trip (wT)

-

Why It Matters

If the gradient from the end of a sentence cannot reach its beginning, the network cannot learn long-range dependencies — like matching "The cat ... was" across 30 words.

This is the problem that LSTM and GRU cells were invented to solve, using gates that let gradients flow undiminished.

Gradient Magnitude vs Distance Travelled Back

-
← t = 1 (start of sequence) t = T (loss is here) →

The Vanishing Gradient Problem: Why RNNs Forget

A geometric series is all it takes to erase long-term memory — and to motivate LSTMs.

Vanishing and Exploding: Two Sides of One Coin

  • w < 1 → vanishing: gradients decay exponentially; early timesteps stop learning; the network only captures short-range patterns.
  • w > 1 → exploding: gradients grow exponentially; updates become huge and training destabilizes into NaNs. (The blunt fix — gradient clipping — just caps the magnitude.)
  • w = 1 exactly: stable — but nothing keeps a trained weight matrix pinned there. The knife-edge is why vanilla RNNs are fundamentally fragile on long sequences.

The Way Out: Gates

LSTM and GRU cells attack the product itself: they add a cell state with additive updates and learned gates, creating a path where the effective factor stays near 1 for as long as the gates choose. Gradients ride this "constant error carousel" across hundreds of steps — which is why LSTMs dominated sequence learning for two decades, until attention offered an even more direct shortcut.

Multiplication through time

Training a recurrent network means unrolling it. A 50-step sequence becomes a 50-layer network sharing one weight matrix, and backpropagation multiplies one Jacobian factor per step.

The gradient reaching step 1 from a loss at step 50 is a product of 49 factors. If they average below 1, the product collapses:

Steps backFactor 0.5Factor 0.9Factor 1.1
100.0010.352.6
253e-80.0710.8
509e-160.005117

At 50 steps with a factor of 0.5 the gradient is one part in a quadrillion — indistinguishable from zero in floating point. The network cannot learn that step 1 mattered, so it cannot learn long-range dependencies at all.

Two things make factors below 1 the default. Tanh's derivative is at most 1 and usually well below it. And the recurrent weight matrix's largest singular value must be near 1 for the product to be stable, which nothing enforces.

What it looks like in practice

The symptoms are specific enough to diagnose:

  • The model learns local patterns and ignores distant ones. It predicts the next word well from the previous two and fails on agreement across a clause.
  • Early-layer or early-timestep gradients are orders of magnitude smaller than late ones.
  • Loss falls and then plateaus at a level that local context alone can explain.
  • Increasing the sequence length does not improve results, because the extra context is unreachable.

The diagnostic is to log gradient norms by time step or by layer:

loss.backward()
for name, p in model.named_parameters():
    if p.grad is not None:
        print(f"{name:<30} {p.grad.norm().item():.3e}")

A steady decay from output to input is the signature. Similar magnitudes throughout is healthy.

The mirror problem

The same multiplication explodes when the factors exceed 1. A factor of 1.1 over 50 steps gives 117; a factor of 2 gives 10¹⁵.

The result is one enormous weight update that pushes the model far outside any sensible region, the next forward pass produces inf, the loss becomes NaN, and the run is over — nothing recovers from NaN weights.

Exploding gradients are, in one sense, the easier problem: they are loud and immediately visible, and gradient clipping fixes them completely. Vanishing gradients are silent — the model trains, converges, and simply fails to learn what you wanted.

 VanishingExploding
SymptomSlow or no learning of long rangeNaN loss
VisibilitySilentImmediate
FixGating, architectureGradient clipping
OnsetFrom the startOften within a few steps

The five fixes, measured against each other

The problem is a product of terms that is almost never exactly 1. Each standard remedy attacks a different factor in that product, and this measures how much each one buys.

example_01.pyNumPy
Output

                

                

                    
Guided experiments

                    
  1. Start at w = 0.70, T = 20. The bars die out about a third of the way back — most of the sentence is unreachable by the learning signal.
  2. Enable log scale. The decay becomes a straight line — the visual signature of an exponential. Now drag T to 40 and read the survival value: it falls off a cliff.
  3. Push w to 1.3. The diagnosis flips to EXPLODING and the bars blow through the top of the chart. Then park w exactly at 1.0 and notice how artificial that stability feels — one nudge either way and it's gone.

                

                

                    
What to remember

                    
The vanishing gradient problem is not a bug or a tuning issue — it is the inescapable arithmetic of multiplying T numbers that aren't exactly 1. It caps how far back a vanilla RNN can learn, and it is the direct reason LSTMs, GRUs, and ultimately attention-based Transformers exist.

                

                

                    
How gating fixes it

                    
The LSTM's answer is an additive path. Its cell state updates as:
cᵗ = fᵗ ⊙ cᵗ₋₁ + iᵗ ⊙ gᵗ
Differentiating with respect to cₜ₋₁ gives the forget gate fₜ. If the network has learned to keep this information, fₜ is near 1, and the gradient passes back through that step essentially unchanged.
Contrast with a plain RNN, where the factor is the derivative of a tanh times a weight matrix — a quantity nothing keeps near 1.
The GRU achieves the same with its update gate: hₜ includes (1 − zₜ) ⊙ hₜ₋₁, so the derivative includes (1 − zₜ), near 1 when the state is being preserved.
This is structurally the same trick as a residual connection: an additive path whose derivative is 1, giving gradients a route that multiplication cannot crush. The transformer's x + sublayer(x), the LSTM's cell state and the GRU's interpolation are three expressions of one idea.
Gating reduces the problem rather than solving it. LSTMs handle hundreds of steps reliably and thousands unreliably, which is part of why attention — where any two positions are one step apart regardless of distance — eventually displaced them.

                

                

                    
The full set of countermeasures

                    
For vanishing:
  • Use an LSTM or GRU rather than a plain RNN. This is the main fix, and it is free.
  • Orthogonal initialisation of the recurrent matrix keeps its singular values at 1, so the product neither grows nor shrinks initially.
  • Shorter sequences, or truncated backpropagation through time, which bounds how far gradients must travel.
  • Skip connections across time steps in deeper recurrent stacks.
  • Attention, which removes the distance dependence entirely.
For exploding:
  • Gradient clipping by norm. Essentially mandatory for recurrent models; max_norm=1.0 to 5.0 is typical.
  • A lower learning rate.
  • Layer normalisation inside the recurrent cell.
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)
optimizer.step()

                

                

                    
Questions people ask

                    
Why does tanh cause vanishing gradients? Its derivative is at most 1 and approaches 0 when saturated, so the per-step factor is usually below 1.
Would ReLU fix it in an RNN? It removes the saturation problem and makes explosion far more likely, since the derivative is exactly 1 on the positive side and the same matrix is applied repeatedly. Gating is the better answer.
Do LSTMs eliminate the problem? They greatly reduce it. Very long dependencies remain hard, and clipping is still needed for the exploding direction.
How do transformers avoid it? Any two positions are one attention step apart, so there is no long product of factors to decay.
What is truncated BPTT? Backpropagating only k steps back rather than to the beginning. It bounds memory and compute, and prevents learning dependencies longer than k.
Is this the same problem as in deep feed-forward networks? Structurally identical — a product of many factors — and the fixes rhyme: ReLU and residual connections there, gating and clipping here.

                

                

                    
Recap in one screen

                    
  • Backpropagation through time multiplies one factor per step, so anything not near 1 compounds.
  • Factors below 1 vanish the gradient and make long-range learning impossible — silently.
  • Factors above 1 explode it into NaN — loudly, and clipping fixes it completely.
  • Gating creates an additive path whose derivative is the gate, near 1 when information is preserved.
  • That is the same idea as a residual connection, and attention removes the distance dependence altogether.

                

            


        

    



        
    



Recall check
0 of 4
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
  1. What is meant by “Use an LSTM or GRU” here?
  2. What is meant by “Orthogonal initialisation” here?
  3. What is meant by “Shorter sequences,” here?
  4. What is meant by “Attention,” here?





 Cheat sheet
Vanishing Gradient Problem in RNN
BPTT multiplies the gradient by ∂hₜ/∂hₜ₋₁ once per timestep. Call that factor's typical size w. After travelling back T steps the gradient is scaled by wT — a geometric series. At w = 0.7 and T = 30, that's 0.7³⁰ ≈ 0.00002: the beginning of the sentence receives two hundred-thousandths of the learning signal.
NLP · vizlearn.in/natural_language_processing/vanishing_gradient_problem_in_rnn.html
More in NLP
NLPWhat is LSTM?NLPForget Gate in LSTMNLPInput Gate in LSTMNLPCandidate Memory in LSTM
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact