vizlearn
Home / Natural Language Processing

How is Back-propagation done through Time?

Natural Language ProcessingBy Updated

To train a loop, unroll it. Run the forward pass, then send the gradient backwards through every timestep and watch the chain rule multiply.

Overview

The Problem With a Loop

Backpropagation works on feed-forward graphs — signals flow one way, gradients flow back the same way. An RNN's feedback loop breaks that picture. The fix, Backpropagation Through Time (BPTT), is beautifully blunt: unroll the loop into T copies of the cell, one per timestep, and the loop becomes an ordinary (deep) feed-forward chain that standard backprop can handle.

Controls

Sequence length T 5

The Chain Rule at Work

∂L/∂h₁ = ∂L/∂hₜ × ∏ ∂hₜ/∂hₜ₋₁

Product so far (factor 0.7 per step)

—

Every step back multiplies by ∂hₜ/∂hₜ₋₁ (≈ 0.7 here). Watch the product shrink as the gradient travels toward t = 1.

The Unrolled Network

IDLE

Narration

The same RNN cell, copied once per timestep. Press Run Forward Pass to feed the sequence through, left to right.

Gradient Magnitude Reaching Each Timestep

Backpropagation Through Time, Step by Step

How gradient descent trains a network that contains a loop.

Forward, Then Backward

  • Forward pass: tokens enter left to right; each cell computes hₜ from xₜ and hₜ₋₁; the loss L is measured at the end.
  • Backward pass: the gradient ∂L/∂h starts at the loss and travels right to left. Crossing each cell multiplies it by the local derivative ∂hₜ/∂hₜ₋₁ — the chain rule, applied once per timestep.
  • Weight update: because every unrolled copy shares the same W and U, their gradient contributions from all timesteps are summed before the single weight update.

Unrolling the loop

A recurrent network is a loop, and gradients cannot flow through a loop directly. So training unrolls it: a 50-step sequence becomes a 50-layer feed-forward network in which every layer shares the same weight matrix.

Ordinary backpropagation then applies to that unrolled graph. The only special feature is what happens to the shared weights.

Each weight's gradient is the sum of its gradients from every time step, because it was used at all of them:

∂L/∂W = Σᵗ ∂Lᵗ/∂W

That summation is the entire difference between BPTT and standard backpropagation. Everything else — the chain rule, the storage of activations, the update — is the same.

Two consequences follow directly. Memory grows with sequence length, because every step's activations must be kept for the backward pass. And the gradient for early steps is a long product of factors, which is where vanishing and exploding gradients come from.

Truncated BPTT

Backpropagating through a 10,000-step sequence is impossible: the memory required is proportional to the length.

Truncated BPTT bounds it by propagating only k steps back, then detaching the hidden state and continuing forward:

hidden = None
for chunk in chunks_of(sequence, size=50):        # k = 50
    out, hidden = model(chunk, hidden)
    loss = criterion(out, targets_for(chunk))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    hidden = tuple(h.detach() for h in hidden)     # cut the gradient path

The detach() is the whole mechanism. The hidden state continues forward, carrying information indefinitely, while the gradient stops at the chunk boundary.

The trade is explicit: the model can use context from arbitrarily far back in the forward direction, and it cannot learn dependencies longer than k, because no gradient connects them. Typical k is 20–100 — long enough for useful structure, short enough to be affordable.

Forgetting to detach is a real bug: the graph grows without bound, memory climbs every step, and eventually the process is killed.

Cost, and where it binds

AspectScaling
Forward computeLinear in sequence length
Backward computeLinear in sequence length
Activation memoryLinear in sequence length
Parallelism across stepsNone — strictly sequential

The last row is the one that mattered historically. Because step t needs hₜ₋₁, the steps cannot be computed simultaneously, so a GPU's thousands of cores sit mostly idle and training time scales with length.

That is a hardware-utilisation limitation rather than a quality one, and gating cannot fix it. It is the main reason transformers — where every position is computed in parallel — displaced recurrent networks for large-scale work, independent of any argument about long-range dependencies.

Exploration guide

  1. Run the forward pass. Cells light up green in sequence, ending at the loss node — this is the network "reading" the sentence.
  2. Backpropagate. Amber arrows sweep right to left. Watch the chain-rule product in the sidebar: 1.0, then 0.7, then 0.49... each timestep multiplies again.
  3. Set T = 7 and repeat. By the time the gradient reaches t = 1 it is 0.7⁶ ≈ 0.12 — barely a whisper. The bar chart makes the decay unmistakable. This shrinkage is the seed of the vanishing gradient problem, covered next.

Unrolling the loop to differentiate it

Training an RNN means treating the loop as a deep network with shared weights. This runs the full backward pass by hand, checks it numerically, and shows why the gradients from every timestep are summed rather than averaged.

example_01.pyNumPy
Output

                

                

                    
Worth remembering

                    
BPTT = unroll the loop, backpropagate through the chain, sum the shared-weight gradients. It makes RNNs trainable — but it also chains together T multiplications, and a long product of numbers below 1 races toward zero. That arithmetic inevitability is the vanishing gradient problem.

                

                

                    
Practical notes

                    
Clip gradients. The summation over time steps plus the product of factors makes large gradients common. clip_grad_norm_ with a threshold of 1–5 is standard, and omitting it eventually produces a NaN run.
Pack variable-length sequences. pack_padded_sequence stops the recurrence at each sequence's real end. Without it, gradients flow through padding steps and the final hidden state summarises the padding rather than the sentence.
Watch memory with sequence length. Doubling the sequence doubles the stored activations. If training fits at 200 steps and fails at 400, this is why — truncate, or reduce the batch size.
Detach the carried state between chunks in stateful training, or the graph never gets freed.
Consider gradient checkpointing for long sequences: store activations only every few steps and recompute the rest during the backward pass, trading about 30% more compute for a large memory saving.

                

                

                    
How it differs from feed-forward backpropagation

                    
 Feed-forwardThrough time
WeightsDistinct per layerShared across steps
Gradient for a weightFrom one placeSummed over all steps
DepthFixed by architectureSet by sequence length
MemoryFixedGrows with sequence length
ParallelismAcross the batchAcross the batch only
The shared weights are what create both the summation and the long product. A network with distinct weights per step would have neither problem — and would need a separate parameter set for every possible sequence length, which is exactly what parameter sharing exists to avoid.

                

                

                    
Questions people ask

                    
Why is it called "through time"? Because the unrolled layers correspond to time steps rather than to distinct network layers.
What value of k should I use? 20–100 typically. Long enough to cover the dependencies you care about, short enough to fit in memory.
Does truncation hurt? It prevents learning dependencies longer than k. The forward state still carries information further, so it is a limit on learning rather than on use.
Do transformers need BPTT? No — there is no recurrence, so ordinary backpropagation applies and all positions are computed in parallel.
Why does my memory grow every batch? A carried hidden state that was never detached, so the graph accumulates.
Is gradient clipping always needed? For recurrent models, effectively yes.

                

                

                    
Recap in one screen

                    
  • BPTT unrolls the recurrence into a deep network sharing one weight matrix, then backpropagates normally.
  • A shared weight's gradient is the sum of its contributions from every time step.
  • Memory and compute grow linearly with sequence length, and the steps cannot be parallelised.
  • Truncated BPTT bounds cost by detaching the state every k steps — and caps what can be learned at k.
  • Clip gradients, pack variable-length batches, and always detach carried state.

                

            


        

    



        
    



Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
  1. Without scrolling back — what is the one-line takeaway from this module?
  2. What does this module say about “The Problem With a Loop”?
  3. What does this module say about “Forward, Then Backward”?





 Cheat sheet
Backpropagation Through Time
To train a loop, unroll it. Run the forward pass, then send the gradient backwards through every timestep and watch the chain rule multiply.
NLP · vizlearn.in/natural_language_processing/backpropagation_through_time.html
More in NLP
NLPVanishing Gradient Problem in RNNNLPWhat is LSTM?NLPForget Gate in LSTMNLPInput Gate in LSTM
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact