To train a loop, unroll it. Run the forward pass, then send the gradient backwards through every timestep and watch the chain rule multiply.
Controls
Sequence length T5
The Chain Rule at Work
∂L/∂h₁ = ∂L/∂hₜ × ∏ ∂hₜ/∂hₜ₋₁
Product so far (factor 0.7 per step)
—
Every step back multiplies by ∂hₜ/∂hₜ₋₁ (≈ 0.7 here). Watch the product shrink as the gradient travels toward t = 1.
The Unrolled Network
IDLE
Narration
The same RNN cell, copied once per timestep. Press Run Forward Pass to feed the sequence through, left to right.
Gradient Magnitude Reaching Each Timestep
Overview
The Problem With a Loop
Backpropagation works on feed-forward graphs — signals flow one way, gradients flow back the same way. An RNN's feedback loop breaks that picture. The fix, Backpropagation Through Time (BPTT), is beautifully blunt: unroll the loop into T copies of the cell, one per timestep. And the loop becomes an ordinary (deep) feed-forward chain that standard backprop can handle. Each unrolled copy shares the same weight matrices W and U. So the gradients from all timesteps are summed into a single weight update. The chain rule applied across T copies gives a product of T Jacobians. And that product is why RNNs struggle with long sequences. If the largest singular value of the recurrent Jacobian is below 1 the gradients shrink fast (vanishing gradients). Above 1 they grow fast (exploding gradients, handled by gradient clipping). Truncated BPTT limits the unrolling to a fixed window of k steps. This caps the memory cost and the gradient path length at the expense of long-range learning. LSTMs and GRUs address the same problem architecturally, by adding gates that let gradients flow through an additive path rather than a multiplicative one. This page unrolls a small RNN, shows the gradient flowing backward through each timestep, and plots the gradient magnitude decaying or exploding as the sequence grows.
How gradient descent trains a network that contains a loop.
The Problem With a Loop
Backpropagation works on feed-forward graphs — signals flow one way, gradients flow back the same way. An RNN's feedback loop breaks that picture. The fix, Backpropagation Through Time (BPTT), is beautifully blunt: unroll the loop into T copies of the cell, one per timestep. And the loop becomes an ordinary (deep) feed-forward chain that standard backprop can handle. Each unrolled copy shares the same weight matrices W and U. So the gradients from all timesteps are summed into a single weight update. The chain rule applied across T copies gives a product of T Jacobians. And that product is why RNNs struggle with long sequences. If the largest singular value of the recurrent Jacobian is below 1 the gradients shrink fast (vanishing gradients). Above 1 they grow fast (exploding gradients, handled by gradient clipping). Truncated BPTT limits the unrolling to a fixed window of k steps. This caps the memory cost and the gradient path length at the expense of long-range learning. LSTMs and GRUs address the same problem architecturally, by adding gates that let gradients flow through an additive path rather than a multiplicative one. This page unrolls a small RNN, shows the gradient flowing backward through each timestep, and plots the gradient magnitude decaying or exploding as the sequence grows.
Forward, Then Backward
Forward pass: tokens enter left to right; each cell computes hₜ from xₜ and hₜ₋₁. The loss L is measured at the end.
Backward pass: the gradient ∂L/∂h starts at the loss and travels right to left. Crossing each cell multiplies it by the local derivative ∂hₜ/∂hₜ₋₁ — the chain rule, applied once per timestep.
Weight update: because every unrolled copy shares the same W and U, their gradient contributions from all timesteps are summed before the single weight update.
Unrolling the loop
A recurrent network is a loop, and gradients cannot flow through a loop right. So training unrolls it: a 50-step sequence becomes a 50-layer feed-forward network in which every layer shares the same weight matrix.
Ordinary backpropagation then applies to that unrolled graph. The only special feature is what happens to the shared weights.
Each weight's gradient is the sum of its gradients from every time step, because it was used at all of them:
∂L/∂W = Σᵗ ∂Lᵗ/∂W
That summation is the entire gap between BPTT and standard backpropagation. All of it else — the chain rule, the storage of activations, the update — is the same.
Two results follow right. Memory grows with sequence length, because every step's activations must be kept for the backward pass. And the gradient for early steps is a long product of factors. This is where vanishing and exploding gradients come from.
Truncated BPTT
Backpropagating through a 10,000-step sequence is not doable: the memory required is tied to to the length.
Truncated BPTT bounds it by propagating only k steps back. Then detaching the hidden state and going on forward:
hidden = None
for chunk in chunks_of(sequence, size=50): # k = 50
out, hidden = model(chunk, hidden)
loss = criterion(out, targets_for(chunk))
optimizer.zero_grad()
loss.backward()
optimizer.step()
hidden = tuple(h.detach() for h in hidden) # cut the gradient path
The detach() is the whole method. The hidden state goes on forward, carrying data indefinitely, while the gradient stops at the chunk boundary.
The trade is explicit: the model can use context from arbitrarily far back in the forward direction. And it cannot learn dependencies longer than k, because no gradient connects them. Typical k is 20–100 — long enough for useful structure, short enough to be affordable.
Forgetting to detach is a real bug: the graph grows without bound, memory climbs every step, and in time the process is killed.
Cost, and where it binds
Aspect
Scaling
Forward compute
Linear in sequence length
Backward compute
Linear in sequence length
Activation memory
Linear in sequence length
Parallelism across steps
None — strictly sequential
The last row is the one that mattered historically. Because step t needs hₜ₋₁, the steps cannot be computed at once. So a GPU's thousands of cores sit mostly idle and training time scales with length.
That is a hardware-utilisation limitation rather than a quality one, and gating cannot fix it. It is the main reason transformers. Where every position is computed in parallel. Displaced recurrent networks for large-scale work, free of any argument about long-range dependencies.
Exploration guide
Run the forward pass. Cells light up green in sequence, ending at the loss node — this is the network "reading" the sentence.
Backpropagate. Amber arrows sweep right to left. Watch the chain-rule product in the sidebar: 1.0. Then 0.7, then 0.49... each timestep multiplies again.
Set T = 7 and repeat. By the time the gradient reaches t = 1 it is 0.7⁶ ≈ 0.12 — barely a whisper. The bar chart makes the decay unmistakable. This shrinkage is the seed of the vanishing gradient problem, covered next.
Unrolling the loop to differentiate it
Training an RNN means treating the loop as a deep network with shared weights. This runs the full backward pass by hand, checks it in numbers, and shows why the gradients from every timestep are summed rather than averaged.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
X, H, T = 2, 3, 14
Wx = rng.normal(0, 0.5, (X, H))
Wh = rng.normal(0, 0.5, (H, H))
Wy = rng.normal(0, 0.5, (H, 1))
b = np.zeros(H)
seq = rng.normal(0, 1.0, (T, X))
target = 0.7
H0 = rng.normal(0, 0.4, H) # a non-zero initial state, so step 0 is
# not degenerate in the gradient below
def forward(Wx, Wh, Wy, b):
h = H0.copy()
hs = [h.copy()]
for x in seq:
h = np.tanh(x @ Wx + h @ Wh + b)
hs.append(h.copy())
out = hs[-1] @ Wy
return hs, out, float((out - target) ** 2)
hs, out, loss = forward(Wx, Wh, Wy, b)
print("a %d-step RNN, one prediction from the final state." % T)
print(" output %.6f, target %.1f, loss %.6f" % (out, target, loss))
print()
print("UNROLLED, this is a %d-layer network where every layer uses the SAME"
% T)
print("Wh. the gradient for Wh gets a contribution from every step.")
Output
Now backpropagate through all T steps and watch how each step's contribution to the gradient shrinks the further back it is.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
X, H, T = 2, 3, 14
Wx, Wh = rng.normal(0, 0.5, (X, H)), rng.normal(0, 0.5, (H, H))
Wy, b = rng.normal(0, 0.5, (H, 1)), np.zeros(H)
seq, target = rng.normal(0, 1.0, (T, X)), 0.7
H0 = rng.normal(0, 0.4, H)
def forward(Wx, Wh, Wy, b):
h = H0.copy()
hs = [h.copy()]
for x in seq:
h = np.tanh(x @ Wx + h @ Wh + b)
hs.append(h.copy())
return hs, hs[-1] @ Wy, float((hs[-1] @ Wy - target) ** 2)
hs, out, _ = forward(Wx, Wh, Wy, b)
dh = (2 * (out - target) * Wy).ravel()
contribs = []
for t in range(T - 1, -1, -1):
dpre = dh * (1 - hs[t + 1] ** 2)
contribs.append((t, np.abs(np.outer(hs[t], dpre)).max()))
dh = dpre @ Wh.T
print("%10s %26s" % ("step", "its contribution to dWh"))
biggest = max(m for _, m in contribs)
for t, mag in contribs:
if t in (T - 1, T - 2, T - 4, T - 7, T // 2, 2, 1, 0):
print("%10d %26.8f %s"
% (t, mag, "#" * max(0, int(40 * mag / biggest))))
print()
late = np.mean([m for t, m in contribs if t >= T - 3])
early = np.mean([m for t, m in contribs if t <= 2])
print(" last 3 steps avg %.6f; first 3 avg %.6f -- factor of %.0f."
% (late, early, late / max(early, 1e-30)))
print(" that decay is the vanishing gradient: early steps barely influence")
print(" the update, so long-range dependencies cannot be learned.")
Output
We can verify those analytic gradients by nudging each weight and measuring the loss change right.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
X, H, T = 2, 3, 14
Wx, Wh = rng.normal(0, 0.5, (X, H)), rng.normal(0, 0.5, (H, H))
Wy, b, seq, target = rng.normal(0, 0.5, (H, 1)), np.zeros(H), rng.normal(0, 1.0, (T, X)), 0.7
H0 = rng.normal(0, 0.4, H)
def forward(Wx, Wh, Wy, b):
h = H0.copy(); hs = [h.copy()]
for x in seq:
h = np.tanh(x @ Wx + h @ Wh + b); hs.append(h.copy())
return hs, hs[-1] @ Wy, float((hs[-1] @ Wy - target) ** 2)
hs, out, _ = forward(Wx, Wh, Wy, b)
dWy = hs[-1].reshape(-1, 1) * 2 * (out - target)
dh = (2 * (out - target) * Wy).ravel()
dWx, dWh, db = np.zeros_like(Wx), np.zeros_like(Wh), np.zeros_like(b)
for t in range(T - 1, -1, -1):
dpre = dh * (1 - hs[t + 1] ** 2)
dWh += np.outer(hs[t], dpre); dWx += np.outer(seq[t], dpre); db += dpre
dh = dpre @ Wh.T
def numeric(param, i, j, h=1e-6):
up = [Wx.copy(), Wh.copy(), Wy.copy(), b.copy()]
dn = [Wx.copy(), Wh.copy(), Wy.copy(), b.copy()]
up[param][i, j] += h; dn[param][i, j] -= h
return (forward(*up)[2] - forward(*dn)[2]) / (2 * h)
print("CHECK. nudge each weight and measure the loss directly:")
print("%10s %8s %18s %18s %12s"
% ("param", "index", "analytic", "numeric", "difference"))
worst = 0.0
for name, p, grad, (i, j) in (("Wx", 0, dWx, (0, 0)), ("Wh", 1, dWh, (0, 0)),
("Wh", 1, dWh, (2, 1)), ("Wy", 2, dWy, (1, 0))):
a, n = grad[i, j], numeric(p, i, j)
worst = max(worst, abs(a - n))
print("%10s %8s %18.10f %18.10f %12.2e"
% (name, "[%d,%d]" % (i, j), a, n, abs(a - n)))
print(" worst disagreement: %.2e -- floating point noise." % worst)
Output
In practice, running the full backward pass over thousands of steps is too expensive. So training uses truncated BPTT.
example_04.pyNumPy
T = 14
print("WHY SUM AND NOT AVERAGE. Wh is one matrix used %d times. the" % T)
print("derivative of the loss with respect to it is the sum over every use:")
print(" dL/dWh = sum over t of (dL/dWh at step t)")
print(" averaging would silently divide your learning rate by the sequence")
print(" length, and it would make gradients from a 10-token and a")
print(" 1000-token sequence incomparable.")
print()
print("TRUNCATED BPTT, because the full version is impractical:")
print("%14s %16s %20s %s" % ("truncation k", "memory", "gradient reaches", ""))
for k in (1, 5, 25, "full"):
kk = T if k == "full" else k
print("%14s %16s %20s %s"
% (k, "%d states" % min(kk, T), "%d steps back" % min(kk, T),
"<- exact" if k == "full" else ""))
print(" truncation caps memory at k states, and accepts that dependencies")
print(" longer than k are invisible to training.")
print()
print(" the usual arrangement carries STATE forward across chunks while")
print(" cutting the GRADIENT at the boundary:")
print(" forward: state flows through the whole sequence")
print(" backward: stops after k steps")
print(" the model can still USE long context at inference; it just cannot")
print(" LEARN dependencies longer than k.")
print()
print("and that is the honest limit. an LSTM extends how far the gradient")
print("survives; truncation caps how far it is allowed to travel. attention")
print("removes the question by giving every pair of positions a direct path.")
Output
Worth recalling
BPTT = unroll the loop, backpropagate through the chain, sum the shared-weight gradients. It makes RNNs trainable — but it also chains together T products. And a long product of numbers below 1 races toward zero. That arithmetic inevitability is the vanishing gradient problem.
Practical notes
Clip gradients. The summation over time steps plus the product of factors makes large gradients common. clip_grad_norm_ with a threshold of 1–5 is standard, and omitting it in time gives a NaN run.
Pack variable-length sequences.pack_padded_sequence stops the recurrence at each sequence's real end. Without it, gradients flow through padding steps and the final hidden state summarises the padding rather than the sentence.
Watch memory with sequence length. Doubling the sequence doubles the stored activations. If training fits at 200 steps and fails at 400, this is why. Truncate, or reduce the batch size.
Detach the carried state between chunks in stateful training, or the graph never gets freed.
Consider gradient checkpointing for long sequences: store activations only every few steps and recompute the rest during the backward pass. This trades about 30% more compute for a large memory saving.
How it differs from feed-forward backpropagation
Feed-forward
Through time
Weights
Distinct per layer
Shared across steps
Gradient for a weight
From one place
Summed over all steps
Depth
Fixed by architecture
Set by sequence length
Memory
Fixed
Grows with sequence length
Parallelism
Across the batch
Across the batch only
The shared weights are what create both the summation and the long product. A network with distinct weights per step would have neither problem. And would need a separate parameter set for every doable sequence length. This is exactly what parameter sharing exists to avoid.
Questions people ask
Why is it called "through time"? Because the unrolled layers correspond to time steps rather than to distinct network layers.
What value of k should I use? 20–100 usually. Long enough to cover the dependencies you care about, short enough to fit in memory.
Does truncation hurt? It prevents learning dependencies longer than k. The forward state still carries data further, so it is a limit on learning rather than on use.
Do transformers need BPTT? No — there is no recurrence, so ordinary backpropagation applies and all positions are computed in parallel.
Why does my memory grow every batch? A carried hidden state that was never detached. So the graph builds up.
Is gradient clipping always needed? For recurrent models, in effect yes.
Recap in one screen
BPTT unrolls the recurrence into a deep network sharing one weight matrix, then backpropagates normally.
A shared weight's gradient is the sum of its contributions from every time step.
Memory and compute grow linearly with sequence length, and the steps cannot be parallelised.
Truncated BPTT bounds cost by detaching the state every k steps. And caps what can be learned at k.
Clip gradients, pack variable-length batches, and always detach carried state.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Problem With a Loop”?
What does this module say about “Forward, Then Backward”?
What does this module say about “Unrolling the loop”?
Cheat sheet
Backpropagation Through Time
Unroll an RNN's loop into T copies and send the gradient backward through each timestep, watching it shrink or explode along the way.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.