Backpropagation and the Computational Graph
Six steps forward to a loss, six steps back to the gradients. Every arrow carries one local derivative, and the chain rule is the multiplication along the way.
Overview
Quick Context
Gradient descent needs a gradient: for every weight in the network, how much the loss would change if that weight moved a little. Backpropagation is how that number is obtained, for millions of weights, at the cost of roughly one extra forward pass.
It is not a learning rule and not an optimiser. It computes derivatives, and nothing else. What to do with them is the optimiser's business.
Run The Pass
Press Step. The forward pass computes a value at each node; the backward pass then walks the same graph in reverse, multiplying local derivatives as it goes.
Inputs
every value and gradient recomputes as you drag — press Reset to walk the whole pass again
The Computational Graph
—Green is the value flowing forward. Orange is the gradient flowing back.
Forward
Gradients
Gradient Check
Nudge w1 by ±0.0001, recompute the loss both times, and divide the difference by 0.0002. It should match the chain-rule answer above — and it is the standard way to catch a bug in a hand-written backward pass.
Backpropagation: A Practical Guide
How a network works out what every weight did wrong, in one pass backwards.
Everything is a graph
A network is a long expression, and any expression can be drawn as a graph of small operations: multiply, add, apply a function. The graph above is a single neuron with two inputs, which is already enough to show every mechanic that matters.
m₁ = w₁x₁ m₂ = w₂x₂ s = m₁+m₂ z = s+b a = σ(z) L = (a − y)²
The forward pass fills in a value at every node. Crucially, those values are kept — the backward pass needs them, which is exactly why training a network takes far more memory than running one.
Each operation knows one thing
Every node needs only its local derivative: how its output responds to its own inputs. It has no idea what the rest of the network looks like, and does not need to.
- Add passes the gradient through unchanged to both inputs. This is why ∂L/∂b equals ∂L/∂z exactly, every time.
- Multiply hands each input the gradient scaled by the other input. So ∂L/∂w₁ = ∂L/∂m₁ · x₁ — a weight's gradient is proportional to the input it was multiplying, which is why an input of zero produces a gradient of zero and why unscaled features distort learning.
- The sigmoid contributes a(1 − a), which peaks at 0.25 and falls to nearly nothing when the neuron saturates.
The chain rule then says: the gradient arriving at a node, times the node's local derivative, is the gradient leaving it. Backpropagation is that single sentence applied repeatedly, in reverse topological order.
Assigning blame to every weight
Training needs one number per weight: if I nudge this weight slightly, how much does the loss change? With ten million weights, computing that separately ten million times is impossible.
Backpropagation gets all of them in a single backward pass, at roughly the cost of one forward pass. That efficiency is the reason deep learning is practical at all.
The mechanism is the chain rule. A network is nested functions, so the effect of an early weight on the loss is a product of factors, one per layer in between:
∂L/∂w(1) = ∂L/∂a(3) × ∂a(3)/∂a(2) × ∂a(2)/∂a(1) × ∂a(1)/∂w(1)
Computed naively, weights in the same layer would each redo most of that product. Backpropagation works from the loss backwards, carrying the accumulated product with it, and hands each layer exactly the factor it needs. Nothing is recomputed.
One neuron, worked through with numbers
A single neuron: z = wx + b, a = sigmoid(z), loss L = (a − y)².
Take x = 2, w = 0.5, b = 0, y = 1.
Forward: z = 1.0, a = sigmoid(1.0) = 0.731, L = (0.731 − 1)² = 0.072.
Backward, three factors:
- ∂L/∂a = 2(a − y) = 2(−0.269) = −0.538
- ∂a/∂z = a(1 − a) = 0.731 × 0.269 = 0.197
- ∂z/∂w = x = 2
Multiply: ∂L/∂w = −0.538 × 0.197 × 2 = −0.212
Negative, so increasing w lowers the loss. With a learning rate of 0.1 the update is w ← 0.5 + 0.0212 = 0.521 — a small step in the right direction, which is exactly what gradient descent is.
Notice a(1 − a): it peaks at 0.25 and approaches zero when the neuron saturates near 0 or 1. A confidently wrong sigmoid neuron therefore learns very slowly, which is precisely the argument for cross-entropy loss, whose derivative cancels that term.
Why the forward pass must be remembered
The backward pass needs the activations computed on the way forward — a(1 − a) needs a. So every intermediate result is stored during the forward pass and kept until the backward pass consumes it.
That storage is activation memory, and it is what usually limits batch size, not the weights. A ResNet-50's weights are about 100MB; its activations for a batch of 64 images run to several gigabytes.
Two standard responses. Gradient checkpointing stores only some activations and recomputes the rest during the backward pass — trading roughly 30% more compute for a large memory saving. And mixed precision stores activations in 16-bit, halving the memory and speeding up the arithmetic.
This also explains why torch.no_grad() matters at inference: without gradients, nothing needs to be stored, and memory use drops sharply.
Exploration guide
- Start from the loss. The page opens with the forward pass already done: values left to right, ending on a single number. Press Reset and Step if you want to watch that part happen too — but it holds no surprises, it is just the network making a prediction.
- Keep stepping. The gradient starts at the loss as 1, then flows right to left. Each node prints the multiplication it performed, so you can read the chain rule rather than take it on faith.
- Compare the two weight gradients. They differ only by their input: ∂L/∂w₁ carries x₁ and ∂L/∂w₂ carries x₂. Set Input x1 to 0 and its weight's gradient goes to exactly zero — that weight cannot learn from this example at all.
- Check the answer. The gradient-check panel recomputes ∂L/∂w₁ numerically, by nudging w₁ and dividing the change in loss by the change in weight. It agrees with the chain rule to several decimal places, which is the test to reach for whenever a hand-written backward pass looks suspicious.
- Saturate the neuron. Push Weight w1 and Bias b to their maximums so that z is large. The sigmoid flattens, a(1 − a) collapses towards zero, and every gradient behind it shrinks with it. That is the vanishing gradient, visible in one number.
- Close the gap. Drag Target y towards the prediction a. The loss collapses and every gradient in the graph shrinks with it, because the whole chain is scaled by 2(a − y) — when there is nothing left to fix, nothing moves.
Why it is fast
The obvious way to get gradients is to nudge each weight and re-run the network, exactly as the gradient check does. That costs one forward pass per weight; on a model with a million weights it is hopeless.
Backpropagation gets all of them in one backward pass, because the gradient arriving at a node is shared by everything feeding it. Work is reused rather than repeated — the same insight that makes dynamic programming fast. This is reverse-mode automatic differentiation, and it is what every deep learning framework implements underneath loss.backward().
What trips people up
- Forgetting to zero the gradients. Frameworks accumulate into .grad by design, so that a batch can be split across several backward passes. Omit the reset and you are descending on the sum of every batch so far.
- Saturation. Sigmoid and tanh have tiny derivatives at their extremes, and a product of tiny numbers vanishes. This is the reason ReLU and its relatives took over, and the reason weight initialisation is treated so carefully.
- Exploding gradients. The mirror image: repeated multiplication by numbers above one, common in deep or recurrent stacks. Gradient clipping exists for this.
- Detaching by accident. Convert a tensor to a plain number mid-network and the graph is cut there; everything behind it silently gets no gradient at all.
- Trusting the derivation. If you write a backward pass by hand, check it numerically. It takes five lines and it finds sign errors immediately.
Where that leaves you
Backpropagation is the chain rule applied to a computational graph in reverse order: each operation contributes only its own local derivative, and the gradient arriving from downstream is multiplied by it on the way past. An add passes the gradient through untouched, a multiply scales it by the other input, and a saturating activation shrinks it — which is where vanishing gradients come from. Because every node reuses the gradient already computed for the nodes it feeds, one backward pass yields every weight's gradient at roughly the cost of one forward pass, and that efficiency is the reason training deep networks is possible at all. The forward values have to be kept in memory for it to work, and a numeric gradient check is the fastest way to prove your backward pass is right.
What goes wrong, and why
Because the chain rule multiplies one factor per layer, deep networks multiply many numbers together — and that is numerically unstable in both directions.
Vanishing gradients. Sigmoid's derivative peaks at 0.25. Ten layers gives 0.25¹⁰ ≈ 0.000001, so early layers barely move. The fixes all target the size of those factors: ReLU (derivative exactly 1 when active), residual connections (an identity path whose derivative is 1), and normalisation layers (which keep activations where the factors stay near 1).
Exploding gradients. If the factors average above 1, the product grows without bound and weights become NaN. Gradient clipping caps the gradient's norm directly and is standard in recurrent models and transformers.
Dead ReLUs. A unit stuck on the negative side has a derivative of exactly 0, receives no gradient, and never recovers. Leaky ReLU, careful initialisation and a lower learning rate all reduce it.
Diagnosing all three is the same action: log the gradient norm per layer. A healthy network shows similar magnitudes throughout; a vanishing one shows orders of magnitude of decay from output to input.
Automatic differentiation in practice
You will never implement backpropagation by hand. Frameworks record every operation into a graph as the forward pass runs, then walk it backwards applying each operation's known derivative rule.
loss = criterion(model(x), y)
optimizer.zero_grad() # clear the previous step's gradients
loss.backward() # backpropagation: fills .grad on every parameter
optimizer.step() # apply the updateThe zero_grad() line is not optional. PyTorch accumulates gradients into .grad rather than replacing them, so forgetting it means each step uses the sum of all previous gradients — training appears to work and converges to nonsense. (The accumulation behaviour is deliberate: it is how you simulate a large batch on a small GPU, by calling backward() several times before step().)
To verify a hand-written gradient, compare it with a numerical one: (f(x+h) − f(x−h)) / 2h with h around 1e-5. If they disagree, the analytic version is wrong.
Every gradient, by hand, checked numerically
Backpropagation is the chain rule applied backwards through a network. Here is a two-layer net where every partial derivative is computed twice -- analytically and numerically -- so you can see they agree.
Questions people ask
Is backpropagation the same as gradient descent? No. Backpropagation computes the gradients; gradient descent uses them to update the weights. Two separate steps.
Why do I need to call zero_grad()? Because gradients accumulate by default. Omitting it is one of the most common PyTorch bugs.
Does it work through any operation? Any differentiable one. Non-differentiable steps (argmax, sampling, hard thresholds) block the gradient and need workarounds such as straight-through estimators or the reparameterisation trick.
How does it handle branching architectures? Where a value feeds several paths, the gradients from all of them are summed. That is the multivariable chain rule.
Why is training about three times a forward pass? Roughly: one forward, one backward of similar cost, plus the weight update.
Is it biologically plausible? Not really — real neurons have no known mechanism for sending precise error signals backwards. It is an engineering algorithm, not a model of the brain.
Recap in one screen
- Backpropagation computes every weight's gradient in one backward pass using the chain rule.
- It reuses shared factors, which is why training costs about the same as a few forward passes rather than one pass per parameter.
- Activations from the forward pass must be stored, and that memory usually limits batch size.
- Multiplying many factors causes vanishing and exploding gradients; ReLU, residuals, normalisation and clipping are the responses.
- Frameworks do it automatically — but call
zero_grad(), because gradients accumulate.