Backpropagation and the Computational Graph

Quick Context

Gradient descent needs a gradient: for every weight in the network, how much the loss would change if that weight moved a little. Backpropagation is how that number is obtained, for millions of weights, at the cost of roughly one extra forward pass.

It is not a learning rule and not an optimiser. It computes derivatives, and nothing else. What to do with them is the optimiser's business.

Everything is a graph

A network is a long expression, and any expression can be drawn as a graph of small operations: multiply, add, apply a function. The graph above is a single neuron with two inputs, which is already enough to show every mechanic that matters.

m₁ = w₁x₁   m₂ = w₂x₂   s = m₁+m₂   z = s+b   a = σ(z)   L = (a − y)²

The forward pass fills in a value at every node. Crucially, those values are kept — the backward pass needs them, which is exactly why training a network takes far more memory than running one.

Each operation knows one thing

Every node needs only its local derivative: how its output responds to its own inputs. It has no idea what the rest of the network looks like, and does not need to.

• Add passes the gradient through unchanged to both inputs. This is why ∂L/∂b equals ∂L/∂z exactly, every time.
• Multiply hands each input the gradient scaled by the other input. So ∂L/∂w₁ = ∂L/∂m₁ · x₁ — a weight's gradient is proportional to the input it was multiplying, which is why an input of zero produces a gradient of zero and why unscaled features distort learning.
• The sigmoid contributes a(1 − a), which peaks at 0.25 and falls to nearly nothing when the neuron saturates.

The chain rule then says: the gradient arriving at a node, times the node's local derivative, is the gradient leaving it. Backpropagation is that single sentence applied repeatedly, in reverse topological order.

Interactive Exploration Guide

1. Start from the loss. The page opens with the forward pass already done: values left to right, ending on a single number. Press Reset and Step if you want to watch that part happen too — but it holds no surprises, it is just the network making a prediction.Run this
2. Keep stepping. The gradient starts at the loss as 1, then flows right to left. Each node prints the multiplication it performed, so you can read the chain rule rather than take it on faith.
3. Compare the two weight gradients. They differ only by their input: ∂L/∂w₁ carries x₁ and ∂L/∂w₂ carries x₂. Set Input x1 to 0 and its weight's gradient goes to exactly zero — that weight cannot learn from this example at all.Run this
4. Check the answer. The gradient-check panel recomputes ∂L/∂w₁ numerically, by nudging w₁ and dividing the change in loss by the change in weight. It agrees with the chain rule to several decimal places, which is the test to reach for whenever a hand-written backward pass looks suspicious.
5. Saturate the neuron. Push Weight w1 and Bias b to their maximums so that z is large. The sigmoid flattens, a(1 − a) collapses towards zero, and every gradient behind it shrinks with it. That is the vanishing gradient, visible in one number.Run this
6. Close the gap. Drag Target y towards the prediction a. The loss collapses and every gradient in the graph shrinks with it, because the whole chain is scaled by 2(a − y) — when there is nothing left to fix, nothing moves.

Why it is fast

The obvious way to get gradients is to nudge each weight and re-run the network, exactly as the gradient check does. That costs one forward pass per weight; on a model with a million weights it is hopeless.

Backpropagation gets all of them in one backward pass, because the gradient arriving at a node is shared by everything feeding it. Work is reused rather than repeated — the same insight that makes dynamic programming fast. This is reverse-mode automatic differentiation, and it is what every deep learning framework implements underneath loss.backward().

What usually goes wrong

• Forgetting to zero the gradients. Frameworks accumulate into .grad by design, so that a batch can be split across several backward passes. Omit the reset and you are descending on the sum of every batch so far.
• Saturation. Sigmoid and tanh have tiny derivatives at their extremes, and a product of tiny numbers vanishes. This is the reason ReLU and its relatives took over, and the reason weight initialisation is treated so carefully.
• Exploding gradients. The mirror image: repeated multiplication by numbers above one, common in deep or recurrent stacks. Gradient clipping exists for this.
• Detaching by accident. Convert a tensor to a plain number mid-network and the graph is cut there; everything behind it silently gets no gradient at all.
• Trusting the derivation. If you write a backward pass by hand, check it numerically. It takes five lines and it finds sign errors immediately.

Key Takeaway

Backpropagation is the chain rule applied to a computational graph in reverse order: each operation contributes only its own local derivative, and the gradient arriving from downstream is multiplied by it on the way past. An add passes the gradient through untouched, a multiply scales it by the other input, and a saturating activation shrinks it — which is where vanishing gradients come from. Because every node reuses the gradient already computed for the nodes it feeds, one backward pass yields every weight's gradient at roughly the cost of one forward pass, and that efficiency is the reason training deep networks is possible at all. The forward values have to be kept in memory for it to work, and a numeric gradient check is the fastest way to prove your backward pass is right.