Residual and Skip Connections

Quick Context

Stack enough layers and, by the chain rule, the gradient reaching an early layer is the product of every local derivative between it and the loss. If each of those derivatives is reliably less than 1 — true of sigmoid and tanh almost everywhere — the product shrinks geometrically. Ten layers at a local derivative of 0.55 already leaves the input gradient at about 0.0025 of what it started as; twenty leaves it at six millionths. That is the vanishing gradient, and it is why very deep plain networks were nearly untrainable before residual connections.

What a skip connection changes

plain:   x_i+1 = f(x_i)   →   dx_i+1/dx_i = f'(x_i)

residual: x_i+1 = x_i + f(x_i)   →   dx_i+1/dx_i = 1 + f'(x_i)

Adding the input back on gives the local derivative a constant +1 term. Whatever f' does — shrink toward zero, even go slightly negative — the identity path contributes exactly 1 at every layer, so the product across the whole stack can no longer collapse toward zero the way a chain of pure f' terms does. The gradient always has at least the unimpeded identity route back to the input.

Interactive Exploration Guide

1. Read the plain stack at 10 layers. The gradient at the input is 0.55¹⁰ ≈ 0.0025 — over 99% of it is gone before it reaches the first layer.
2. Turn on skip connections. The same ten layers, the same 0.55 local derivative, and the input gradient jumps to 1.55¹⁰ ≈ 80 — nowhere near zero.Run this
3. Push the layer count to 16. Without skips, the bars nearest the input are visually gone — the y-axis is logarithmic and they are still off the bottom. With skips, every bar stays a real, usable number.
4. Note what does not change. The per-layer factor is identical in both cases — 0.55, the same squashing. Nothing about the layers themselves improved; only the path the gradient can take did.

Key Takeaway

A residual connection adds the block's input back onto its output, which adds a constant 1 to that layer's local derivative during backpropagation. A chain of derivatives all below 1 shrinks geometrically and vanishes; a chain that includes a guaranteed +1 at every step cannot collapse toward zero regardless of how small the learned part's derivative is. This is why residual connections, not just wider layers or better optimisers, were the change that made networks with over a hundred layers trainable at all.