Residual and Skip Connections

What this is

Stack enough layers and, by the chain rule, the gradient reaching an early layer is the product of each local derivative between it and the loss. If each of those derivatives is reliably less than 1 — true of sigmoid and tanh almost all over — the product shrinks geometrically. Ten layers at a local derivative of 0.55 now leaves the input gradient at about 0.0025 of what it started as. Twenty leaves it at six millionths. That is the vanishing gradient. And it is why very deep plain networks were nearly untrainable before residual connections. A residual block adds the input back to the output — y = F(x) + x — and that single change transforms the gradient dynamics: the local derivative turns 1 + F'(x) instead of F'(x). So the product across many layers always has at least the identity path adding 1 at each stage. The gradient can no longer collapse to zero through multiplication alone. ResNet showed the consequence: networks of 50, 101. And still 152 layers trained stably and each extra block improved accuracy. There plain networks of the same depth diverged or stalled. The same idea reappears in transformers (where each attention and feed-forward sublayer is wrapped in a residual), in DenseNet (which concatenates rather than adds), and in highway networks. This page shows the gradient flow through a plain and a residual stack side by side, lets you add layers and watch the gradient magnitude at the input, and covers the dimension-matching projection needed when the block changes the tensor shape.

What a skip connection changes

plain:   x_i+1 = f(x_i)   →   dx_i+1/dx_i = f'(x_i)

residual: x_i+1 = x_i + f(x_i)   →   dx_i+1/dx_i = 1 + f'(x_i)

Adding the input back on gives the local derivative a constant +1 term. Whatever f' does — shrink toward zero, still go slightly negative — the identity path adds just 1 at each layer. So the product across the whole stack can no longer collapse toward zero the way a chain of pure f' terms does. The gradient always has at least the unimpeded identity route back to the input.

Add the input back

A residual connection adds a layer's input to its output:

output = F(x) + x

F is whatever the block computes — often two convolutions with normalisation and activation. Or an attention sublayer in a transformer. The + x is the shortcut. And it costs nothing: no params, one addition.

That one addition is arguably the most key architectural idea in modern deep learning. Before it, networks beyond about 20 layers trained badly. After it, 152 layers became routine and 100-layer transformers followed.

The two reasons it works

Identity turns easy to represent. Ask a block to give the output right and it must learn a mapping. Ask it to give the gap and it can output nothing — drive the weights towards zero — and the block passes its input through unchanged.

That matters since deep plain networks were failing on training data, not just test data. A 56-layer network scored worse than a 20-layer one at fitting the training set, despite being able in principle to copy the shallower network and make the extra layers do nothing. It could not find that solution. Residual connections make it the default starting point.

Gradients get a clean path. Differentiating F(x) + x with respect to x gives F'(x) + 1. That constant term means the gradient reaching earlier layers can never be crushed to nothing by the product of many small factors — there is always a route back with a factor of just 1.

Where the addition goes

Two details matter in build.

The addition happens before the final activation in the first ResNet block:

x → Conv → BN → ReLU → Conv → BN → (+x) → ReLU

Pre-activation ordering trains deeper. Moving normalisation and activation to the start of the block — BN → ReLU → Conv → BN → ReLU → Conv → (+x) — leaves the identity path with nothing on it at all. And was shown to train 1,000-layer networks. Transformers use this arrangement: x + sublayer(norm(x)).

Shapes must match. F(x) + x needs both terms to have the same shape. When a block changes the channel count or downsamples, the shortcut needs a 1×1 convolution with matching stride to project x. Frameworks handle it. Hand-written blocks often forget, and the error is immediate.

class ResidualBlock(nn.Module):
    def __init__(self, cin, cout, stride=1):
        super().__init__()
        self.body = nn.Sequential(
            nn.Conv2d(cin, cout, 3, stride, 1, bias=False), nn.BatchNorm2d(cout),
            nn.ReLU(inplace=True),
            nn.Conv2d(cout, cout, 3, 1, 1, bias=False), nn.BatchNorm2d(cout),
        )
        self.short = nn.Identity() if (stride == 1 and cin == cout) else nn.Sequential(
            nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))

    def forward(self, x):
        return torch.relu(self.body(x) + self.short(x))

Exploration guide

1. Read the plain stack at 10 layers. The gradient at the input is 0.55¹⁰ ≈ 0.0025 — over 99% of it is gone before it reaches the first layer.
2. Turn on skip connections. The same ten layers, the same 0.55 local derivative. And the input gradient jumps to 1.55¹⁰ ≈ 80 — nowhere near zero.
3. Push the layer count to 16. Without skips, the bars nearest the input are visually gone — the y-axis is logarithmic and they are still off the bottom. With skips, each bar stays a real, usable count.
4. Note what does not change. The per-layer factor is identical in both cases — 0.55, the same squashing. Nothing about the layers them improved. Just the path the gradient can take did.

Where that leaves you

A residual connection adds the block's input back onto its output. It adds a constant 1 to that layer's local derivative in backpropagation. A chain of derivatives all below 1 shrinks geometrically and vanishes. A chain that includes a guaranteed +1 at each step cannot collapse toward zero no matter how small the learned part's derivative is. This is why residual connections, not just wider layers or better optimisers, were the change that made networks with over a hundred layers trainable at all.

Add or concatenate?

Two ways to reuse an earlier layer's output, with other trade-offs.

Addition (ResNet) keeps the channel count constant and costs nothing. The two signals are summed, so they must be comparable in scale.

Concatenation (DenseNet, U-Net) stacks them along the channel axis, preserving both apart and letting the next layer decide how to weigh them. It grows the width, so the next convolution is more costly.

U-Net's choice is instructive. Its skip connections carry high-resolution edge detail from the encoder to the decoder. There the deep path carries semantics. Concatenating keeps those two very other signals distinct rather than summing them into one. And the next convolution learns how to mix them.

All over, in one form or another

The idea escaped computer vision within a year:

• Transformers. Each block is x + attention(norm(x)) then x + ffn(norm(x)). Without these, deep transformers do not train.
• U-Net. Encoder-to-decoder skips are what keep segmentation edges sharp.
• LSTM and GRU. The additive cell-state path is the recurrent equivalent. And it is why they handle far longer sequences than a plain RNN.
• Diffusion nets. Built on residual U-Nets.
• DenseNet. Each layer gets the concatenated outputs of all last ones.
• Highway networks preceded ResNet with a gated version. There a learned gate decided how much to pass through. ResNet showed the ungated version works better and is simpler.

If you take one design rule from this topic: any network deeper than about ten layers should have residual connections.

The ensemble view

A network with n residual blocks holds 2ⁿ possible paths from input to output, since each block can be traversed or effectively skipped.

The empirical evidence for this reading is striking: removing a single block from a trained ResNet barely changes its accuracy. Removing a layer from a plain deep network destroys it. A residual network behaves much more like an ensemble of many shallow networks than like one very deep one.

That also explains why residual networks are unusually reliable to layer dropping and to architectural surgery. And why stochastic depth — randomly skipping whole blocks in training — works as a regulariser.

Add the input back, and depth stops hurting

A plain 40-layer network trains worse than a 10-layer one. The same network with skip connections does not. And the reason is visible in one line of the backward pass.

import numpy as np

rng = np.random.default_rng(0)
WIDTH, N = 32, 256

def build(depth, seed=0):
    r = np.random.default_rng(seed)
    return [r.normal(0, np.sqrt(2 / WIDTH) * 0.75, (WIDTH, WIDTH)) for _ in range(depth)]

def measure(depth, mode, seed=0):
    Ws = build(depth, seed)
    x = rng.normal(size=(N, WIDTH))
    zs, invs, a = [], [], x
    for W in Ws:
        z = a @ W; zs.append(z)
        a = np.maximum(0, z) + (a if mode != "plain" else 0.0)
        if mode == "residual+norm":
            inv = 1.0 / (a.std(axis=1, keepdims=True) + 1e-5)
            a = (a - a.mean(axis=1, keepdims=True)) * inv
            invs.append(inv)
        else:
            invs.append(None)
    out_scale = np.abs(a).mean()
    g = rng.normal(size=a.shape) / np.sqrt(N)
    for i in range(depth - 1, -1, -1):
        if mode == "residual+norm":
            g = g * invs[i]
        gz = g * (zs[i] > 0)
        g = gz @ Ws[i].T + (g if mode != "plain" else 0.0)
    return out_scale, np.abs(g).mean()

print("weights set below He scale (0.75x) so each layer shrinks the signal.\n")
fmt = "%7d | %11.2e %11.2e | %11.2e %11.2e | %11.2e %11.2e"
print("%7s | %11s %11s | %11s %11s | %11s %11s" % (
    "depth", "plain out", "plain grad", "res out", "res grad", "res+norm out", "r+n grad"))
for depth in (5, 10, 20, 40, 80):
    p_o, p_g = measure(depth, "plain")
    r_o, r_g = measure(depth, "residual")
    n_o, n_g = measure(depth, "residual+norm")
    print(fmt % (depth, p_o, p_g, r_o, r_g, n_o, n_g))

The table shows the three regimes: plain gradients vanish, raw residual gradients explode. And residual-with-norm stays flat.

print("the plain stack loses a constant factor per layer -- exponential in")
print("depth. by 80 layers the gradient is indistinguishable from zero.\n")
print("the raw residual stack has the opposite problem: relu outputs are")
print("non-negative, so the running sum grows without bound.\n")
print("residual + norm is what real blocks do: normalisation bounds the")
print("stream while the skip path keeps the gradient multiplier at 1.\n")
print("here is that path, in the one line that matters:")
print("    plain    :  g = (g * relu') @ W.T")
print("    residual :  g = (g * relu') @ W.T  +  g\n")
print("the '+ g' is not scaled by anything. for y = x + F(x):")
print("    dy/dx = 1 + dF/dx\n")
print("a product of '1 + something small' does not collapse the way a")
print("product of small terms does:")
for f in (0.7, 0.9):
    print("  20 layers at factor %.1f      : %.3e" % (f, f ** 20))
    print("  20 layers at (1 + %.1f x 0.1): %.3e" % (f, (1 + f * 0.1) ** 20))

The next block shows what a residual block really learns and why shapes must match for the addition.

import numpy as np

print("what a residual block learns: if the target is close to the identity,")
print("F only has to learn the DIFFERENCE:\n")
target = np.array([1.0, 2.0, 3.0, 4.0])
nearly = target * 1.02 + 0.01
print("  input                      :", np.round(target, 4))
print("  the layer should output     :", np.round(nearly, 4))
print("  a plain layer must learn    :", np.round(nearly, 4), "from scratch")
print("  a residual layer learns F = :", np.round(nearly - target, 4))
print("  F = 0 -- small random weights -- is already a working block.")
print("  adding depth cannot make it worse: extra layers learn to do nothing.\n")
print("the shape rule: x and F(x) must have the same shape to be added.")
print("when a block changes width or downsamples, the skip path needs a")
print("projection:")
print("  same shape     : out = x + F(x)")
print("  changing shape : out = W_s @ x + F(x)   (a 1x1 conv, or a linear)")
print("  that projection is the only place a 'skip connection' has")
print("  parameters, and it is why ResNet diagrams show dotted arrows.")

Questions people ask

Do residual connections have params? Not when the shapes match. A projection shortcut has a 1×1 convolution's worth.

Can I add them to any network? Yes, wherever the shapes match. And it often helps beyond a few layers deep.

Why not just use fewer layers? Depth buys representational efficiency — composed features rather than enumerated ones. Residuals are what make depth usable.

Do they stop overfitting? No, they address an optimisation problem. Regularisation is a split concern.

What if I scale the shortcut? Multiplying it by a constant less than 1 reintroduces decay through depth. Keep it at just 1.

Is initialising the last normalisation to zero a real technique? Yes — setting γ = 0 in the final normalisation of each block makes the block start as an exact identity. It stabilises very deep and very large nets.

Recap in one screen

• output = F(x) + x: the block learns the gap. So doing nothing is trivially on hand.
• The derivative gains a +1, giving gradients a route back that cannot vanish.
• Shapes must match. Use a 1×1 projection when channels or stride change.
• Addition keeps width constant. Concatenation keeps both signals split and grows it.
• Present in transformers, U-Net, LSTMs and diffusion nets — anything deep uses some version of it.