Residual and Skip Connections

What a skip connection changes

plain:   x_i+1 = f(x_i)   →   dx_i+1/dx_i = f'(x_i)

residual: x_i+1 = x_i + f(x_i)   →   dx_i+1/dx_i = 1 + f'(x_i)

Adding the input back on gives the local derivative a constant +1 term. Whatever f' does — shrink toward zero, even go slightly negative — the identity path contributes exactly 1 at every layer, so the product across the whole stack can no longer collapse toward zero the way a chain of pure f' terms does. The gradient always has at least the unimpeded identity route back to the input.

Add the input back

A residual connection adds a layer's input to its output:

output = F(x) + x

F is whatever the block computes — typically two convolutions with normalisation and activation, or an attention sublayer in a transformer. The + x is the shortcut, and it costs nothing: no parameters, one addition.

That one addition is arguably the most important architectural idea in modern deep learning. Before it, networks beyond about 20 layers trained badly. After it, 152 layers became routine and 100-layer transformers followed.

The two reasons it works

Identity becomes easy to represent. Ask a block to produce the output directly and it must learn a mapping. Ask it to produce the difference and it can output nothing — drive the weights towards zero — and the block passes its input through unchanged.

That matters because deep plain networks were failing on training data, not just test data. A 56-layer network scored worse than a 20-layer one at fitting the training set, despite being able in principle to copy the shallower network and make the extra layers do nothing. It could not find that solution. Residual connections make it the default starting point.

Gradients get a clean path. Differentiating F(x) + x with respect to x gives F'(x) + 1. That constant term means the gradient reaching earlier layers can never be crushed to nothing by the product of many small factors — there is always a route back with a factor of exactly 1.

Where the addition goes

Two details matter in implementation.

The addition happens before the final activation in the original ResNet block:

x → Conv → BN → ReLU → Conv → BN → (+x) → ReLU

Pre-activation ordering trains deeper. Moving normalisation and activation to the start of the block — BN → ReLU → Conv → BN → ReLU → Conv → (+x) — leaves the identity path with nothing on it at all, and was shown to train 1,000-layer networks. Transformers use this arrangement: x + sublayer(norm(x)).

Shapes must match. F(x) + x requires both terms to have the same shape. When a block changes the channel count or downsamples, the shortcut needs a 1×1 convolution with matching stride to project x. Frameworks handle it; hand-written blocks frequently forget, and the error is immediate.

class ResidualBlock(nn.Module):
    def __init__(self, cin, cout, stride=1):
        super().__init__()
        self.body = nn.Sequential(
            nn.Conv2d(cin, cout, 3, stride, 1, bias=False), nn.BatchNorm2d(cout),
            nn.ReLU(inplace=True),
            nn.Conv2d(cout, cout, 3, 1, 1, bias=False), nn.BatchNorm2d(cout),
        )
        self.short = nn.Identity() if (stride == 1 and cin == cout) else nn.Sequential(
            nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))

    def forward(self, x):
        return torch.relu(self.body(x) + self.short(x))

Exploration guide

1. Read the plain stack at 10 layers. The gradient at the input is 0.55¹⁰ ≈ 0.0025 — over 99% of it is gone before it reaches the first layer.
2. Turn on skip connections. The same ten layers, the same 0.55 local derivative, and the input gradient jumps to 1.55¹⁰ ≈ 80 — nowhere near zero.
3. Push the layer count to 16. Without skips, the bars nearest the input are visually gone — the y-axis is logarithmic and they are still off the bottom. With skips, every bar stays a real, usable number.
4. Note what does not change. The per-layer factor is identical in both cases — 0.55, the same squashing. Nothing about the layers themselves improved; only the path the gradient can take did.

Where that leaves you

A residual connection adds the block's input back onto its output, which adds a constant 1 to that layer's local derivative during backpropagation. A chain of derivatives all below 1 shrinks geometrically and vanishes; a chain that includes a guaranteed +1 at every step cannot collapse toward zero regardless of how small the learned part's derivative is. This is why residual connections, not just wider layers or better optimisers, were the change that made networks with over a hundred layers trainable at all.

Add or concatenate?

Two ways to reuse an earlier layer's output, with different trade-offs.

Addition (ResNet) keeps the channel count constant and costs nothing. The two signals are summed, so they must be comparable in scale.

Concatenation (DenseNet, U-Net) stacks them along the channel axis, preserving both separately and letting the next layer decide how to weigh them. It grows the width, so the following convolution is more expensive.

U-Net's choice is instructive. Its skip connections carry high-resolution boundary detail from the encoder to the decoder, where the deep path carries semantics. Concatenating keeps those two very different signals distinct rather than summing them into one, and the following convolution learns how to combine them.

Everywhere, in one form or another

The idea escaped computer vision within a year:

• Transformers. Every block is x + attention(norm(x)) then x + ffn(norm(x)). Without these, deep transformers do not train.
• U-Net. Encoder-to-decoder skips are what keep segmentation boundaries sharp.
• LSTM and GRU. The additive cell-state path is the recurrent equivalent, and it is why they handle far longer sequences than a plain RNN.
• Diffusion models. Built on residual U-Nets.
• DenseNet. Every layer receives the concatenated outputs of all previous ones.
• Highway networks preceded ResNet with a gated version, where a learned gate decided how much to pass through. ResNet showed the ungated version works better and is simpler.

If you take one design rule from this topic: any network deeper than about ten layers should have residual connections.

The ensemble view

A network with n residual blocks contains 2ⁿ possible paths from input to output, since each block can be traversed or effectively skipped.

The empirical evidence for this reading is striking: removing a single block from a trained ResNet barely changes its accuracy. Removing a layer from a plain deep network destroys it. A residual network behaves much more like an ensemble of many shallow networks than like one very deep one.

That also explains why residual networks are unusually robust to layer dropping and to architectural surgery, and why stochastic depth — randomly skipping whole blocks during training — works as a regulariser.

Add the input back, and depth stops hurting

A plain 40-layer network trains worse than a 10-layer one. The same network with skip connections does not, and the reason is visible in one line of the backward pass.

import numpy as np

rng = np.random.default_rng(0)
WIDTH, N = 32, 256

def build(depth, seed=0):
    r = np.random.default_rng(seed)
    return [r.normal(0, np.sqrt(2 / WIDTH) * 0.75, (WIDTH, WIDTH))
            for _ in range(depth)]

def measure(depth, mode, seed=0):
    # mode: "plain", "residual", or "residual+norm"
    Ws = build(depth, seed)
    x = rng.normal(size=(N, WIDTH))
    zs, invs, a = [], [], x
    for W in Ws:
        z = a @ W
        zs.append(z)
        a = np.maximum(0, z) + (a if mode != "plain" else 0.0)
        if mode == "residual+norm":
            inv = 1.0 / (a.std(axis=1, keepdims=True) + 1e-5)
            a = (a - a.mean(axis=1, keepdims=True)) * inv
            invs.append(inv)
        else:
            invs.append(None)
    out_scale = np.abs(a).mean()
    g = rng.normal(size=a.shape) / np.sqrt(N)
    for i in range(depth - 1, -1, -1):
        if mode == "residual+norm":
            g = g * invs[i]            # the norm's scaling, on the way back
        gz = g * (zs[i] > 0)
        g = gz @ Ws[i].T + (g if mode != "plain" else 0.0)
    return out_scale, np.abs(g).mean()

print("weights deliberately set a little below He scale (0.75x), so each")
print("layer shrinks the signal slightly. that is the normal situation --")
print("nothing is ever exactly at the knife edge.")
print()
print("%7s | %11s %11s | %11s %11s | %11s %11s"
      % ("depth", "plain out", "plain grad", "res out", "res grad",
         "res+norm out", "r+n grad"))
for depth in (5, 10, 20, 40, 80):
    p_o, p_g = measure(depth, "plain")
    r_o, r_g = measure(depth, "residual")
    n_o, n_g = measure(depth, "residual+norm")
    print("%7d | %11.2e %11.2e | %11.2e %11.2e | %11.2e %11.2e"
          % (depth, p_o, p_g, r_o, r_g, n_o, n_g))
print()
print("read the three pairs. the PLAIN stack loses a constant factor per")
print("layer, so it is exponential in depth: by 80 layers the first layer's")
print("gradient is indistinguishable from zero and those weights never move.")
print()
print("the RAW residual stack has the opposite problem. every block adds a")
print("non-negative relu output to the running sum, so the signal grows")
print("without bound -- and the gradient with it. the skip path solved the")
print("vanishing problem by creating an exploding one.")
print()
print("the third pair is what a real residual block actually does: a")
print("normalisation inside the block bounds the stream, while the skip path")
print("keeps the gradient's multiplier at 1. the output column is flat at")
print("~0.81 for every depth from 5 to 80.")
print()
print("be precise about the gradient though: it still grows, from 0.08 to")
print("about 50 across a 16x increase in depth. that is polynomial growth,")
print("not exponential, and a growing gradient is a problem you solve with")
print("one line of clipping. a gradient of 4e-13 is not a problem you solve")
print("at all. that is the whole difference, and it is why the combination")
print("of skip plus norm is the architecture -- neither piece works alone.")
print()

print("here is that path, in the one line that matters:")
print("    plain    :  g = (g * relu') @ W.T")
print("    residual :  g = (g * relu') @ W.T  +  g")
print()
print("the '+ g' term is not scaled by anything. so however small the first")
print("term becomes, the gradient reaching the layer below is at least the")
print("gradient that arrived. formally, for y = x + F(x):")
print("    dy/dx = 1 + dF/dx")
print("and a product of terms that are each '1 plus something small' does")
print("not collapse the way a product of small terms does:")
for f in (0.7, 0.9):
    print("  20 layers at factor %.1f      : %.3e" % (f, f ** 20))
    print("  20 layers at (1 + %.1f x 0.1): %.3e" % (f, (1 + f * 0.1) ** 20))
print()

print("what a residual block actually learns. if the useful function is close")
print("to the identity, F only has to learn the DIFFERENCE:")
target = np.array([1.0, 2.0, 3.0, 4.0])
nearly = target * 1.02 + 0.01
print("  input                      :", np.round(target, 4))
print("  the layer should output     :", np.round(nearly, 4))
print("  a plain layer must learn    :", np.round(nearly, 4), "from scratch")
print("  a residual layer learns F = :", np.round(nearly - target, 4))
print("  the second is a much smaller thing to represent, and F = 0 -- which")
print("  is what small random weights already nearly are -- is a working")
print("  block. that is why adding depth to a residual network cannot make")
print("  it worse: the extra layers can learn to do nothing.")
print()

print("the shape rule that trips people up: x and F(x) must have the same")
print("shape to be added. when a block changes width or downsamples, the")
print("skip path needs a projection:")
print("  same shape     : out = x + F(x)")
print("  changing shape : out = W_s @ x + F(x)   (a 1x1 conv, or a linear)")
print("  that projection is the only place a 'skip connection' has")
print("  parameters, and it is why ResNet diagrams show dotted arrows.")

Questions people ask

Do residual connections have parameters? Not when the shapes match. A projection shortcut has a 1×1 convolution's worth.

Can I add them to any network? Yes, wherever the shapes match, and it usually helps beyond a few layers deep.

Why not just use fewer layers? Depth buys representational efficiency — composed features rather than enumerated ones. Residuals are what make depth usable.

Do they prevent overfitting? No, they address an optimisation problem. Regularisation is a separate concern.

What if I scale the shortcut? Multiplying it by a constant less than 1 reintroduces decay through depth. Keep it at exactly 1.

Is initialising the last normalisation to zero a real technique? Yes — setting γ = 0 in the final normalisation of each block makes the block start as an exact identity, which stabilises very deep and very large models.

Recap in one screen

• output = F(x) + x: the block learns the difference, so doing nothing is trivially available.
• The derivative gains a +1, giving gradients a route back that cannot vanish.
• Shapes must match; use a 1×1 projection when channels or stride change.
• Addition keeps width constant; concatenation keeps both signals separate and grows it.
• Present in transformers, U-Net, LSTMs and diffusion models — anything deep uses some version of it.