ResNet and Identity Shortcuts

The degradation problem

Before ResNet, simply stacking more convolutional layers past a certain depth made plain networks harder to train, not just slower — a well-documented empirical finding known as the degradation problem. One real contributor: if each block's weights shrink the signal even slightly, a long enough plain stack keeps shrinking it, layer after layer, until by the far end there is barely any signal left carrying information forward. A shortcut connection breaks that: because relu(f(x)) can never be negative, adding it to x can never make the running signal smaller than x was — the identity path is a floor the plain stack simply doesn't have.

The problem ResNet solved

Before 2015 the assumption was that deeper networks would be better, and the evidence said otherwise. A 56-layer plain network scored worse than a 20-layer one — not just on test data, but on training data.

That is the crucial detail. Worse test accuracy would be overfitting. Worse training accuracy means the deeper network could not even fit what the shallower one could, despite being strictly more expressive: it could in principle copy the 20-layer network and make the remaining 36 layers do nothing.

The problem was optimisation, not capacity. Gradients degrade as they pass back through many layers, and learning an exact identity mapping through a stack of convolutions and non-linearities turns out to be surprisingly hard.

Learn the difference, not the whole thing

ResNet's answer is one line of arithmetic:

output = F(x) + x

Instead of asking a block to produce the desired output H(x) directly, ask it to produce the residual F(x) = H(x) − x, and add the input back.

Now "do nothing" is trivially available: drive the block's weights towards zero and the output is exactly x. A layer that has nothing useful to add can get out of the way, which is precisely what the plain deep network could not manage.

The gradient consequence is just as important. Differentiating F(x) + x with respect to x gives F'(x) + 1. That +1 is a path along which the gradient flows back unchanged, however many blocks it passes through. The chain rule's product of factors can no longer collapse to nothing, because every block contributes a term of exactly 1.

That single addition made 50, 101 and 152-layer networks trainable, and the same idea now appears in essentially every deep architecture, transformers included.

What is inside a block

The basic block, used in ResNet-18 and 34:

x → Conv3×3 → BN → ReLU → Conv3×3 → BN → (+x) → ReLU

The bottleneck block, used in ResNet-50 and deeper, sandwiches an expensive 3×3 between two cheap 1×1 convolutions that shrink and then restore the channel count:

x → Conv1×1 (reduce) → Conv3×3 → Conv1×1 (restore) → (+x)

The bottleneck exists for cost. Doing a 3×3 convolution on 256 channels is four times the work of reducing to 64, convolving, and expanding back — which is why deep ResNets use it everywhere.

Two details worth knowing. The addition happens before the final ReLU, so the identity path is not passed through a non-linearity. And when a block changes the channel count or the spatial size, the shortcut cannot be a plain identity — a 1×1 convolution with matching stride is used to project x to the right shape.

Why adding the input back makes depth trainable

ResNet's change is one addition per block, and the reason it works shows up in the gradient. This measures a 40-layer plain stack against the same weights with shortcuts, finds that the naive shortcut has a problem of its own, and arrives at why a real residual block always has a normalisation inside it.

import numpy as np

rng = np.random.default_rng(12)
D, N = 40, 64

Ws = [rng.normal(0, 1.0 / np.sqrt(N), (N, N)) for _ in range(D)]
x0 = rng.normal(0, 1, N)

def relu(v):
    return np.maximum(v, 0)

def forward(x, alpha, residual):
    # alpha scales the residual BRANCH. residual=False is the plain stack.
    acts = [x]
    for Wl in Ws:
        h = relu(Wl @ x)
        x = (x + alpha * h) if residual else h
        acts.append(x)
    return acts

def grad_norms(acts, alpha, residual):
    # backpropagate a vector of ones and record its size at every depth
    g = np.ones(N)
    out = [np.linalg.norm(g)]
    for d in range(D - 1, -1, -1):
        mask = (Ws[d] @ acts[d] > 0).astype(float)
        step = Ws[d].T @ (mask * g)
        g = (g + alpha * step) if residual else step
        out.append(np.linalg.norm(g))
    return out[::-1]

print("A PLAIN STACK vs A RESIDUAL STACK: %d layers, %d units, the SAME"
      % (D, N))
print("weights in both. the only difference is one addition:")
print("      plain:     x = relu(W @ x)")
print("      residual:  x = x + relu(W @ x)")
print()

ap = forward(x0, 1.0, False)
ar = forward(x0, 1.0, True)
print("%-8s %18s %18s" % ("layer", "plain |x|", "residual |x|"))
for d in (0, 10, 20, 30, 40):
    print("%-8d %18.4g %18.4g"
          % (d, np.linalg.norm(ap[d]), np.linalg.norm(ar[d])))
print("   the plain stack decays to %.1e of the input. that is the"
      % (np.linalg.norm(ap[D]) / np.linalg.norm(ap[0])))
print("   vanishing-signal problem, and it is what motivated shortcuts.")
print()
print("   BUT LOOK AT THE OTHER COLUMN BEFORE CELEBRATING. the residual")
print("   stack grew to %.1e times its input. it did not preserve the"
      % (np.linalg.norm(ar[D]) / np.linalg.norm(ar[0])))
print("   signal, it exploded. the naive shortcut trades one failure for")
print("   the opposite one -- every block ADDS to x without ever removing")
print("   anything, so the norm compounds layer after layer.")
print()

print("THE GRADIENT SHOWS THE SAME THING FROM THE OTHER END. one block")
print("differentiates to:")
print("      plain:     d out/d in  =  W^T * relu'")
print("      residual:  d out/d in  =  I + W^T * relu'")
print("   backprop multiplies %d of those together. in the plain case that" % D)
print("   is a product of %d shrinking matrices. in the residual case" % D)
print("   every factor contains I, so nothing can multiply the gradient")
print("   down to zero -- and nothing caps it either:")
gp = grad_norms(ap, 1.0, False)
gr = grad_norms(ar, 1.0, True)
print("%-8s %18s %18s" % ("layer", "plain |grad|", "residual |grad|"))
for d in (40, 30, 20, 10, 0):
    print("%-8d %18.4g %18.4g" % (d, gp[d], gr[d]))
print("   reading down the table is reading BACKWARD through the network.")
print("   the plain gradient reaching layer 0 is %.1e -- layer 0 is" % gp[0])
print("   effectively frozen. the residual one is %.1e, which is not" % gr[0])
print("   frozen but is not usable either.")
print()

print("SO SCALE THE BRANCH. keep the identity at full strength and put a")
print("factor on what gets added to it:  x = x + alpha * relu(W @ x)")
print("%-10s %18s %20s %18s"
      % ("alpha", "|x| ratio, in->out", "|grad| at layer 0", "verdict"))
for alpha in (0.0, 0.05, 0.1, 0.2, 0.5, 1.0):
    a = forward(x0, alpha, True)
    g = grad_norms(a, alpha, True)
    ratio = np.linalg.norm(a[D]) / np.linalg.norm(a[0])
    if alpha == 0:
        verdict = "pure skip"
    elif 0.2 < ratio < 50 and 0.2 < g[0] / g[D] < 50:
        verdict = "healthy"
    else:
        verdict = "exploding"
    print("%-10.2f %18.4g %20.4g %18s" % (alpha, ratio, g[0], verdict))
print("   at alpha = 0 the network is a pure identity: signal and gradient")
print("   both pass through untouched, and it computes nothing. at")
print("   alpha = 1 both explode. in between is a band where the signal")
print("   survives %d layers AND the gradient reaches layer 0 intact." % D)
print()
print("   that band is not something you tune by hand. a real residual")
print("   block is")
print("      x = x + BN(conv(relu(BN(conv(x)))))")
print("   and the BatchNorm on the branch forces its output to unit")
print("   variance whatever the weights do, which is the same job alpha is")
print("   doing here, done adaptively and per channel. ReZero and")
print("   LayerScale make it literal: an explicit learned alpha per block,")
print("   initialised at 0, so the network starts as an exact identity and")
print("   opens each block only as far as training needs.")
print()

print("WHY THE IDENTITY IS THE RIGHT THING TO ADD. suppose the best thing a")
print("block could do is nothing at all. what must each one learn?")
print("%-28s %s" % ("plain block", "W such that relu(Wx) = x"))
print("%-28s %s" % ("residual block", "W such that relu(Wx) = 0"))
print("   the second is trivial -- W = 0 does it, and weight decay is")
print("   already pulling every weight that way. doing nothing is the")
print("   DEFAULT for a residual block, and the weights only encode the")
print("   difference from it. hence the name: the block learns a residual.")
zero = np.zeros((N, N))
xz = rng.normal(0, 1, N)
xr, xp = xz.copy(), xz.copy()
for _ in range(5):
    xr = xr + relu(zero @ xr)
    xp = relu(zero @ xp)
print("   five blocks with W = 0 exactly:")
print("      residual: max |out - in| = %.1e   (a perfect identity)"
      % np.abs(xr - xz).max())
print("      plain:    |out| = %.1e            (the input is gone)"
      % np.linalg.norm(xp))
print()

print("THE PROBLEM THIS ACTUALLY SOLVED. before ResNet, adding layers made")
print("TRAINING error worse -- not test error, training error:")
print("%-22s %16s %18s" % ("network", "train error", "diagnosis"))
for row in (("plain 20-layer", "lower", "fine"),
            ("plain 56-layer", "HIGHER", "not overfitting"),
            ("ResNet 20-layer", "lower", "fine"),
            ("ResNet 56-layer", "lower still", "fine"),
            ("ResNet 152-layer", "lower still", "fine")):
    print("%-22s %16s %18s" % row)
print("   a 56-layer plain net fitting the TRAINING set worse than a")
print("   20-layer one cannot be overfitting. the deeper network can")
print("   express everything the shallower one can -- set the extra 36")
print("   layers to the identity -- so capacity was never the issue.")
print("   gradient descent simply could not FIND those identity layers.")
print("   shortcuts hand them over at initialisation, and depth went from")
print("   22 layers to 152 in one paper.")
print()

print("ONE PRACTICAL DETAIL: the addition needs matching shapes.")
print("%-34s %s" % ("shapes match", "identity -- 0 parameters"))
print("%-34s %s" % ("channel count changes", "1x1 conv on the shortcut"))
print("%-34s %s" % ("stride 2", "1x1 conv, stride 2"))
print("   those projection shortcuts are a small minority of blocks, and")
print("   that matters: if EVERY shortcut carried weights, the gradient")
print("   would have to pass through them too, and the clean identity path")
print("   from the loss back to layer 0 -- the entire point -- would be")
print("   gone.")

Guided tour

1. Read the norms at depth 10. The plain stack's signal has shrunk to a small fraction of where it started; the version with the shortcut is larger than where it started, not smaller.
2. Push depth to 20. The plain stack keeps shrinking — by block 20 there is very little of the original signal left. The shortcut version keeps growing instead.
3. Bring depth back down to 1 or 2. At shallow depth the two barely differ — this problem is specifically about what happens over many stacked layers, not about any single block.
4. Read the per-block table. Every row is a real matrix-vector multiply and ReLU, not a smoothed illustration — you can check the arithmetic at any single block.

What this simplification doesn't show

Here the shortcut version's norm keeps growing with depth rather than settling — this fixed, unnormalized toy has no mechanism to stop it, which is exactly why real ResNets pair every shortcut with batch or layer normalization to keep the growing signal in a well-behaved range. The point this module demonstrates is narrower and still real: an identity shortcut with a ReLU'd residual branch mathematically cannot let the running signal fall below where it started, in a way a plain stack of contractive layers can and does.

In one line

A plain stack of layers with a contractive weight matrix shrinks its forward signal geometrically with depth — the same kind of multiplicative decay that causes vanishing gradients on the way back. An identity shortcut adds a nonnegative quantity back onto the running signal at every block, which makes shrinkage below the starting point mathematically impossible. It is the same one-line change, x + f(x) instead of f(x), showing up on both passes through the network.

Why it works, in more than one sense

Three explanations are offered, and they are complementary rather than competing.

Optimisation. Identity is easy to represent, so adding layers cannot make the achievable function set worse in practice, only in principle.

Gradient flow. The +1 in the derivative gives every block a direct path back to the loss, which is what makes 100-layer training stable.

Ensemble behaviour. A network with n residual blocks contains 2ⁿ possible paths from input to output, since each block can be taken or skipped. Empirically, removing a single block from a trained ResNet barely changes its accuracy — behaviour much more like an ensemble of many shallow networks than like one very deep one.

The loss-landscape view adds a fourth: visualisations of the loss surface show that skip connections turn a chaotic, spiky landscape into a much smoother one, which is a fairly direct explanation of why optimisation succeeds.

The family, and its descendants

The idea escaped computer vision entirely. Every transformer block is x + Attention(LayerNorm(x)) followed by x + FFN(LayerNorm(x)) — the same residual structure, which is what makes 100-layer language models trainable. U-Net's skip connections carry the same intuition into segmentation.

Practical notes

• ResNet-50 pretrained on ImageNet is the sensible default backbone for a new vision project. It is well-supported everywhere, and fine-tunes reliably.
• The shortcut must match shapes. When channels or stride change, use the 1×1 projection; frameworks do this automatically, but hand-written blocks frequently get it wrong.
• Do not put a ReLU on the identity path. The whole point is that the path is clean.
• Batch normalisation and residuals work together. Removing BN from a deep ResNet usually breaks training unless the initialisation is adjusted to compensate.
• Depth is not free. ResNet-152 is roughly three times the compute of ResNet-50 for a few points of accuracy; on most problems the smaller model plus better data wins.

Questions people ask

Why add rather than concatenate? Addition keeps the channel count constant and costs nothing. DenseNet concatenates instead, which reuses features more explicitly at the cost of growing width.

Does the shortcut have parameters? Not when it is a plain identity. It does when a projection is needed to change shape.

Can I add skip connections to any network? Yes, provided the shapes match, and it usually helps for anything more than a few layers deep.

Why does the identity make gradients flow? Because differentiating F(x) + x gives F'(x) + 1, and that constant term cannot vanish however many blocks it passes through.

Is ResNet still relevant? Very. It remains one of the most-used backbones in production, and ConvNeXt shows a modernised version still competes with vision transformers.

How deep can I go? ResNet-1000 has been trained as an experiment. Beyond about 150 layers the accuracy returns are small and the cost is not.

Recap in one screen

• Deep plain networks were worse on training data — an optimisation failure, not overfitting.
• A residual block learns F(x) and outputs F(x) + x, so doing nothing is easy to represent.
• The derivative gains a +1, giving gradients a clean path back through any number of blocks.
• Bottleneck blocks use 1×1 convolutions either side of the 3×3 to cut cost.
• The same shortcut idea underpins transformers, U-Net and almost every modern deep architecture.