ResNet and Identity Shortcuts

The idea in brief

ResNet's headline idea is the identity shortcut: instead of a block computing x_{i+1} = f(x_i), it computes x_{i+1} = x_i + f(x_i), adding the block's own input back onto its output. The usual explanation is about gradients flowing backward through very deep networks. This module shows the forward-pass version of the same mechanism — what happens to the signal it, going forward, as it passes through many stacked blocks. The shortcut connection means f(x) just needs to learn the *residual* — the gap between the desired output and the input — rather than the full mapping. If the best transformation at some layer is close to identity (which is common in deep networks), learning a near-zero residual is much easier than learning the identity function from scratch. This is why adding layers to a ResNet never hurts: a block that learns f(x) = 0 passes the input through unchanged. So extra depth at worst does nothing. Without shortcuts, deeper networks paradoxically do *worse* than shallower ones on both training and test data — the degradation problem that motivated the design. ResNet-50 uses "bottleneck" blocks (1x1 → 3x3 → 1x1 convolutions) to reduce computation. ResNet-18 and ResNet-34 use simpler two-layer blocks. When the shortcut crosses a dimension change (spatial downsampling or channel increase), a 1x1 convolution with stride 2 projects the input to the new shape. This page stacks blocks with and without shortcuts, shows the signal magnitude at each layer. And covers the degradation problem with a depth experiment.

The degradation problem

Before ResNet, simply stacking more convolutional layers past a certain depth made plain networks harder to train, not just slower — a well-documented empirical finding known as the degradation problem. One real contributor is signal decay. If each block's weights shrink the signal still slightly, a long enough plain stack keeps shrinking it, layer after layer, until by the far end there is barely any signal left carrying information forward. A shortcut connection breaks that. Since relu(f(x)) can never be negative, adding it to x can never make the running signal less than x was. The identity path is a floor the plain stack simply does not have.

The problem ResNet solved

Before 2015 the assumption was that deeper networks would be better. And the evidence said otherwise. A 56-layer plain network scored worse than a 20-layer one — not just on test data. But on training data.

That is the key detail. Worse test accuracy would be overfitting. Worse training accuracy means the deeper network could not still fit what the shallower one could, despite being strictly more expressive. It could in principle copy the 20-layer network and make the left 36 layers do nothing.

The problem was optimisation, not capacity. Gradients degrade as they pass back through many layers. And learning an exact identity mapping through a stack of convolutions and non-linearities turns out to be surprisingly hard.

Learn the gap, not the whole thing

ResNet's answer is one line of arithmetic.

output = F(x) + x

Instead of asking a block to give the desired output H(x) right, ask it to give the residual F(x) = H(x) − x. And add the input back.

Now "do nothing" is trivially on hand: drive the block's weights towards zero and the output is just x. A layer that has nothing useful to add can get out of the way. It is precisely what the plain deep network could not manage.

The gradient consequence is just as key. Differentiating F(x) + x with respect to x gives F'(x) + 1. That +1 is a path along which the gradient flows back unchanged, however many blocks it passes through. The chain rule's product of factors can no longer collapse to nothing, since each block adds a term of just 1.

That single addition made 50, 101 and 152-layer networks trainable. And the same idea now appears in in effect each deep design, transformers included.

What is inside a block

The basic block, used in ResNet-18 and 34.

x → Conv3×3 → BN → ReLU → Conv3×3 → BN → (+x) → ReLU

The bottleneck block, used in ResNet-50 and deeper, sandwiches an costly 3×3 between two cheap 1×1 convolutions that shrink and then restore the channel count.

x → Conv1×1 (reduce) → Conv3×3 → Conv1×1 (restore) → (+x)

The bottleneck exists for cost. Doing a 3×3 convolution on 256 channels is four times the work of reducing to 64, convolving. And expanding back — which is why deep ResNets use it all over.

Two details worth knowing. The addition happens before the final ReLU. So the identity path is not passed through a non-linearity. And when a block changes the channel count or the spatial size, the shortcut cannot be a plain identity. A 1×1 convolution with matching stride is used to project x to the right shape.

Why adding the input back makes depth trainable

ResNet's change is one addition per block. And the reason it works shows up in the gradient. This measures a 40-layer plain stack against the same weights with shortcuts. It finds that the naive shortcut has a problem of its own. And arrives at why a real residual block always has a normalisation inside it.

import numpy as np

rng = np.random.default_rng(12)
D, N = 40, 64

Ws = [rng.normal(0, 1.0 / np.sqrt(N), (N, N)) for _ in range(D)]
x0 = rng.normal(0, 1, N)

def relu(v):
    return np.maximum(v, 0)

def forward(x, alpha, residual):
    # alpha scales the residual BRANCH. residual=False is the plain stack.
    acts = [x]
    for Wl in Ws:
        h = relu(Wl @ x)
        x = (x + alpha * h) if residual else h
        acts.append(x)
    return acts

def grad_norms(acts, alpha, residual):
    # backpropagate a vector of ones and record its size at every depth
    g = np.ones(N)
    out = [np.linalg.norm(g)]
    for d in range(D - 1, -1, -1):
        mask = (Ws[d] @ acts[d] > 0).astype(float)
        step = Ws[d].T @ (mask * g)
        g = (g + alpha * step) if residual else step
        out.append(np.linalg.norm(g))
    return out[::-1]

print("A PLAIN STACK vs A RESIDUAL STACK: %d layers, %d units, the SAME"
      % (D, N))
print("weights in both. the only difference is one addition:")
print("      plain:     x = relu(W @ x)")
print("      residual:  x = x + relu(W @ x)")
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

ap = forward(x0, 1.0, False)
ar = forward(x0, 1.0, True)
print("%-8s %18s %18s" % ("layer", "plain |x|", "residual |x|"))
for d in (0, 10, 20, 30, 40):
    print("%-8d %18.4g %18.4g"
          % (d, np.linalg.norm(ap[d]), np.linalg.norm(ar[d])))
print("   the plain stack decays to %.1e of the input. that is the"
      % (np.linalg.norm(ap[D]) / np.linalg.norm(ap[0])))
print("   vanishing-signal problem, and it is what motivated shortcuts.")
print()
print("   BUT LOOK AT THE OTHER COLUMN BEFORE CELEBRATING. the residual")
print("   stack grew to %.1e times its input. it did not preserve the"
      % (np.linalg.norm(ar[D]) / np.linalg.norm(ar[0])))
print("   signal, it exploded. the naive shortcut trades one failure for")
print("   the opposite one -- every block ADDS to x without ever removing")
print("   anything, so the norm compounds layer after layer.")
print()

The next block covers the left steps.

import numpy as np

print("THE GRADIENT SHOWS THE SAME THING FROM THE OTHER END. one block")
print("differentiates to:")
print("      plain:     d out/d in  =  W^T * relu'")
print("      residual:  d out/d in  =  I + W^T * relu'")
print("   backprop multiplies %d of those together. in the plain case that" % D)
print("   is a product of %d shrinking matrices. in the residual case" % D)
print("   every factor contains I, so nothing can multiply the gradient")
print("   down to zero -- and nothing caps it either:")
gp = grad_norms(ap, 1.0, False)
gr = grad_norms(ar, 1.0, True)
print("%-8s %18s %18s" % ("layer", "plain |grad|", "residual |grad|"))
for d in (40, 30, 20, 10, 0):
    print("%-8d %18.4g %18.4g" % (d, gp[d], gr[d]))
print("   reading down the table is reading BACKWARD through the network.")
print("   the plain gradient reaching layer 0 is %.1e -- layer 0 is" % gp[0])
print("   effectively frozen. the residual one is %.1e, which is not" % gr[0])
print("   frozen but is not usable either.")
print()

The next block continues with the left demonstrations.

import numpy as np

print("SO SCALE THE BRANCH. keep the identity at full strength and put a")
print("factor on what gets added to it:  x = x + alpha * relu(W @ x)")
print("%-10s %18s %20s %18s"
      % ("alpha", "|x| ratio, in->out", "|grad| at layer 0", "verdict"))
for alpha in (0.0, 0.05, 0.1, 0.2, 0.5, 1.0):
    a = forward(x0, alpha, True)
    g = grad_norms(a, alpha, True)
    ratio = np.linalg.norm(a[D]) / np.linalg.norm(a[0])
    if alpha == 0:
        verdict = "pure skip"
    elif 0.2 < ratio < 50 and 0.2 < g[0] / g[D] < 50:
        verdict = "healthy"
    else:
        verdict = "exploding"
    print("%-10.2f %18.4g %20.4g %18s" % (alpha, ratio, g[0], verdict))
print("   at alpha = 0 the network is a pure identity: signal and gradient")
print("   both pass through untouched, and it computes nothing. at")
print("   alpha = 1 both explode. in between is a band where the signal")
print("   survives %d layers AND the gradient reaches layer 0 intact." % D)
print()
print("   that band is not something you tune by hand. a real residual")
print("   block is")
print("      x = x + BN(conv(relu(BN(conv(x)))))")
print("   and the BatchNorm on the branch forces its output to unit")
print("   variance whatever the weights do, which is the same job alpha is")
print("   doing here, done adaptively and per channel. ReZero and")
print("   LayerScale make it literal: an explicit learned alpha per block,")
print("   initialised at 0, so the network starts as an exact identity and")
print("   opens each block only as far as training needs.")
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

print("WHY THE IDENTITY IS THE RIGHT THING TO ADD. suppose the best thing a")
print("block could do is nothing at all. what must each one learn?")
print("%-28s %s" % ("plain block", "W such that relu(Wx) = x"))
print("%-28s %s" % ("residual block", "W such that relu(Wx) = 0"))
print("   the second is trivial -- W = 0 does it, and weight decay is")
print("   already pulling every weight that way. doing nothing is the")
print("   DEFAULT for a residual block, and the weights only encode the")
print("   difference from it. hence the name: the block learns a residual.")
zero = np.zeros((N, N))
xz = rng.normal(0, 1, N)
xr, xp = xz.copy(), xz.copy()
for _ in range(5):
    xr = xr + relu(zero @ xr)
    xp = relu(zero @ xp)
print("   five blocks with W = 0 exactly:")
print("      residual: max |out - in| = %.1e   (a perfect identity)"
      % np.abs(xr - xz).max())
print("      plain:    |out| = %.1e            (the input is gone)"
      % np.linalg.norm(xp))
print()

The next block covers the left steps.

import numpy as np

print("THE PROBLEM THIS ACTUALLY SOLVED. before ResNet, adding layers made")
print("TRAINING error worse -- not test error, training error:")
print("%-22s %16s %18s" % ("network", "train error", "diagnosis"))
for row in (("plain 20-layer", "lower", "fine"),
            ("plain 56-layer", "HIGHER", "not overfitting"),
            ("ResNet 20-layer", "lower", "fine"),
            ("ResNet 56-layer", "lower still", "fine"),
            ("ResNet 152-layer", "lower still", "fine")):
    print("%-22s %16s %18s" % row)
print("   a 56-layer plain net fitting the TRAINING set worse than a")
print("   20-layer one cannot be overfitting. the deeper network can")
print("   express everything the shallower one can -- set the extra 36")
print("   layers to the identity -- so capacity was never the issue.")
print("   gradient descent simply could not FIND those identity layers.")
print("   shortcuts hand them over at initialisation, and depth went from")
print("   22 layers to 152 in one paper.")
print()

print("ONE PRACTICAL DETAIL: the addition needs matching shapes.")
print("%-34s %s" % ("shapes match", "identity -- 0 parameters"))
print("%-34s %s" % ("channel count changes", "1x1 conv on the shortcut"))
print("%-34s %s" % ("stride 2", "1x1 conv, stride 2"))
print("   those projection shortcuts are a small minority of blocks, and")
print("   that matters: if EVERY shortcut carried weights, the gradient")
print("   would have to pass through them too, and the clean identity path")
print("   from the loss back to layer 0 -- the entire point -- would be")
print("   gone.")

Guided tour

1. Read the norms at depth 10. The plain stack's signal has shrunk to a small fraction of where it started. The version with the shortcut is more than where it started, not less.
2. Push depth to 20. The plain stack keeps shrinking — by block 20 there is very little of the first signal left. The shortcut version keeps growing instead.
3. Bring depth back down to 1 or 2. At shallow depth the two barely differ — this problem is namely about what happens over many stacked layers, not about any single block.
4. Read the per-block table. Each row is a real matrix-vector multiply and ReLU, not a smoothed illustration — you can check the arithmetic at any single block.

What this simplification doesn't show

Here the shortcut version's norm keeps growing with depth rather than settling. This fixed, unnormalized toy has no mechanism to stop it. It is just why real ResNets pair each shortcut with batch or layer normalization to keep the growing signal in a well-behaved range. The point this module shows is narrower and still real. An identity shortcut with a ReLU'd residual branch mathematically cannot let the running signal fall below where it started, in a way a plain stack of contractive layers can and does.

In one line

A plain stack of layers with a contractive weight matrix shrinks its forward signal geometrically with depth — the same kind of multiplicative decay that causes vanishing gradients on the way back. An identity shortcut adds a nonnegative quantity back onto the running signal at each block. It makes shrinkage below the starting point mathematically impossible. It is the same one-line change, x + f(x) instead of f(x), showing up on both passes through the network.

Why it works, in more than one sense

Three explanations are offered, and they are complementary rather than competing.

Optimisation. Identity is easy to represent. So adding layers cannot make the achievable function set worse in practice, just in principle.

Gradient flow. The +1 in the derivative gives each block a direct path back to the loss. It is what makes 100-layer training stable.

Ensemble behaviour. A network with n residual blocks holds 2ⁿ possible paths from input to output, since each block can be taken or skipped. Empirically, removing a single block from a trained ResNet barely changes its accuracy — behaviour much more like an ensemble of many shallow networks than like one very deep one.

The loss-landscape view adds a fourth. Visualisations of the loss surface show that skip connections turn a chaotic, spiky landscape into a much smoother one. It is a fairly direct explanation of why optimisation succeeds.

The family, and its descendants

The idea escaped computer vision fully. Each transformer block is x + Attention(LayerNorm(x)) followed by x + FFN(LayerNorm(x)) — the same residual structure. It is what makes 100-layer language nets trainable. U-Net's skip connections carry the same intuition into segmentation.

Real notes

• ResNet-50 pretrained on ImageNet is the sensible default backbone for a new vision project. It is well-supported all over, and fine-tunes reliably.
• The shortcut must match shapes. When channels or stride change, use the 1×1 projection. Frameworks do this automatically. But hand-written blocks often get it wrong.
• Do not put a ReLU on the identity path. The whole point is that the path is clean.
• Batch normalisation and residuals work together. Removing BN from a deep ResNet often breaks training unless the initialisation is adjusted to compensate.
• Depth is not free. ResNet-152 is roughly three times the compute of ResNet-50 for a few points of accuracy. On most problems the less net plus better data wins.

Questions people ask

Why add rather than concatenate? Addition keeps the channel count constant and costs nothing. DenseNet concatenates instead, which reuses features more clearly at the cost of growing width.

Does the shortcut have params? Not when it is a plain identity. It does when a projection is needed to change shape.

Can I add skip connections to any network? Yes, provided the shapes match. And it often helps for anything more than a few layers deep.

Why does the identity make gradients flow? Since differentiating F(x) + x gives F'(x) + 1. And that constant term cannot vanish however many blocks it passes through.

Is ResNet still relevant? Very. It stays one of the most-used backbones in production. And ConvNeXt shows a modernised version still competes with vision transformers.

How deep can I go? ResNet-1000 has been trained as an experiment. Beyond about 150 layers the accuracy returns are small and the cost is not.

Recap in one screen

• Deep plain networks were worse on training data — an optimisation failure, not overfitting.
• A residual block learns F(x) and outputs F(x) + x. So doing nothing is easy to represent.
• The derivative gains a +1, giving gradients a clean path back through any count of blocks.
• Bottleneck blocks use 1×1 convolutions either side of the 3×3 to cut cost.
• The same shortcut idea underpins transformers, U-Net and almost each modern deep design.