Semantic Segmentation and U-Net Skip Connections

The idea in brief

Semantic splitting labels each pixel with a class, not just the pic as a whole. A CNN sorter's usual move. pool down repeatedly to build up wide receptive fields and semantic context. Is just what a splitting decoder then has to undo, pixel by pixel, to give a full-resolution output mask.

U-Net is the design that made this real. It has an encoder (the downward path) that contracts the pic through convolutions and pooling, building up what each region means while shrinking the spatial resolution. It has a decoder (the upward path) that expands the resolution back to the first size with transposed convolutions or upsampling. The key idea is the skip links between matching levels of the encoder and decoder: high-res spatial detail from the encoder is joined with the semantic data from the decoder. So the network can draw precise edges rather than blobby outlines.

The output is a tensor with the same height and width as the input pic and one channel per class. At each pixel the net predicts a chance spread over classes. And the class with the highest chance turns that pixel's label. The loss function is often cross-entropy found pixel by pixel, though class imbalance (roads are common, pedestrians are rare) often needs a weighted variant or dice loss.

Semantic splitting differs from instance splitting. Semantic splitting labels each pixel with a class but cannot distinguish two cars from each other. Both are "car". Instance splitting (Mask R-CNN) assigns a split mask to each single object. Panoptic splitting mixes both: each pixel gets a class label and an instance id. U-Net and its variants (U-Net++, Attention U-Net, nnU-Net) remain the dominant design for medical imaging, satellite imagery and any task where pixel-level score matters more than real-time speed.

What pooling really destroys

Max or bulk pooling is a real, lossy step: a 2×2 block of pixels turns one pixel. And whatever varied within that block is gone for good. Naively upsampling the coarse result back up. Nearest-neighbour or bilinear, with nothing else. Cannot recover that lost detail, since the information needed to recover it no longer exists anywhere in the coarse map. U-Net's skip connections route the encoder's full-resolution feature maps right across to the matching decoder stage. So the fine spatial detail pooling destroyed never has to be reconstructed from nothing. It's simply still there, carried across.

Labelling each pixel

Sorting gives one label per pic. Detection gives a box per object. Semantic splitting gives a label to each single pixel. Road, building, sky, person. Giving an output the same size as the input.

Three tied tasks are easy to confuse:

The architectural trouble is a genuine tension. Recognising what something is needs a wide view and abstract features. It means downsampling. Knowing just which pixels it occupies needs full resolution. It downsampling destroys.

U-Net's answer: go down, come back up, and remember

U-Net is shaped like a U. And the shape is the idea.

The encoder (the left side) is a standard convolutional stack: convolutions and downsampling, halving the resolution and doubling the channels at each stage. By the bottom it has a rich, abstract, low-resolution form. It knows what is in the pic.

The decoder (the right side) upsamples back to full resolution, halving the channels at each stage.

The skip connections are what make it work. At each level, the encoder's feature map is concatenated onto the decoder's. So the decoder has both the semantic data from the deep path and the fine spatial detail from the matching shallow one.

Without those links, the output is correct in group and blurry at the edges, since the edge data was thrown away in downsampling and there is no way to invent it back. With them, edges are sharp.

Note that U-Net concatenates rather than adds, unlike ResNet. Joining keeps both sources apart and lets the next convolution decide how to mix them.

Why it works on small datasets

U-Net was designed for biomedical pics. There a dataset might hold thirty annotated cases. A few of its properties make that feasible.

• It is fully convolutional, with no dense layers. So it accepts any input size and has fairly few params.
• Heavy tweaking — very elastic deformation. It is realistic for tissue — multiplies the good dataset.
• Each pixel is a training case. A single 512×512 pic gives 262,144 labelled pixels. It is why so few pics can be enough.
• Patch-based training lets large pics be handled in tiles, with overlap to avoid edge artefacts.

The encoder is now often replaced with a pretrained backbone. ResNet, EfficientNet — which improves results more. That variant is what most splitting libraries give you by default.

Why U-Net has to be a U

A splitting network needs both a wide view and a precise one. And downsampling gives you the first at the cost of the second. U-Net's answer is the skip connection. And this measures just what it recovers -- by building the same decoder with and without it.

import numpy as np

H = 32
# a 1-D "image": two regions with a boundary that is exactly at 17
EDGE = 21
truth = np.zeros(H)
truth[EDGE:] = 1.0
rng = np.random.default_rng(3)
sig = truth * 60 + 30 + rng.normal(0, 4, H)

print("A 1-D EXAMPLE, %d pixels, so every array fits on a line. the label" % H)
print("is 0 to the left of pixel %d and 1 from %d on:" % (EDGE, EDGE))
print("   input:  " + " ".join("%3.0f" % v for v in sig[:16]))
print("           " + " ".join("%3.0f" % v for v in sig[16:]))
print("   label:  " + "".join(str(int(v)) for v in truth))
print()

def pool(a):
    return a.reshape(-1, 2).mean(axis=1)

def up(a):
    return np.repeat(a, 2)

print("THE ENCODER: pool by 2, four times. each step doubles the receptive")
print("field and halves the resolution:")
enc = [sig]
for i in range(4):
    enc.append(pool(enc[-1]))
print("%-10s %10s %16s %20s" % ("level", "length", "receptive field",
                                "pixels per cell"))
for i, a in enumerate(enc):
    print("%-10d %10d %16d %20d" % (i, len(a), 2 ** i, 2 ** i))
print("   at the bottom, one cell summarises %d input pixels. that is the"
      % (2 ** 4))
print("   CONTEXT the network needs -- to decide whether this pixel is")
print("   road or pavement you have to see more than the pixel.")
print()

Building on the same setup, the next block shows the left cases.

import numpy as np
H = 32

print("BUT LOOK WHAT HAPPENED TO THE BOUNDARY. the true edge is at %d:" % EDGE)
print("%-10s %10s %18s %18s"
      % ("level", "length", "edge cell index", "input pixels it covers"))
for i, a in enumerate(enc):
    d = np.abs(np.diff(a))
    k = int(np.argmax(d)) + 1
    lo, hi = k * (2 ** i), (k + 1) * (2 ** i) - 1
    print("%-10d %10d %18d %18s" % (i, len(a), k, "%d..%d" % (lo, hi)))
print("   at level 4 the boundary is known only to within %d pixels."
      % (2 ** 4))
print("   the fact that it sits at %d rather than anywhere else in that" % EDGE)
print("   16-pixel block has been averaged away. it is not attenuated or")
print("   hidden -- it is gone, and no decoder can invent it back.")
print()

print("A PLAIN DECODER: upsample four times from the bottom, nothing else.")
plain = enc[-1]
for _ in range(4):
    plain = up(plain)
print("   output: " + " ".join("%3.0f" % v for v in plain[:16]))
print("           " + " ".join("%3.0f" % v for v in plain[16:]))
pred_plain = (plain > plain.mean()).astype(int)
print("   thresholded: " + "".join(str(v) for v in pred_plain))
print("   label:       " + "".join(str(int(v)) for v in truth))
err_plain = int((pred_plain != truth).sum())
print("   %d pixel%s wrong, every one of them at the boundary."
      % (err_plain, "" if err_plain == 1 else "s"))
print("   the region labels are right; only the EDGE is wrong, and it is")
print("   wrong by as much as the coarsest level was uncertain.")
print()

The next block covers the left steps.

import numpy as np
H = 32

print("A U-NET DECODER: at every upsampling step, CONCATENATE the encoder")
print("map from the matching level before continuing. here that is a")
print("simple average of the two, which is what a 1-filter conv over the")
print("concatenation would compute:")
u = enc[-1]
for i in range(4):
    u = up(u)
    skip = enc[len(enc) - 2 - i]
    u = (u + skip) / 2.0
    d = np.abs(np.diff(u))
    k = int(np.argmax(d)) + 1
    print("      after skip from level %d: length %2d, sharpest change at %d"
          % (len(enc) - 2 - i, len(u), k))
print("   output: " + " ".join("%3.0f" % v for v in u[:16]))
print("           " + " ".join("%3.0f" % v for v in u[16:]))
pred_u = (u > u.mean()).astype(int)
print("   thresholded: " + "".join(str(v) for v in pred_u))
print("   label:       " + "".join(str(int(v)) for v in truth))
err_u = int((pred_u != truth).sum())
print("   %d pixel%s wrong." % (err_u, "" if err_u == 1 else "s"))
print()

print("%-34s %14s %18s" % ("decoder", "pixels wrong", "boundary found at"))
for name, pred in (("upsample only", pred_plain), ("with skips", pred_u)):
    idx = np.where(np.diff(pred) != 0)[0]
    print("%-34s %14d %18s"
          % (name, int((pred != truth).sum()),
             str(idx[0] + 1) if len(idx) else "none"))
print("%-34s %14d %18d" % ("truth", 0, EDGE))
print()

The next block continues with the left demonstrations.

import numpy as np
H = 32

print("WHY THE SKIP IS THE ONLY THING THAT COULD WORK. the two halves of")
print("the network hold different things, and neither is sufficient:")
print("%-22s %-24s %s" % ("", "encoder, deep", "encoder, shallow"))
for row in (("resolution", "coarse", "full"),
            ("receptive field", "large", "tiny"),
            ("knows WHAT", "yes", "no"),
            ("knows WHERE", "no", "yes")):
    print("%-22s %-24s %s" % row)
print("   a skip connection is how the decoder gets both at once: WHAT")
print("   from below, WHERE from the side. that is the entire idea, and")
print("   the U shape in the diagram is just what the wiring looks like")
print("   when you draw it.")
print()

print("ONE MORE THING THE SKIPS BUY, which is easy to miss: they are also")
print("a gradient path. the encoder's early layers are %d stages away from"
      % (2 * 4))
print("the loss through the U, and 1 step away through a skip. that is the")
print("same argument as a residual connection, and it is why U-Net trains")

Building on the same setup, the next block shows the left cases.

print("on a few hundred images when it was published for biomedical data")
print("where a few hundred is all there is.")
print()
print("AND THE THING TO WATCH: the concatenation needs matching sizes.")
print("%-30s %14s %14s" % ("input length", "after 4 pools", "back up"))
for n in (32, 30, 36, 64):
    k = n
    ok = True
    for _ in range(4):
        if k % 2:
            ok = False
            break
        k //= 2
    print("%-30d %14s %14s"
          % (n, str(k) if ok else "not divisible",
             str(k * 16) if ok else "shapes mismatch"))
print("   an input that is not divisible by 2^depth comes back the wrong")
print("   size and the concatenation fails -- or worse, a library pads it")
print("   silently and every mask is off by a pixel. that is why U-Net")
print("   implementations insist on input sizes that are multiples of 32,")
print("   and why the original paper cropped the encoder features before")
print("   concatenating rather than padding them.")

Guided tour

1. Read all three masks at 1 downsampling level. The no-skip reconstruction is now visibly blockier at the edge than the ground truth circle.
2. Push depth to 3 levels. The no-skip mask's pixel score drops more. Each extra pooling level throws away more edge detail that nearest-neighbour upsampling has no way to invent back.
3. Check against the with-skip mask. It matches the ground truth just, at any depth. Since this simplified version literally routes the first full-resolution mask across, rather than a pooled-and-corrupted feature map.

What this simplification doesn't show

A real U-Net's skip connections carry learned feature maps across, not the raw ground-truth labels. Its decoder still has to learn to fuse coarse semantic features with fine spatial ones. And that fusion is imperfect. A trained U-Net does not hit 100% pixel score just since it has skip connections. What this toy shows honestly is the underlying trick: recovering resolution that pooling destroyed by carrying it across right, rather than trying to hallucinate it back from a coarse map alone. That trick is just why U-Net needs skip connections at all.

What to remember

Downsampling for context and needing pixel-precise output are in direct tension. The same pooling that builds semantic grasp destroys the spatial detail a splitting mask needs back. Skip links resolve that tension not by improving the upsampling step. But by making it partly unneeded: the fine detail rides across the bridge instead of being rebuilt from a coarse map that no longer holds it.

Losses for splitting

Splitting has a trait problem: class imbalance within the pic. A tumour might occupy 1% of the pixels. And a net that predicts "background" all over scores 99% pixel score.

The standard losses respond to this right:

Dice loss is derived from the Dice coefficient, 2|A∩B| / (|A|+|B|). And since it is a ratio it is largely insensitive to how much background there is. Mixing it with cross-entropy is the most common default in medical splitting, since the two have complementary gradient behaviour.

For test, mean IoU across classes is the standard metric, with the Dice coefficient (the same to F1 over pixels) more common in medical work. Pixel score is reported at times and should be treated with the same suspicion as score on any imbalanced problem.

Options and refinements

DeepLab uses dilated convolutions to widen the receptive field without downsampling at all, plus an atrous spatial pyramid that looks at a few dilation rates in parallel. Strong on natural scenes.

FCN was the earlier fully-convolutional way: take a sorter, replace the dense layers with convolutions, and upsample. Simpler and less precise at edges.

SegFormer and Mask2Former bring transformers to splitting, with strong results at more training cost.

Segment Anything (SAM) is a promptable foundation net that gives high-quality masks with no task-set training. And is increasingly used to accelerate annotation rather than replace task-set nets.

For most applied projects, a U-Net with a pretrained encoder stays the pragmatic starting point. And segmentation_models_pytorch gives it in a few lines.

Common mistakes

• Resizing masks with bilinear interpolation. This blends label vals into meaningless intermediates — class 3.7. Always use nearest-neighbour for masks.
• Tweaking the pic and not the mask, or applying other random params to each.
• Reporting pixel score on an imbalanced problem.
• Ignoring the ignore label. Many datasets mark unlabelled pixels with a special value that must be excluded from the loss.
• Tiling without overlap, which gives visible seams at tile edges.
• Forgetting that the output needs an argmax across the channel dimension to turn a label map.

Questions people ask

How many annotated pics do I need? Tens can work with heavy tweaking and a pretrained encoder. Hundreds for reliable results on natural scenes.

Why concatenate instead of add in the skip connections? Concatenation keeps both feature sets intact and lets the next convolution learn how to weigh them. Addition assumes they are right comparable.

What input size should I use? Any multiple of the total downsampling factor — often 32. Mismatched sizes cause shape errors in the decoder.

How do I handle very large pics? Tile with overlap and blend the predictions. Or downsample if fine detail is not needed.

Can U-Net do instance splitting? Not right. It does not split touching objects of the same class. Mask R-CNN or a watershed post-handling step handles that.

Is U-Net outdated? No. It stays the default in medical imaging and is a strong, cheap baseline all over else.

Recap in one screen

• Semantic splitting assigns a class to each pixel. The output is the same size as the input.
• The tension is between semantic depth (needs downsampling) and spatial precision (needs resolution).
• U-Net downsamples, upsamples. And concatenates matching encoder features into the decoder. Which is what keeps edges sharp.
• It works on tiny datasets since each pixel is a training case and tweaking goes a long way.
• Use Dice plus cross-entropy for imbalanced foregrounds, test with mean IoU. And never interpolate masks bilinearly.