Semantic Segmentation and U-Net Skip Connections
Pooling throws resolution away on purpose, to build context. Pixel-level segmentation needs that resolution back. U-Net's skip connections are how it gets it.
Overview
The idea in brief
Semantic segmentation labels every pixel with a class, not just the image as a whole. A CNN classifier's usual move — pool down repeatedly to build up wide receptive fields and semantic context — is exactly what a segmentation decoder then has to undo, pixel by pixel, to produce a full-resolution output mask.
Downsample Depth
a 16×16 mask, majority-vote 2×2 pooling per level, nearest-neighbour upsampling back
Ground Truth vs Reconstructions
—Pixel Accuracy
Semantic Segmentation and U-Net: A Practical Guide
Classification asks what's in the image. Segmentation asks, for every single pixel.
What pooling actually destroys
Max or majority pooling is a real, lossy operation: a 2×2 block of pixels becomes one pixel, and whatever varied within that block is gone for good. Naively upsampling the coarse result back up — nearest-neighbour or bilinear, with nothing else — cannot recover that lost detail, because the information needed to recover it no longer exists anywhere in the coarse map. U-Net's skip connections route the encoder's full-resolution feature maps directly across to the matching decoder stage, so the fine spatial detail pooling destroyed never has to be reconstructed from nothing — it's simply still there, carried across.
Labelling every pixel
Classification gives one label per image. Detection gives a box per object. Semantic segmentation gives a label to every single pixel — road, building, sky, person — producing an output the same size as the input.
Three related tasks are easy to confuse:
| Task | Output | Distinguishes individual objects? |
|---|---|---|
| Semantic segmentation | A class per pixel | No — all cars are "car" |
| Instance segmentation | A mask per object | Yes |
| Panoptic segmentation | Both, combined | Yes, plus background classes |
The architectural difficulty is a genuine tension. Recognising what something is needs a wide view and abstract features, which means downsampling. Knowing exactly which pixels it occupies needs full resolution, which downsampling destroys.
U-Net's answer: go down, come back up, and remember
U-Net is shaped like a U, and the shape is the idea.
The encoder (the left side) is a standard convolutional stack: convolutions and downsampling, halving the resolution and doubling the channels at each stage. By the bottom it has a rich, abstract, low-resolution representation — it knows what is in the image.
The decoder (the right side) upsamples back to full resolution, halving the channels at each stage.
The skip connections are what make it work. At each level, the encoder's feature map is concatenated onto the decoder's, so the decoder has both the semantic information from the deep path and the fine spatial detail from the corresponding shallow one.
Without those connections, the output is correct in category and blurry at the edges, because the boundary information was thrown away during downsampling and there is no way to invent it back. With them, boundaries are sharp.
Note that U-Net concatenates rather than adds, unlike ResNet. Concatenation preserves both sources separately and lets the following convolution decide how to combine them.
Why it works on small datasets
U-Net was designed for biomedical images, where a dataset might contain thirty annotated examples. Several of its properties make that feasible.
- It is fully convolutional, with no dense layers, so it accepts any input size and has relatively few parameters.
- Heavy augmentation — particularly elastic deformation, which is realistic for tissue — multiplies the effective dataset.
- Every pixel is a training example. A single 512×512 image provides 262,144 labelled pixels, which is why so few images can be enough.
- Patch-based training lets large images be processed in tiles, with overlap to avoid edge artefacts.
The encoder is now usually replaced with a pretrained backbone — ResNet, EfficientNet — which improves results further. That variant is what most segmentation libraries give you by default.
Why U-Net has to be a U
A segmentation network needs both a wide view and a precise one, and downsampling gives you the first at the cost of the second. U-Net's answer is the skip connection, and this measures exactly what it recovers -- by building the same decoder with and without it.
Guided tour
- Read all three masks at 1 downsampling level. The no-skip reconstruction is already visibly blockier at the boundary than the ground truth circle.
- Push depth to 3 levels. The no-skip mask's pixel accuracy drops further — each extra pooling level throws away more boundary detail that nearest-neighbour upsampling has no way to invent back.
- Compare against the with-skip mask. It matches the ground truth exactly, at any depth — because this simplified version literally routes the original full-resolution mask across, rather than a pooled-and-corrupted feature map.
What this simplification doesn't show
A real U-Net's skip connections carry learned feature maps across, not the raw ground-truth labels — its decoder still has to learn to fuse coarse semantic features with fine spatial ones, and that fusion is imperfect. A trained U-Net does not hit 100% pixel accuracy just because it has skip connections. What this toy demonstrates honestly is the underlying mechanism: recovering resolution that pooling destroyed by carrying it across directly, rather than trying to hallucinate it back from a coarse map alone — that mechanism is exactly why U-Net needs skip connections at all.
What to remember
Downsampling for context and needing pixel-precise output are in direct tension — the same pooling that builds semantic understanding destroys the spatial detail a segmentation mask needs back. Skip connections resolve that tension not by improving the upsampling step, but by making it partly unnecessary: the fine detail rides across the encoder-decoder bridge instead of being reconstructed from a coarse map that no longer contains it.
Losses for segmentation
Segmentation has a characteristic problem: class imbalance within the image. A tumour might occupy 1% of the pixels, and a model that predicts "background" everywhere scores 99% pixel accuracy.
The standard losses respond to this directly:
| Loss | Behaviour |
|---|---|
| Cross-entropy | Per-pixel classification; dominated by the majority class |
| Weighted cross-entropy | Rare classes weighted up |
| Dice loss | Optimises overlap directly; good for small foregrounds |
| Focal loss | Down-weights easy pixels, focuses on hard ones |
| Dice + cross-entropy | The common practical combination |
Dice loss is derived from the Dice coefficient, 2|A∩B| / (|A|+|B|), and because it is a ratio it is largely insensitive to how much background there is. Combining it with cross-entropy is the most common default in medical segmentation, because the two have complementary gradient behaviour.
For evaluation, mean IoU across classes is the standard metric, with the Dice coefficient (equivalent to F1 over pixels) more common in medical work. Pixel accuracy is reported occasionally and should be treated with the same suspicion as accuracy on any imbalanced problem.
Alternatives and refinements
DeepLab uses dilated convolutions to widen the receptive field without downsampling at all, plus an atrous spatial pyramid that looks at several dilation rates in parallel. Strong on natural scenes.
FCN was the earlier fully-convolutional approach: take a classifier, replace the dense layers with convolutions, and upsample. Simpler and less precise at boundaries.
SegFormer and Mask2Former bring transformers to segmentation, with strong results at higher training cost.
Segment Anything (SAM) is a promptable foundation model that produces high-quality masks with no task-specific training, and is increasingly used to accelerate annotation rather than replace task-specific models.
For most applied projects, a U-Net with a pretrained encoder remains the pragmatic starting point, and segmentation_models_pytorch gives it in a few lines.
Common mistakes
- Resizing masks with bilinear interpolation. This blends label values into meaningless intermediates — class 3.7. Always use nearest-neighbour for masks.
- Augmenting the image and not the mask, or applying different random parameters to each.
- Reporting pixel accuracy on an imbalanced problem.
- Ignoring the ignore label. Many datasets mark unlabelled pixels with a special value that must be excluded from the loss.
- Tiling without overlap, which produces visible seams at tile boundaries.
- Forgetting that the output needs an argmax across the channel dimension to become a label map.
Questions people ask
How many annotated images do I need? Tens can work with heavy augmentation and a pretrained encoder; hundreds for reliable results on natural scenes.
Why concatenate instead of add in the skip connections? Concatenation keeps both feature sets intact and lets the next convolution learn how to weigh them. Addition assumes they are directly comparable.
What input size should I use? Any multiple of the total downsampling factor — typically 32. Mismatched sizes cause shape errors in the decoder.
How do I handle very large images? Tile with overlap and blend the predictions, or downsample if fine detail is not required.
Can U-Net do instance segmentation? Not directly — it does not separate touching objects of the same class. Mask R-CNN or a watershed post-processing step handles that.
Is U-Net outdated? No. It remains the default in medical imaging and is a strong, cheap baseline everywhere else.
Recap in one screen
- Semantic segmentation assigns a class to every pixel; the output is the same size as the input.
- The tension is between semantic depth (needs downsampling) and spatial precision (needs resolution).
- U-Net downsamples, upsamples, and concatenates matching encoder features into the decoder — which is what keeps boundaries sharp.
- It works on tiny datasets because every pixel is a training example and augmentation goes a long way.
- Use Dice plus cross-entropy for imbalanced foregrounds, evaluate with mean IoU, and never interpolate masks bilinearly.