Segmentation is classification with an address
A classifier answers one question per image. A segmentation network answers one per pixel: for a 512×512 input it must give 262,144 labelled outputs, each of which needs both *semantic* info (this is cell wall, not background) and *spatial* info (this exact pixel, not the one next to it).
Those two needs fight. Semantics needs a large receptive field, which means pooling, which destroys spatial precision. Precision needs full detail, which means no pooling, which starves the receptive field. each segmentation architecture is a way of having both, and U-Net's is the most direct: get the semantics by pooling all the way down, then get the precision back by wiring the pre-pooling activations forward.
The contracting path
Each level is two 3×3 convolutions with ReLU, then a 2×2 max pool with stride 2. Channels double at each level: 64, 128, 256, 512, and 1024 at the bottom.
Follow the level table in the explorer. At the paper's settings, a 572×572 tile turns 568×568 after two unpadded convolutions, then 284×284 after pooling, then 280, then 140, and so on down to 28×28 with 1024 channels. Each pooling step quadruples the receptive field of all after it, which is how a network of 3×3 kernels ends up seeing a large enough zone to know what it's looking at.
The widest layer is at the bottom. Read its param count in the table: at base 64 it holds around 6 M of the 31 M total, in one place, at the smallest spatial size. This is the same pattern as ResNet's fourth stage — deep and narrow spatially, so wide channels are affordable in find though they're costly in memory.
The expanding path, and the crop
Each decoder level does a 2×2 up-convolution — a transposed convolution that doubles the spatial size and halves the channels — then concatenates the matching encoder activation, then two more 3×3 convolutions.
The concatenation is the entire point of the architecture. The upsampled tensor carries 512 channels of semantic summary found from a 68×68 view. The skip carries 512 channels of detail found at 136×136, before that view was thrown away. Concatenating gives these convolutions both, and lets them learn how to combine them.
Turn Skip links off in the explorer. The param count drops — the decoder convolutions now read half as many input channels — and that is the entire gain. What you lose is stated in the note that appears: the finest spatial detail available to the decoder is now what survived four rounds of pooling, a 28×28 grid for a 572×572 input. Boundaries come back rounded and blobby, and no amount of decoder room fixes it, because the info is *gone*, not hidden.
Valid convolutions, and why the output is smaller
Switch the padding control between valid and same and watch the output size change.
The first uses unpadded ("valid") convolutions, so each 3×3 eats one pixel from each border. Over the whole network that adds up: 572×572 in, 388×388 out. The skips so don't line up with the upsampled tensors either, and each has to be centre-cropped. By 8, 32, 80 and 176 pixels at the four levels, as the table shows.
That looks like an annoyance and is really a deliberate ensure. each output pixel is found from a zone that was fully present in the input. There are no border pixels whose context was invented by zero-padding. For a network run on tiles of a much larger microscopy image, that matters: the paper's "share-tile" strategy feeds shared tiles and keeps only the valid central output of each. So A large image is segmented smoothly with no edge artefacts at the tile joins.
Almost each setup since uses padding=1 instead, so input and output are the same size and no cropping is needed. It's simpler and often fine. The cost is that the outermost pixels are predicted from partly-fabricated context, which shows up as a thin unreliable border — often skipped, at times the source of a bug someone spends a day on.
Trained on thirty images
U-Net won the 2015 ISBI cell tracking challenge with a training set of 30 images. That is that made it famous, and it wasn't reached by architecture alone.
The other half was boost, in detail elastic deformation: random smooth warps of the image and its mask. This Is a realistic model of how biological tissue really varies. Shift, rotation and flip boost make images that look like other images from the same microscope. Elastic deformation makes images that look like other *specimens*. For a domain with thirty labelled examples, that difference is all.
There's a third piece worth knowing: a weighted cross-entropy loss with a weight map found per image that heavily upweights the narrow gaps between touching cells. Without it a network trained on this data merges adjacent cells into one blob, because getting the thin separating line wrong costs almost nothing in unweighted pixel fit. This is the general lesson: with a class imbalance of a few thousand to one, the loss has to be told what matters, and per-pixel fit won't tell it.
import torch
import torch.nn as nn
def block(cin, cout):
return nn.Sequential(
nn.Conv2d(cin, cout, 3, padding=1), nn.BatchNorm2d(cout), nn.ReLU(inplace=True),
nn.Conv2d(cout, cout, 3, padding=1), nn.BatchNorm2d(cout), nn.ReLU(inplace=True))
class UNet(nn.Module):
def __init__(self, cin=3, classes=2, base=64, depth=4):
super().__init__()
chans = [base * 2 ** i for i in range(depth + 1)]
self.downs = nn.ModuleList()
for c in chans[:-1]:
self.downs.append(block(cin, c)); cin = c
self.bottom = block(chans[-2], chans[-1])
self.ups = nn.ModuleList()
self.convs = nn.ModuleList()
for c in reversed(chans[:-1]):
self.ups.append(nn.ConvTranspose2d(c * 2, c, 2, stride=2))
self.convs.append(block(c * 2, c)) # c from the skip + c upsampled
self.head = nn.Conv2d(base, classes, 1)
self.pool = nn.MaxPool2d(2)
def forward(self, x):
skips = []
for d in self.downs:
x = d(x); skips.append(x); x = self.pool(x)
x = self.bottom(x)
for up, conv, skip in zip(self.ups, self.convs, reversed(skips)):
x = up(x)
x = conv(torch.cat([skip, x], dim=1)) # the skip, concatenated
return self.head(x)
The line to stare at is block(c * 2, c) in the decoder. That c * 2 is the skip's channels plus the upsampled tensor's channels, and it's what the explorer's "decoder input" column is reporting. Drop the concatenation and it turns block(c, c) — which is why turning the skips off saves params.
Choosing the depth
The Levels control is the one with a real trade behind it, and the level table is where to read it.
Each level doubles the receptive field of all below it and quarters the spatial size. Too few levels and the deepest layer has never seen a zone large enough to spot a structure by its context — it will segment texture rather than objects. Too many and the bottom of the U is a handful of pixels holding a thousand channels. This Is a lot of params showing very little, and the decoder has more upsampling to invent.
The rule of thumb that comes out of it: the bottom of the U should be roughly the size of the largest structure you need to reason about, in units of the bottom's own stride. For 512×512 tiles of cell imagery, four levels puts the bottom at 32×32 with a receptive field of about 140 pixels — comfortably larger than a cell, comfortably smaller than the tile.
Watch what happens at the extremes in the explorer. Drop to two levels with a 572-pixel tile and the widest layer is only 256 channels. The param count falls by most of the model. Push to five with a small tile and the valid- convolution math runs out fully. This The explorer reports rather than silently making a nonsense number. The tile size has to be chosen so each pooling step cuts evenly, and that constraint is why the paper's input is the odd-looking 572 rather than 512.
What came after
Focus U-Net puts a gate on each skip so the decoder can suppress irrelevant regions the encoder passed forward. U-Net++ replaces the four direct skips with a dense nest of mid convolutions, on the arg that the encoder and decoder features at the same level are semantically mismatched. nnU-Net did a thing more notable: it left the architecture alone and automated all around it — prep, patch size, batch size, boost, postprocessing — and beat specialised architectures across dozens of medical benchmarks. That result is worth sitting with. Ten years on, the strongest arg in the area is still that a plain U-Net, configured well, is hard to beat.
Questions people ask
Concatenate the skip, or add it? Concatenate, in the first. Adding forces the encoder's detail and the decoder's semantics into the same channels. These convolution can't tell them apart. Concatenating keeps them separate and lets it weigh them. Addition is cheaper and appears in ResNet-style decoders. But It's a different architecture rather than an setup shortcut.
Why 572×572 and not 512? Because unpadded convolutions shrink the map and each pooling step still has to divide evenly. 572 is chosen so the math lives four levels down and back. With padding=1 the constraint relaxes to "divisible by 2levels". This Is why modern setups use round numbers.
Valid or same padding? Same (padding=1) unless you're tiling a larger image. Valid convolutions ensure each output pixel had genuine context. This Is what makes the share-tile strategy smooth. Same padding needs no cropping and costs you a thin border predicted partly from zeros.
Transposed convolution or upsample-then-convolve? Either works. Transposed convolutions can give checkerboard artefacts when the kernel size isn't divisible by the stride. Bilinear upsampling followed by a 3×3 avoids that and carries fewer params. This Is why most current setups choose it.
Why does my model merge touching objects? Because an unweighted per-pixel loss barely notices the thin line between them. A few hundred pixels out of a quarter of a million. The paper's answer was a per-image weight map upweighting just those gaps. The modern answers are Dice or focal loss, or predicting the boundary as its own class.
Does it really only need thirty images? The result is real. Not from the architecture alone. Elastic deformation is what made it work. A smooth random warp of tissue makes a thing that looks like a different *specimen*. Here Shift and flip only give another photo of the same one. With weak boost, thirty images isn't enough.
Recap in one screen
- Segmentation needs semantics. This Wants pooling. Precision, which doesn't. U-Net gets both by pooling all the way down and wiring the pre-pooling activations forward.
- Each encoder level is two 3×3s then a 2×2 pool with channels doubling. Each decoder level up-converts, concatenates the skip, and convolves twice.
- The skips *are* the architecture. Without them the finest detail the decoder can see is what survived four poolings. A 28×28 grid for a 572×572 tile. And no decoder room recovers it.
- Choose depth so the bottom of the U is roughly the size of the largest structure you must reason about, measured in the bottom's own stride.
- nnU-Net is the result to note: configured well, the plain architecture is still hard to beat.