vizlearn

Mask R-CNN

Computer VisionBy Updated

The paper's whole contribution is that RoIPool rounds twice and a mask cannot survive it. Drag a proposal and watch the rounding move it eight pixels.

Overview

The architecture, in four parts

Mask R-CNN inherits almost everything from Faster R-CNN and adds one branch.

  1. A backbone with a feature pyramid — usually ResNet-50 or -101 with FPN — producing feature maps at strides 4, 8, 16 and 32.
  2. A region proposal network sliding over those maps, emitting a few thousand class-agnostic "something is here" boxes, cut to around 1,000 by NMS.
  3. RoIAlign, which crops a fixed-size feature patch for each proposal.
  4. Three heads on that patch: classification, box refinement, and the new one, a mask.

The heads are where the parameter budget sits, and the split is unintuitive:

HeadInputStructureOutputParameters
classification7×7×256fc 12544→1024, fc 1024→102481 scores~13.9 M
boxshares those two fc layers4 × 80 offsets~0.33 M
mask14×14×2564 × conv3×3(256), deconv, conv1×180 × 28 × 28~2.64 M

The mask head is the *cheapest* of the three despite producing 62,720 numbers, because it is fully convolutional and the box head is not.

RoIAlign against RoIPool, on a real feature map

This explorer needs JavaScript: every shape, parameter count and curve on it is computed in the page rather than downloaded as an image.

Worth knowing

Mask R-CNN is Faster R-CNN plus a third head. The third head only works because of RoIAlign.
RoIPool rounds twice: once snapping the region to whole cells, once snapping the bin boundaries. At stride 16 half a cell is 8 pixels.
The mask head predicts one 28×28 mask per class and the loss only touches the ground-truth class's channel.
Classification never noticed the misalignment. A per-pixel output notices immediately — the paper reports up to a 50% relative gain on mask AP under strict IoU.

Mask R-CNN

Faster R-CNN with a third head, and one fixed rounding bug that is the entire reason the third head works.

The bug that mattered

RoIPool, inherited from Fast R-CNN, converts an arbitrary region of a feature map into a fixed 7×7 grid. It does so with two roundings:

  1. The proposal's floating-point coordinates are snapped to whole feature cells.
  2. The 7×7 bin boundaries inside that region are snapped to whole cells too.

Drag the proposal in the explorer and watch the dashed red box — what RoIPool actually pools — jump around the solid orange one, which is what the RPN proposed. The statistic underneath translates the gap into input pixels: at stride 16, being half a feature cell out is 8 pixels in the original image. At stride 32 it is 16.

For classification this genuinely does not matter. The question "is this a cat" has the same answer whether your crop is eight pixels off, and the pooled features are a summary either way. For a mask it matters enormously, because the output is a per-pixel map that gets pasted back onto the image at that location. An eight-pixel systematic offset is visible in every single prediction.

RoIAlign

RoIAlign removes both roundings. The region keeps its floating-point coordinates, the bins are divided exactly, and inside each bin four sample points are read by bilinear interpolation from the four nearest feature cells — then averaged.

The dots in the explorer are those sample points. Watch them slide smoothly as you drag, where the red box jumps in discrete steps.

The smoothness is the technical point, not just an aesthetic one. Bilinear interpolation is differentiable with respect to the sampling location, so gradients flow back to the box coordinates. Rounding is a step function whose derivative is zero everywhere it is defined, so RoIPool silently cut that path.

The two pooled maps are drawn side by side with their mean absolute difference. Nudge the proposal width by a tenth of a cell: the RoIAlign map barely moves, the RoIPool map can change substantially. The paper reports that this change alone improves mask AP by around 10 points relative on COCO, and by up to 50% relative under the strict IoU=0.75 criterion — where being eight pixels out is precisely what decides a match.

Decoupling mask from class

The second design decision is easy to miss. The mask head predicts K separate masks, one per class, and the loss is only applied to the channel of the ground-truth class.

The alternative — one mask with a per-pixel softmax over classes — is what FCN-style semantic segmentation does, and it forces the classes to compete pixel by pixel. Mask R-CNN does not need that competition, because the classification head has already decided what the object is. Letting the mask branch answer "which pixels belong to *this* object" independently of "what class is it" is worth several points of AP, and it is why the mask branch never learns to suppress one class in favour of another.

At inference you take the classification head's argmax and read only that channel's 28×28 mask, resize it to the predicted box, and threshold at 0.5. The resize is why instance masks from this family have a characteristically soft, slightly blobby boundary: the real resolution of the answer is 28×28 inside the box, however large the box is.

Which pyramid level a proposal is read from

One detail sits between the RPN and RoIAlign and it is the reason the stride control on this page has three settings.

With a feature pyramid there are four candidate maps to crop from, at strides 4, 8, 16 and 32, and a proposal has to be assigned to one. Cropping a small object from the stride-32 map would give a region a couple of cells across, which contains almost nothing; cropping a large one from stride 4 wastes work and gives features with too little context.

FPN assigns by size: a proposal of area *A* goes to level k = floor(4 + log2(sqrt(A) / 224)), clamped to the available range. A 224×224 proposal — the ImageNet size, chosen deliberately — lands on level 4, at stride 16. Anything smaller drops to a finer level and anything larger rises to a coarser one.

Switch the stride control in the explorer and watch the misalignment figure scale with it. The same half-cell rounding error is 2 pixels at stride 4 and 16 pixels at stride 32, which is why the coarse levels are where RoIPool hurt most, and why the effect was largest on exactly the large objects that a detector otherwise finds easy.

The loss, and the multi-task balance

L = L_cls + L_box + L_mask

Three terms, equally weighted, plus the RPN's own two. L_mask is an average binary cross-entropy over the 28×28 grid of the correct class's channel only. It is worth noticing that nothing here is tuned: the paper does not weight the mask loss up or down, which is unusual for a multi-task network and suggests the three tasks are genuinely compatible rather than competing for capacity.

import torchvision
from torchvision.models.detection import maskrcnn_resnet50_fpn

model = maskrcnn_resnet50_fpn(weights="DEFAULT").eval()
out = model([image_tensor])[0]

keep = out["scores"] > 0.5
boxes = out["boxes"][keep]           # [N, 4]
masks = out["masks"][keep]           # [N, 1, H, W], already pasted, soft
labels = out["labels"][keep]

binary = masks[:, 0] > 0.5           # the 0.5 threshold, applied explicitly

Two things in that snippet catch people. masks comes back already resized and pasted to full image resolution, and it is *soft* — probabilities, not booleans — so a threshold has to be applied, and 0.5 is a choice rather than a law. And the masks overlap: this is instance segmentation, so two instances can both claim a pixel, and resolving that into a single label per pixel is panoptic segmentation, which is a different task with a different metric.

What it left behind

Mask R-CNN was state of the art in 2017 and is now the baseline that faster methods are measured against. YOLACT and SOLO produce instance masks in one stage; Mask2Former and the DETR family replace the whole propose-and-crop structure with set prediction and attention, and have no RoIAlign, no NMS and no anchors at all.

But RoIAlign itself outlived the architecture. Any time you need to read a feature map at a location that is not on the grid — deformable convolutions, spatial transformers, keypoint heads, the 3-D detectors that project points into image features — the answer is bilinear sampling for exactly the reasons here: it is accurate and it is differentiable in the coordinate. The lesson generalises past detection: if a location is continuous, do not round it.

Check yourself

0 of 4

Answer without scrolling back up.

  1. RoIPool rounds twice. Where?

  2. Why did that misalignment not matter for Faster R-CNN?

  3. Besides accuracy, what does bilinear sampling give that rounding does not?

  4. The mask head predicts one 28x28 mask per class rather than one mask with a per-pixel class softmax. Why?

Cheat sheet

Mask R-CNN

The paper's whole contribution is that RoIPool rounds twice and a mask cannot survive it. Drag a proposal and watch the rounding move it eight pixels.

COMPUTER VISION · vizlearn.in/computer_vision/mask_rcnn.html

More in Computer Vision

Computer VisionVision Transformer PatchesComputer VisionGrad-CAMComputer VisionHow Neural Networks Process ImagesComputer VisionGrayscale Image Processing

Further reading

About the author

Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.

About VizLearnGitHubLinkedInContact