Object Detection with Bounding Boxes

What a detector outputs and how it is trained

An object detector's output head does two jobs per candidate region: classify what is there. And regress four counts describing where it is precisely. Often a box centre, width and height. Or the offsets needed to nudge a fixed anchor box onto the real object. The sorting half is an ordinary softmax problem. The localisation half is what this module is about. And IoU sits at the centre of it twice: once inside the training loss. And again in how the net gets graded afterward.

This page covers the two box forms (corner and centre forms), how anchor boxes work, what the regression target really is. And the IoU-based loss that replaced smooth-L1 in most modern designs. The visualisation above lets you drag predicted and ground-truth boxes and watch IoU, the sorting label. And the loss value update in real time.

Anchor boxes are the starting point: a grid of fixed rectangles at multiple scales and aspect ratios, each one a hypothesis the net refines. The regression head predicts small offsets (dx, dy, dw, dh) relative to each anchor rather than absolute coordinates. It keeps the counts small and training stable. At test time, non-max suppression collapses overlapping predictions into one box per object, with IoU to decide which boxes are duplicates.

IoU as loss, IoU as grade

A natural localization loss is simply 1 − IoU between the predicted box and the ground truth box: 0 when they coincide just, approaching 1 as they stop overlapping at all. At test time, the same IoU count gets a other job. A detection is often scored as a true positive just if its IoU with the matching ground truth box clears a threshold, conventionally 0.5 under the PASCAL VOC convention. A box can be visibly "close" and still count as a complete miss if it doesn't clear that bar.

Sorting says what. Detection says what and where

A sorter answers one question about a whole pic. A detector answers two questions about each object in it: what is it. And where is it — expressed as a rectangle.

A bounding box is four counts. And there are three common conventions:

Mixing them up is the single most common bug in detection code. And it fails quietly: boxes appear in roughly plausible places and the metrics are mysteriously poor. Always check which convention a dataset and a net each expect.

Each box also carries a class label and a confidence score. So one detection is: a rectangle, a name. And a count between 0 and 1.

The hard parts

Detection is much harder than sorting for reasons that are structural rather than incidental.

The count of outputs varies. A network gives a fixed-size output. An pic may hold zero objects or fifty. Each detector design is, at heart, a way of handling that mismatch.

Objects vary enormously in scale. A pedestrian may be 20 pixels tall or 500. A single feature map has one effective scale. It is why detectors predict from a few feature maps at once.

Localisation and sorting are other tasks with other losses, trained jointly. Often a cross-entropy for the class and a regression loss (smooth L1, or a direct IoU loss) for the coordinates.

The background dominates. Most candidate positions hold nothing at all, often by a thousand to one. That extreme imbalance is why focal loss. Which down-weights easy, well-classified cases — was invented for detection namely.

Two families of detector

Two-stage detectors (R-CNN, Fast R-CNN, Faster R-CNN) first propose regions that might hold something, then classify and refine each proposal. More exact, historically slower. And still preferred where precision matters more than latency.

One-stage detectors (YOLO, SSD, RetinaNet) predict boxes and classes right from the feature maps in a single pass. Much faster, and the score gap has largely closed. Which is why most production systems now use them.

Anchor-based versus anchor-free is the other axis. Anchor-based ways place a set of predefined box shapes at each position and learn offsets from them. Anchor-free ways (FCOS, CenterNet, recent YOLO versions) predict boxes right, removing the anchor hyperparameters that were always awkward to tune.

DETR and its successors take a third route fully, treating detection as set prediction with a transformer and removing the need for non-max suppression — conceptually cleaner. And slower to train.

Four counts, and all that depends on them

A bounding box is four counts. And almost each detection bug comes from a disagreement about which four. This works through the formats, the conversions, IoU, non-max suppression and the loss -- all of it arithmetic you can check line by line.

import numpy as np

print("THE SAME BOX, IN FOUR CONVENTIONS. every library picks one and")
print("every one of them is called 'the box':")
box_xyxy = (20.0, 30.0, 80.0, 90.0)
x1, y1, x2, y2 = box_xyxy
w, h = x2 - x1, y2 - y1
cx, cy = x1 + w / 2, y1 + h / 2
IMGW, IMGH = 200.0, 150.0
print("%-26s %-34s %s" % ("format", "values", "used by"))
print("%-26s %-34s %s"
      % ("xyxy (corners)", str(box_xyxy), "torchvision, COCO eval"))
print("%-26s %-34s %s"
      % ("xywh (top-left + size)", str((x1, y1, w, h)), "COCO annotations"))
print("%-26s %-34s %s"
      % ("cxcywh (centre + size)", str((cx, cy, w, h)), "YOLO, DETR"))
print("%-26s %-34s %s"
      % ("normalised cxcywh",
         "(%.3f, %.3f, %.3f, %.3f)"
         % (cx / IMGW, cy / IMGH, w / IMGW, h / IMGH), "YOLO label files"))
print("   they describe the identical rectangle. mixing two of them")
print("   produces boxes that are silently wrong rather than an error,")
print("   which is why this is the first thing to check when a detector")
print("   trains to a loss that falls but a mAP that does not.")
print()
print("   the conversions, both directions, so you can check them:")
print("      xyxy -> cxcywh:  cx=(x1+x2)/2  cy=(y1+y2)/2  w=x2-x1  h=y2-y1")
print("      cxcywh -> xyxy:  x1=cx-w/2  y1=cy-h/2  x2=cx+w/2  y2=cy+h/2")
back = (cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2)
print("      round trip: %s -- max error %.1e"
      % (str(back), max(abs(a - b) for a, b in zip(back, box_xyxy))))

Another subtle disagreement is whether coordinates refer to pixel indices or pixel edges.

print("ONE MORE DISAGREEMENT, and it is the nastiest: is the coordinate a")
print("pixel INDEX or a pixel EDGE?")
print("      a box covering pixel columns 0 to 9 inclusive is")
print("         x1=0, x2=9   if coordinates are indices  -> width 9")
print("         x1=0, x2=10  if coordinates are edges     -> width 10")
print("   the difference is one pixel, and it changes every area, every")
print("   IoU and every reported score by a little. modern code uses")
print("   edges, so width is exactly x2-x1. the original Pascal VOC used")
print("   indices and added 1, which is why VOC numbers and COCO numbers")
print("   for the same model differ slightly.")

IoU measures how well a predicted box overlaps the ground truth. And the max/min pattern makes it straightforward to compute.

def iou(a, b):
    ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])
    ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])
    inter = max(0.0, ix2 - ix1) * max(0.0, iy2 - iy1)
    aa = (a[2] - a[0]) * (a[3] - a[1])
    bb = (b[2] - b[0]) * (b[3] - b[1])
    return inter / (aa + bb - inter)

print("IoU: intersection over union. the intersection is a rectangle, so")
print("it is four max/min operations and a multiply:")
truth = (20.0, 30.0, 80.0, 90.0)
cases = [("exact", (20., 30., 80., 90.)),
         ("shifted 10px right", (30., 30., 90., 90.)),
         ("half the size, centred", (35., 45., 65., 75.)),
         ("twice the size, centred", (-10., 0., 110., 120.)),
         ("touching, not overlapping", (80., 30., 140., 90.)),
         ("miles away", (150., 10., 190., 50.))]
print("%-28s %12s %12s %12s %10s"
      % ("prediction", "intersect", "union", "IoU", "counts?"))
for name, b in cases:
    ix1, iy1 = max(truth[0], b[0]), max(truth[1], b[1])
    ix2, iy2 = min(truth[2], b[2]), min(truth[3], b[3])
    inter = max(0.0, ix2 - ix1) * max(0.0, iy2 - iy1)
    union = ((truth[2] - truth[0]) * (truth[3] - truth[1])
             + (b[2] - b[0]) * (b[3] - b[1]) - inter)
    v = inter / union
    print("%-28s %12.0f %12.0f %12.4f %10s"
          % (name, inter, union, v, "yes" if v >= 0.5 else "no"))
print("   note the fourth row. a prediction that CONTAINS the object")
print("   completely still scores only %.2f, because union punishes the"
      % iou(truth, cases[3][1]))
print("   extra area. IoU cares about agreement, not about coverage --")
print("   a box around the whole image finds every object and scores")
print("   almost nothing.")

NMS removes near-duplicate detections by keeping the most confident box and suppressing overlapping ones.

import numpy as np

def iou(a, b):
    ix = max(0., min(a[2],b[2])-max(a[0],b[0])) * max(0., min(a[3],b[3])-max(a[1],b[1]))
    return ix / ((a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - ix)

DETS = [((20., 30., 80., 90.), 0.94),
        ((24., 33., 84., 92.), 0.88),
        ((18., 28., 76., 86.), 0.79),
        ((110., 40., 170., 100.), 0.72),
        ((115., 45., 175., 104.), 0.65),
        ((38., 48., 98., 108.), 0.51)]

print("NON-MAX SUPPRESSION. a detector fires many times on one object, so")
print("the duplicates have to be removed. sort by confidence, take the")
print("best, delete everything that overlaps it too much, repeat:")
def nms(dets, thr):
    order = sorted(range(len(dets)), key=lambda i: -dets[i][1])
    keep, killed = [], {}
    while order:
        i = order.pop(0)
        keep.append(i)
        rest = []
        for j in order:
            v = iou(dets[i][0], dets[j][0])
            if v > thr:
                killed[j] = (i, v)
            else:
                rest.append(j)
        order = rest
    return keep, killed
keep, killed = nms(DETS, 0.5)
print("threshold 0.5 -> keeps %d of %d" % (len(keep), len(DETS)))

Lowering the threshold suppresses more boxes. Check the two settings side by side.

import numpy as np

def iou(a, b):
    ix = max(0., min(a[2],b[2])-max(a[0],b[0])) * max(0., min(a[3],b[3])-max(a[1],b[1]))
    return ix / ((a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - ix)

DETS = [((20., 30., 80., 90.), 0.94),
        ((24., 33., 84., 92.), 0.88),
        ((18., 28., 76., 86.), 0.79),
        ((110., 40., 170., 100.), 0.72),
        ((115., 45., 175., 104.), 0.65),
        ((38., 48., 98., 108.), 0.51)]

def nms(dets, thr):
    order = sorted(range(len(dets)), key=lambda i: -dets[i][1])
    keep, killed = [], {}
    while order:
        i = order.pop(0)
        keep.append(i)
        rest = []
        for j in order:
            v = iou(dets[i][0], dets[j][0])
            if v > thr:
                killed[j] = (i, v)
            else:
                rest.append(j)
        order = rest
    return keep, killed

for thr in (0.5, 0.3):
    keep, killed = nms(DETS, thr)
    print("threshold %.1f -> keeps %d of %d" % (thr, len(keep), len(DETS)))

The threshold trades duplicate removal against the risk of merging genuinely distinct objects.

def iou(a, b):
    ix = max(0., min(a[2],b[2])-max(a[0],b[0])) * max(0., min(a[3],b[3])-max(a[1],b[1]))
    return ix / ((a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - ix)

DETS = [((20., 30., 80., 90.), 0.94),
        ((24., 33., 84., 92.), 0.88),
        ((18., 28., 76., 86.), 0.79),
        ((110., 40., 170., 100.), 0.72),
        ((115., 45., 175., 104.), 0.65),
        ((38., 48., 98., 108.), 0.51)]

print("   the threshold is a real trade-off and there is no safe default.")
print("   det 5 is the interesting one: its IoU with det 0 is %.3f, so it"
      % iou(DETS[5][0], DETS[0][0]))
print("   survives at 0.5 and is deleted at 0.3. it is a genuinely")
print("   different object that happens to overlap the first one, and no")
print("   threshold can tell that apart from a duplicate -- NMS sees only")
print("   the overlap, never what is inside the boxes.")
print("   too high a threshold leaves duplicates; too low deletes real")
print("   objects in crowds. this is precisely the failure that")
print("   soft-NMS (decay the score instead of deleting) and DETR")
print("   (predict a fixed set, no NMS at all) were designed around.")

Detection also needs a regression loss on the box coordinates. And the standard choices each have a weakness.

def iou(a, b):
    ix = max(0., min(a[2],b[2])-max(a[0],b[0])) * max(0., min(a[3],b[3])-max(a[1],b[1]))
    return ix / ((a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - ix)

print("THE LOSS. detection needs a regression loss on the coordinates, and")
print("the obvious choices are all subtly wrong:")
print("%-22s %14s %14s %16s"
      % ("prediction error", "L2 loss", "L1 loss", "smooth L1"))
for e in (0.1, 0.5, 1.0, 3.0, 10.0):
    sl1 = 0.5 * e * e if abs(e) < 1 else abs(e) - 0.5
    print("%-22.1f %14.3f %14.3f %16.3f" % (e, e * e, abs(e), sl1))
print("   L2 explodes on outliers: a 10-pixel error contributes %.0f,"
      % (10.0 ** 2))
print("   %.0fx as much as a 1-pixel error, so a single badly-placed box"
      % (10.0 ** 2 / 1.0))
print("   dominates the batch. L1 does not, but its gradient is constant,")
print("   so it never settles down near zero. smooth L1 is L2 below 1 and")
print("   L1 above -- fine gradients where you are close, bounded")
print("   influence where you are far.")
print()
print("   AND EVEN SMOOTH L1 IS OPTIMISING THE WRONG THING. two")
print("   predictions with the same coordinate error can have very")
print("   different IoU:")
for name, b in (("small object, off by 10", (20., 30., 40., 50.)),
                ("large object, off by 10", (20., 30., 120., 130.))):
    shifted = (b[0] + 10, b[1], b[2] + 10, b[3])
    print("      %-28s L1 error %2.0f, IoU %.3f"
          % (name, 10.0, iou(b, shifted)))
print("   identical loss, very different quality. that mismatch is why")
print("   modern detectors optimise IoU directly -- GIoU, DIoU, CIoU --")
print("   rather than the coordinates. the metric and the loss should be")
print("   the same thing whenever you can arrange it.")

Things to try

1. Read the starting position. The predicted box is offset and undersized relative to the ground truth. Overlap is low, loss is close to 1. And it does not count as a match.
2. Drag dx and dy toward 0. IoU climbs as the boxes line up spatially. And the loss falls in lockstep. This is the gradient signal a real localization head would be trained on.
3. Now close dw and dh too. Position alone isn't enough. A box in the right place but the wrong size still loses IoU on both sides.
4. Watch the Match readout as IoU crosses 0.5. It flips from a miss to a match at just that edge, still though the loss was now improving smoothly well before the flip.

Summing up

1 − IoU is a smooth, differentiable signal that a network can really be trained against. Whether a detection counts as correct at test time is not smooth at all. It's a hard threshold on that same IoU count. A net can make genuine, loss-reducing progress on a box for a long stretch before that progress ever shows up as a correct detection in your metrics.

How detections are scored

IoU (Intersection over Union) measures how well a predicted box overlaps a true one:

IoU = area of overlap / area of union

It runs from 0 (no overlap) to 1 (perfect). A prediction often counts as correct if its IoU with a ground-truth box of the same class exceeds a threshold. 0.5 is the classic choice.

Non-max suppression is the cleanup step. A detector often fires a few times on the same object. So NMS keeps the highest-scoring box and discards any box overlapping it by more than a threshold. Without it, each object is reported three or four times.

mAP (mean Average Precision) is the standard headline metric. For each class, sweep the confidence threshold, build a precision-recall curve, take its area. Then average across classes. COCO's version averages more over IoU thresholds from 0.5 to 0.95. It rewards precise localisation rather than approximate boxes.

The real reading: mAP@0.5 tells you whether objects are found. MAP@0.5:0.95 tells you whether the boxes are tight.

Annotation, and where projects really fail

Detection datasets are costly. And the quality of the boxes sets the ceiling on the net.

• Be consistent about tightness. Does the box include the shadow? The handle? The occluded part? Write the rule down before annotating, since a net trained on inconsistent boxes learns the inconsistency.
• Label each instance. A missed object is trained as background. It actively teaches the net not to detect it.
• Decide on occlusion and truncation rules for objects half out of frame.
• Watch class imbalance. Rare classes need either more cases or a weighted loss.
• Match augmentation to the boxes. Flipping, rotating or cropping an pic must transform the coordinates too. Libraries such as Albumentations do this correctly. And hand-rolled pipelines often do not.

A few hundred well-annotated pics plus a pretrained backbone will often beat thousands of sloppy ones.

Questions people ask

Which detector should I start with? A recent YOLO or a torchvision Faster R-CNN, fine-tuned from pretrained weights. Training a detector from scratch needs far more data than most projects have.

How much data do I need? With transfer learning, a few hundred annotated pics per class gets a usable net. Thousands for production reliability.

What IoU threshold should I use for NMS? 0.5 is typical. Less it if duplicates persist, raise it if genuinely overlapping objects are being suppressed.

Why are small objects missed? Since they occupy few pixels in the feature maps used for prediction. Feature pyramids, more-resolution input and tiled test time are the standard replies.

What is the gap from segmentation? Detection gives boxes. Semantic segmentation labels each pixel. Instance segmentation gives per-object masks. Boxes are cheaper to annotate and often enough.

How do I detect objects in video? Detect per frame, then associate detections across frames with a tracker (SORT, ByteTrack) to give each object a persistent identity.

Recap in one screen

• Detection outputs a box, a class and a confidence for each object present.
• Box formats differ between datasets — corners, corner-plus-size. And scaled centre-plus-size.
• Two-stage detectors propose then classify. One-stage detectors predict right and are now the real default.
• IoU measures overlap, NMS removes duplicates. And mAP is the headline metric.
• Annotation consistency sets the ceiling. Augmentation must transform the boxes as well as the pic.