IoU and Non-Max Suppression
A detector never proposes just one box per object — it proposes dozens. IoU measures how much two boxes overlap; NMS uses that number to collapse the pile back down to one box per object.
Overview
The problem it solves
A detector's region proposal stage doesn't stop at one box per object — it scores hundreds of candidate boxes and keeps every one above a confidence floor, which for a single real object usually means a cluster of overlapping, near-duplicate boxes. Intersection over Union (IoU) is the standard way to measure how much two boxes overlap: the area they share, divided by the total area either one covers. IoU = 1 means identical boxes; IoU = 0 means no overlap at all.
NMS Threshold
boxes A-D are four raw proposals from one detector pass, four confidence scores, two actual objects
Confidence
Proposals
—Pairwise IoU
Result
IoU and Non-Max Suppression: A Practical Guide
Turning a pile of overlapping boxes into one box per object.
Non-Max Suppression
NMS turns that overlap score into a cleanup rule. Sort every proposal by confidence, descending. Take the top one, keep it, and discard every remaining box whose IoU with it exceeds a threshold — those are treated as duplicate detections of the same object. Move to the next surviving box by confidence and repeat, until nothing is left to process. The threshold is the only knob: too low and boxes on genuinely different but nearby objects get merged into one; too high and duplicate boxes on the same object all survive.
Measuring overlap with one number
IoU compares two boxes by dividing the area they share by the area they cover together:
IoU = intersection / union
Worked through. Box A spans (0, 0) to (100, 100); box B spans (50, 50) to (150, 150).
- Intersection: from (50, 50) to (100, 100) — 50 × 50 = 2,500.
- Union: 10,000 + 10,000 − 2,500 = 17,500.
- IoU = 2,500 / 17,500 = 0.14.
Reading the scale:
| IoU | Interpretation |
|---|---|
| > 0.9 | Near-perfect box |
| 0.7–0.9 | Good localisation |
| 0.5–0.7 | Acceptable by the traditional threshold |
| < 0.5 | Usually counted as a miss |
| 0 | No overlap at all |
Two properties make it the standard measure. It is scale-invariant — a 20-pixel error on a small object scores the same as a proportionally similar error on a large one — and it is bounded, so it can be thresholded and averaged.
def iou(a, b):
x1, y1 = max(a[0], b[0]), max(a[1], b[1])
x2, y2 = min(a[2], b[2]), min(a[3], b[3])
inter = max(0, x2 - x1) * max(0, y2 - y1)
area_a = (a[2] - a[0]) * (a[3] - a[1])
area_b = (b[2] - b[0]) * (b[3] - b[1])
return inter / (area_a + area_b - inter)The two max(0, ...) calls are what handle boxes that do not overlap; without them a negative width times a negative height produces a positive area and a nonsensical result.
Non-maximum suppression, step by step
A detector fires at many nearby positions, so one object produces a cluster of boxes. NMS reduces each cluster to one.
- Discard every box below a confidence threshold.
- Sort the rest by confidence, highest first.
- Take the top box and keep it.
- Remove every remaining box whose IoU with it exceeds the NMS threshold.
- Repeat from step 3 with what is left.
Two thresholds control the outcome, and they do different jobs. The confidence threshold decides how sure the model must be before a box is considered at all — raise it for fewer false positives, lower it for better recall. The NMS IoU threshold decides how much overlap counts as a duplicate; 0.45–0.5 is typical.
The failure mode is genuinely overlapping objects: two people standing close together produce boxes with high IoU, and standard NMS deletes one of them. Soft-NMS addresses this by reducing the confidence of overlapping boxes rather than deleting them outright, so a strong second detection can still survive.
Note also that NMS is applied per class in most implementations — a dog box and a person box overlapping heavily are not duplicates of each other.
Scoring box overlap, then removing the duplicates
Intersection over union is four max/min operations. Non-max suppression is a sort and a loop. Both are written out here, along with the two cases where each one behaves badly.
Try it yourself
- Read the stage at the default threshold. Box A (confidence 0.95) is kept. Box D, a separate object with no real overlap with A, is also kept. Boxes B and C overlap A too heavily and are suppressed as duplicates.
- Lower the threshold toward 0.1. Nothing changes here — B and C already fail a much looser bar, so tightening it further has no effect on this particular layout.
- Raise the threshold past roughly 0.59. Box B's overlap with A no longer exceeds the (now looser) suppression bar, so B is kept as a second detection of what is really the same object.
- Raise it past roughly 0.81. Now C survives too — at this threshold NMS considers all four boxes different enough to keep, even though A, B and C clearly describe one dog.
The short of it
IoU is a pure geometry calculation — it knows nothing about confidence or class. NMS is the policy layer that uses it: keep the most confident box, discard anything that overlaps it past a threshold, repeat. Set the threshold too low and it merges distinct nearby objects into one detection; set it too high and duplicate boxes on the same object all survive. Every object detector that outputs boxes runs some form of this after its raw proposals come out.
IoU as a loss function
Since IoU is what evaluation measures, training directly on it is appealing — and it took several iterations to make it work.
Plain IoU has a fatal gradient problem: when two boxes do not overlap at all, IoU is 0 for every possible position, so there is no gradient telling the box which way to move. Three refinements fix it:
- GIoU adds a penalty based on the smallest box enclosing both, so non-overlapping boxes still receive a useful gradient.
- DIoU adds a term for the distance between the box centres, which converges faster.
- CIoU adds an aspect-ratio term as well, and is the default regression loss in several modern detectors.
The alternative, still widely used, is smooth L1 on the coordinates — simple and stable, but it optimises a proxy rather than the metric, and it treats a 10-pixel error the same on a small object as on a large one.
Where IoU appears beyond detection
- Assigning anchors during training. An anchor box is treated as a positive example if its IoU with a ground-truth box exceeds a threshold, and as background below a lower one. Anchors in between are ignored.
- Segmentation. The same ratio computed over pixel masks rather than boxes is the Jaccard index, and mean IoU is the standard segmentation metric.
- Tracking. Frame-to-frame association is usually IoU-based — a detection is matched to the track whose last box it overlaps most.
- Evaluation at several thresholds. COCO's primary metric averages mAP over IoU thresholds from 0.5 to 0.95, which is what makes it reward tight boxes rather than approximate ones.
Common mistakes
- Forgetting to clamp negative overlaps, giving a positive intersection for disjoint boxes.
- Mixing box formats. Applying corner-format code to centre-format boxes produces plausible-looking nonsense.
- Running NMS across classes when it should be per class, deleting legitimate detections of different objects.
- An NMS threshold that is too low, which removes genuine nearby objects; or too high, which leaves duplicates.
- Tuning the confidence threshold on the test set rather than on validation data.
- Assuming NMS is cheap. With tens of thousands of candidate boxes it can dominate inference time; batched GPU implementations exist for this reason.
Questions people ask
What IoU threshold means "correct"? 0.5 by the Pascal VOC convention. COCO averages from 0.5 to 0.95 in steps of 0.05, which is stricter and now more standard.
Can IoU exceed 1? No — the intersection can never exceed the union.
Is NMS still needed? For most detectors, yes. DETR-style set-prediction models avoid it by design, which is one of their main attractions.
How do I handle heavily overlapping objects? Soft-NMS, a higher NMS threshold, or an architecture that predicts a set directly.
Should I train with an IoU-based loss? CIoU or DIoU generally beats smooth L1 on localisation quality, and both are readily available in modern detection libraries.
Why is my mAP low despite good-looking boxes? Often the strict IoU thresholds: boxes that look right to a human may score 0.6, which passes at 0.5 and fails at 0.75.
Recap in one screen
- IoU is overlap divided by union — scale-invariant, bounded between 0 and 1.
- Clamp negative overlaps to zero, or disjoint boxes report a false positive area.
- NMS keeps the highest-confidence box and removes overlapping duplicates, per class.
- Two thresholds matter: confidence (how sure) and NMS IoU (how much overlap is a duplicate).
- GIoU, DIoU and CIoU make IoU usable as a training loss by supplying gradients when boxes do not overlap.