The number every detection paper reports, built from a precision-recall curve one ranked prediction at a time.
Overview
Deciding what counts as correct
Classification has an obvious notion of right. Spotting doesn't: a box that covers most of an object is partly right, and "partly" has to be resolved before any of it can be counted.
The detail is a threshold on IoU. A guess is a true plus if its IoU with a ground-truth box of the same class is at least the threshold, and if that ground-truth box hasn't already been claimed by a higher-trust guess. Or else it's a false plus. Ground-truth boxes nothing matched are false negatives.
That second rule matters. Without it, ten shared guesses on one object would count as ten successes.
Mean Average Precision
This module needs JavaScript: the
images are computed in the page rather than downloaded.
Worth knowing
A prediction counts as correct if its IoU with an unmatched ground-truth box clears a threshold.
Sort every prediction by confidence, walk down the list, and plot precision against recall as you go.
AP is the area under that curve for one class. mAP averages AP across classes.
COCO's headline number averages again over IoU thresholds from 0.50 to 0.95, which is why it is so much lower than the old VOC number.
Mean Average Precision
How a list of boxes with trust scores turns one number. What that number isn't telling you.
Building the curve
Sort all guesses for a class by trust, highest first, and walk down the list. After each one, recompute precision and recall over all seen so far, and plot the point.
The curve in the visualisation is just that walk, one dot per guess. It starts high — the most confident guesses are often right — and sags as the list goes on. Later guesses are less solid while the recall denominator stays fixed.
Mean precision is the area under that curve. It summarises the whole ranking rather than speed at one operating point. This Is what makes it handy: a finder that ranks its correct guesses above its incorrect ones scores well even so of how its confidences happen to be calibrated.
The exact area math has varied. VOC 2007 sampled precision at eleven recall levels. VOC 2010 onward and COCO interpolate more finely. The differences are small and the meanings aren't the same. This Is a reason to compare numbers only within one rule.
From AP to mAP
AP is per class. mAP is the mean of AP over all classes — an unweighted mean, so a class with ten instances counts as much as a class with ten thousand.
That is a deliberate choice and worth knowing. It means mAP is dragged down hard by rare classes the model handles badly. A model can improve its mAP more by fixing one rare class than by getting slightly better at a common one.
Why COCO's numbers look worse
Drag the IoU control. At 0.5 a loosely fitting box counts as a hit. At 0.9 the box has to be nearly exact.
VOC reported mAP at IoU 0.5 alone. COCO's headline metric means mAP over ten thresholds from 0.50 to 0.95 in steps of 0.05, often written mAP@[.5:.95]. It rewards precise localisation rather than rough spotting, and it's much harsher: a finder reporting 0.80 under VOC might report 0.45 under COCO on the same guesses.
So a COCO mAP and a VOC mAP are different readings and must never be checked right. COCO also reports mAP@0.5 alongside, partly so that check remains possible.
COCO breaks the number down further. By object size (small, medium, large) and at fixed spotting counts. Those breakdowns are often more helpful than the headline: "our finder is bad at small objects" is actionable in a way that "our mAP is 0.42" isn't.
What mAP does not tell you
It has no threshold. AP integrates over each trust level, so it says nothing about which threshold to deploy. That is a separate choice.
It skips the cost of errors. A missed pedestrian and a spurious traffic cone count the same.
It means away the failures you care about. A model that is great on 19 classes and useless on the 20th can outscore one that is decent on all of them.
It says nothing about duplicates before NMS. mAP is found on post-processed output, so a finder that fires many shared boxes can look fine if suppression cleans up.
AP is the area, worked
Mean precision is the area under the precision-recall curve, and finding one on a handful of points shows what it rewards. Walk down the trust-sorted guesses, recomputing precision and recall after each:
recall
precision
0.2
1.00
0.4
1.00
0.6
0.67
0.8
0.50
1.0
0.40
Summing precision across each recall step gives AP ≈ 0.71. Because it integrates over the whole ranking rather than one operating point, a finder that puts its correct guesses above its wrong ones scores well even so of how its confidences are calibrated — which is just what you want from a summary of a ranking. The catch is that AP has no threshold. It tells you nothing about which trust to deploy, and mAP means it *unweighted* across classes, so a rare class the model handles badly drags the number down as hard as a common one. And COCO's headline means this again over ten IoU thresholds from 0.50 to 0.95, which is why a finder reporting 0.80 under VOC's single 0.50 threshold might report 0.45 under COCO on the same guesses. The numbers are different readings and can't be checked.
What mAP hides is as key as what it reports. It has no threshold, so it says nothing about which trust to deploy. It skips the cost of errors, counting a missed pedestrian and a spurious traffic cone the same. And being an unweighted mean over classes, a model great on nineteen and useless on the twentieth can outscore one that is decent on all. COCO's breakdowns. By object size, at fixed spotting counts — are often more actionable than the headline, because "bad at small objects" tells you what to fix where "mAP 0.42" doesn't. And tuning NMS to lift mAP is a trap: the operating point that raises an integral over all thresholds isn't the one you ship. So It often makes the deployed finder worse.
Where it goes wrong with Mean Average Precision
Checking across datasets or rules. VOC against COCO, or one AP interpolation against another, aren't close.
Reporting mAP alone. Report the per-class breakdown. It's where the info is.
Tuning NMS to raise mAP. Easy to do and often makes the deployed finder worse, because the operating point that raises an integral over all thresholds isn't the one you ship.
Questions people ask about Mean Average Precision
What does mAP measure? How well a detector ranks correct detections above wrong ones, across all classes. It averages the area under each class's precision-recall curve.
What does mAP@0.5 mean? A detection counts as correct if its IoU with a ground-truth box is at least 0.5. mAP@[0.5:0.95], used by COCO, averages across IoU thresholds from 0.5 to 0.95.
How is average precision computed for one class? Sort detections by confidence, walk down the list marking true and false positives, and trace precision against recall. The area under that curve is AP.
Why is mAP preferred over accuracy? Detection has no fixed set of negatives, and accuracy ignores box placement and ranking. mAP rewards both correct localisation and sensible confidence scores.
What is the difference between VOC and COCO mAP? VOC uses a single IoU of 0.5, while COCO averages over ten thresholds and so rewards tighter boxes. The same model gets a lower score under COCO.
Does a high mAP mean the model is ready for production? Not by itself. It hides the operating threshold, per-size performance and rare-class failures, so check the precision-recall curve at the confidence you will deploy at.
From one PR curve to a single number
mAP is the standard score for spotting. It's assembled from parts that each discard a thing. This builds it from raw detections upward -- ranking, matching, precision and recall, interpolation, the class mean -- so that when a number comes back low you know which stage to look at.
example_01.pyNumPy
GT = [(0,10,10,50,50),(0,60,60,90,90),(1,20,20,60,60),(1,70,10,95,40),(2,30,30,70,70)]
DETS = [(0,12,12,52,48,0.95),(1,22,18,58,62,0.91),(0,15,15,45,45,0.88),
(2,33,28,68,72,0.80),(1,10,70,40,95,0.75),(0,58,62,88,92,0.62),
(2,60,10,80,30,0.55),(1,68,12,92,38,0.40)]
def iou(a, b):
x1, y1 = max(a[0],b[0]), max(a[1],b[1])
x2, y2 = min(a[2],b[2]), min(a[3],b[3])
inter = max(0,x2-x1)*max(0,y2-y1)
return inter / float((a[2]-a[0])*(a[3]-a[1])+(b[2]-b[0])*(b[3]-b[1])-inter)
print("%d ground-truth boxes, %d detections." % (len(GT), len(DETS)))
print("STEP 1 -- SORT BY CONFIDENCE. ranking matters: a detector that ranks")
print("badly scores badly even if it found everything.")
print()
def evaluate(thresh):
order = sorted(range(len(DETS)), key=lambda i: -DETS[i][5])
used, rows = set(), []
for rank, i in enumerate(order):
d = DETS[i]; best, best_j = 0.0, -1
for j, g in enumerate(GT):
if g[0] != d[0]: continue
v = iou(d[1:5], g[1:5])
if v > best: best, best_j = v, j
if best >= thresh and best_j not in used:
used.add(best_j); verdict, note = "TP", "gt %d" % best_j
elif best >= thresh:
verdict, note = "FP", "gt %d taken" % best_j
else:
verdict, note = "FP", "IoU %.2f" % best
rows.append((rank+1, d[5], best, verdict, note))
return rows
rows = evaluate(0.5)
print("STEP 2 -- MATCH. TP if IoU >= 0.5 with an unclaimed box:")
print("%-6s %8s %8s %6s %s" % ("rank", "conf", "IoU", "type", "why"))
for rank, conf, best, verdict, note in rows:
print("%-6d %8.2f %8.2f %6s %s" % (rank, conf, best, verdict, note))
print(" rank 3 overlaps gt 0 well but gt 0 was already claimed at rank 1.")
print(" duplicates cost you; that is why NMS runs before evaluation.")
Output
The next block walks down the ranked list accumulating precision and recall at each step.
example_02.pyNumPy
GT = [(0,10,10,50,50),(0,60,60,90,90),(1,20,20,60,60),(1,70,10,95,40),(2,30,30,70,70)]
DETS = [(0,12,12,52,48,0.95),(1,22,18,58,62,0.91),(0,15,15,45,45,0.88),
(2,33,28,68,72,0.80),(1,10,70,40,95,0.75),(0,58,62,88,92,0.62),
(2,60,10,80,30,0.55),(1,68,12,92,38,0.40)]
def iou(a, b):
x1, y1 = max(a[0],b[0]), max(a[1],b[1])
x2, y2 = min(a[2],b[2]), min(a[3],b[3])
inter = max(0,x2-x1)*max(0,y2-y1)
return inter / float((a[2]-a[0])*(a[3]-a[1])+(b[2]-b[0])*(b[3]-b[1])-inter)
def evaluate(thresh):
order = sorted(range(len(DETS)), key=lambda i: -DETS[i][5])
used, rows = set(), []
for rank, i in enumerate(order):
d = DETS[i]; best, best_j = 0.0, -1
for j, g in enumerate(GT):
if g[0] != d[0]: continue
v = iou(d[1:5], g[1:5])
if v > best: best, best_j = v, j
if best >= thresh and best_j not in used:
used.add(best_j); verdict = "TP"
else: verdict = "FP"
rows.append((rank+1, d[5], verdict))
return rows
rows = evaluate(0.5)
print("STEP 3 -- PRECISION AND RECALL after each detection:")
print(" precision = TP/(TP+FP) recall = TP/all_GT")
tp = fp = 0; P, R = [], []
print("%-6s %6s %6s %6s %10s %10s" % ("rank","type","TP","FP","prec","recall"))
for rank, conf, verdict in rows:
if verdict == "TP": tp += 1
else: fp += 1
p, r = tp/float(tp+fp), tp/float(len(GT))
P.append(p); R.append(r)
print("%-6d %6s %6d %6d %10.4f %10.4f" % (rank, verdict, tp, fp, p, r))
print(" recall only rises. precision jitters at every FP -- that is why")
print(" the curve must be interpolated before computing area under it.")
Output
The next block interpolates the precision-recall curve and computes AP as the area under it, comparing the all-points and 11-point methods.
example_03.pyNumPy
import numpy as np
P = [1.0, 1.0, 0.6667, 0.75, 0.6, 0.6667, 0.5714, 0.625]
R = [0.2, 0.4, 0.4, 0.6, 0.6, 0.8, 0.8, 1.0]
print("STEP 4 -- INTERPOLATE. replace each precision by the best at that")
print("recall or higher:")
Pi = np.array(P).copy()
for i in range(len(Pi)-2, -1, -1):
Pi[i] = max(Pi[i], Pi[i+1])
print("%-10s %10s %12s %10s" % ("recall", "prec", "interpolated", "change"))
for i in range(len(P)):
print("%-10.4f %10.4f %12.4f %10s"
% (R[i], P[i], Pi[i], "" if abs(Pi[i]-P[i])<1e-4 else "raised"))
print(" this removes dips. you would never deploy at a threshold whose")
print(" precision is beaten by a more permissive one.")
print()
print("STEP 5 -- AP = area under the interpolated curve:")
ap, prev_r = 0.0, 0.0
for i in range(len(R)):
ap += (R[i]-prev_r)*Pi[i]; prev_r = R[i]
print(" AP at IoU 0.5 = %.4f (all-points method, used by COCO)." % ap)
ap11 = 0.0
for t in np.arange(0, 1.01, 0.1):
at = [Pi[i] for i in range(len(R)) if R[i] >= t]
ap11 += (max(at) if at else 0.0) / 11.0
print(" AP, 11-point method = %.4f" % ap11)
print(" AP numbers from different papers are not comparable unless they")
print(" use the same method, threshold and dataset.")
Output
The next block sweeps the IoU threshold from 0.50 to 0.95 the way COCO does, then averages the column to get the headline COCO AP.
example_04.pyNumPy
import numpy as np
GT = [(0,10,10,50,50),(0,60,60,90,90),(1,20,20,60,60),(1,70,10,95,40),(2,30,30,70,70)]
DETS = [(0,12,12,52,48,0.95),(1,22,18,58,62,0.91),(0,15,15,45,45,0.88),
(2,33,28,68,72,0.80),(1,10,70,40,95,0.75),(0,58,62,88,92,0.62),
(2,60,10,80,30,0.55),(1,68,12,92,38,0.40)]
def iou(a, b):
x1, y1 = max(a[0],b[0]), max(a[1],b[1])
x2, y2 = min(a[2],b[2]), min(a[3],b[3])
inter = max(0,x2-x1)*max(0,y2-y1)
return inter / float((a[2]-a[0])*(a[3]-a[1])+(b[2]-b[0])*(b[3]-b[1])-inter)
def evaluate(thresh):
order = sorted(range(len(DETS)), key=lambda i: -DETS[i][5])
used, rows = set(), []
for rank, i in enumerate(order):
d = DETS[i]; best, best_j = 0.0, -1
for j, g in enumerate(GT):
if g[0] != d[0]: continue
v = iou(d[1:5], g[1:5])
if v > best: best, best_j = v, j
if best >= thresh and best_j not in used:
used.add(best_j); rows.append("TP")
else: rows.append("FP")
return rows
print("STEP 6 -- SWEEP THE THRESHOLD. COCO averages AP from 0.50 to 0.95:")
print("%-12s %8s %8s %10s" % ("IoU", "TP", "FP", "AP"))
aps = []
for t in np.arange(0.5, 0.96, 0.05):
rs = evaluate(t); ntp = rs.count("TP")
tp_, fp_, PP, RR = 0, 0, [], []
for v in rs:
if v == "TP": tp_ += 1
else: fp_ += 1
PP.append(tp_/float(tp_+fp_)); RR.append(tp_/float(len(GT)))
Q = np.array(PP)
for i in range(len(Q)-2,-1,-1): Q[i] = max(Q[i], Q[i+1])
a, pr = 0.0, 0.0
for i in range(len(RR)): a += (RR[i]-pr)*Q[i]; pr = RR[i]
aps.append(a)
print("%-12.2f %8d %8d %10.4f" % (t, ntp, len(rs)-ntp, a))
print(" COCO AP = %.4f -- gap to AP@0.5 measures localisation." % np.mean(aps))
Output
The next block shows the 'm' in mAP: the unweighted average over classes, which gives rare classes equal weight.
example_05.pyNumPy
import numpy as np
print("STEP 7 -- THE 'm'. mAP is the unweighted mean of AP over classes:")
print("%-16s %10s %10s" % ("class", "instances", "AP"))
demo = [("person", 10777, 0.62), ("car", 1918, 0.55), ("toaster", 9, 0.05)]
for name, n, a in demo:
print("%-16s %10d %10.2f" % (name, n, a))
mAP = np.mean([a for _, _, a in demo])
weighted = sum(n*a for _, n, a in demo) / sum(n for _, n, _ in demo)
print("%-16s %10s %10.4f" % ("mAP", "--", mAP))
print(" mAP is UNWEIGHTED: toaster (%d instances) counts as much as"
% demo[2][1])
print(" person (%d). weighting would give %.4f instead of %.4f."
% (demo[0][1], weighted, mAP))
print(" that is deliberate -- it stops a model looking good by acing")
print(" common classes alone -- but a single rare hard class can")
print(" dominate your headline number.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Why must a matched ground-truth box be excluded from later matches?
Only the highest-confidence prediction claims a ground-truth box; the rest become false positives. Without that rule a detector could inflate its score by firing repeatedly.
Why is a COCO mAP much lower than a VOC mAP on the same predictions?
VOC reported at IoU 0.5 alone. Averaging up to 0.95 demands nearly exact boxes, so the two numbers are different measurements and cannot be compared.
What does mAP say about which confidence threshold to deploy?
AP summarises the whole ranking. Choosing an operating point is a separate decision, made from the costs of the two kinds of error.
Cheat sheet
Mean Average Precision
Sort detections by confidence, compute precision and recall at each step, and watch the area under that curve collapse into one mAP number.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.