Detectors do not invent boxes. They start from a fixed set at every position and learn how to nudge them.
Overview
Why detectors do not just predict boxes
A finder must output an unknown number of boxes at unknown spots and sizes. Neural networks are much better at making a fixed-size output than a variable-length one, and much better at making a small correction than at making a coord from nothing.
Anchors convert the hard problem into the easy one. Place a fixed set of ref boxes at each spot of a feature map, and ask the network two things per anchor: is there an object here, and how should this box be adjusted to fit it?
The output is now a fixed-size tensor, and the regression target is a small offset instead of an abs spot.
Anchor Boxes
This module needs JavaScript: the
images are computed in the page rather than downloaded.
Worth knowing
At every position in a feature map the detector places several boxes of fixed size and aspect ratio.
Each anchor is assigned by IoU against the ground truth: high is positive, low is negative, and the band between is ignored.
The network predicts an offset from its anchor, not a box. Regressing a small correction is far easier than regressing coordinates.
Anchor-free detectors predict the box directly from a point, which removes the tuning but not the assignment problem.
Anchor Boxes
The prior that turns 'find each object' into 'adjust these boxes slightly'. The set rule that trains it.
What the visualisation shows
The dashed grid is the feature map: one set of anchors per cell. The orange rectangle is a ground-truth object. Drag the cell control to move the anchor set around, and the size control to change the scale.
Three anchors are drawn at each spot, one per aspect ratio — wide, square and tall — because objects come in shapes and one square box fits few of them. Real finders use three scales as well, so nine anchors per spot is a common setup.
The readout gives the best IoU against the ground truth and what that IoU means for training.
The assignment rule
each anchor gets a label before training, decided by IoU:
IoU with the best ground-truth box
Label
Above ~0.5
positive — predict this object, regress the offset
Below ~0.3
negative — predict background, no box loss
In between
ignored — contributes nothing to the loss
The skip band exists because those anchors are truly ambiguous, and forcing them either way teaches the network a thing untrue.
The page opens on a plus anchor. Drag the size control down and the readout walks through all three labels — plus, then skipped, then minus — without the ground-truth box moving at all. That transition is where an anchor stops being handy for an object, and it's why the anchor set has to match the objects you expect.
The imbalance nobody warns you about
A finder places tens of thousands of anchors on an image. An image with three objects makes almost each one of them minus. Ratios of a thousand to one are normal.
Trained naively, the classification loss is dominated by easy background anchors and the model learns to say "background" very confidently and nothing else. Three responses became standard:
Hard minus mining — keep only the worst-scoring negatives so the ratio stays around 3:1.
Focal loss — down-weight examples the model already gets right, so the easy negatives stop dominating. This is what RetinaNet added, and it's why one-stage finders caught up with two-stage ones.
Two-stage spotting — a region proposal network filters down to a few hundred picks before classification, which sidesteps the imbalance.
Choosing the anchor set
Anchors are a prior, and a wrong prior is costly. Scales and ratios tuned for everyday photos do badly on aerial imagery. Here Objects are small and often square, or on text spotting. Here Boxes are extremely wide.
YOLOv2 added picking them by running k-means over the ground-truth box shapes in the training set, using IoU as the distance. That replaced a hand-tuned guess with a fitted one, and it's still the sensible default when a dataset's objects are unusual.
Anchor-free detectors
FCOS, CenterNet and DETR drop anchors fully. FCOS predicts, for each point inside an object, the four distances to the box edges. CenterNet predicts object centres as a heatmap. DETR predicts a fixed set of boxes right and matches them to ground truth with a Hungarian set.
They drop the anchor settings. This Is a real simplification. What they don't drop is the set problem — deciding which guess is in charge for which object — which is simply solved differently.
The assignment, by IoU
each anchor is labelled before training by its IoU with the nearest ground-truth box, and working three cases shows the bands. Anchor at [0,0,4,4] against three possible objects:
ground-truth box
IoU
label
[0,0,4,5]
0.80
positive — regress the offset
[1,1,5,5]
0.39
ignored — too ambiguous to train on
[3,3,7,7]
0.03
negative — predict background
The skip band (roughly 0.3-0.5) exists because forcing an ambiguous anchor either way teaches the network a thing untrue. The deeper problem the numbers expose is imbalance: a finder places tens of thousands of anchors, and an image with three objects makes almost all of them minus. Ratios of 1000:1 are normal, so a naively trained model just says "background" very confidently. That single fact drove three fixes — hard-minus mining to hold the ratio near 3:1, focal loss to down-weight the easy negatives (which is what let one-stage finders catch up), and two-stage proposal networks to filter first. And because anchors are a *prior*, a set tuned for photos fails on aerial or text data, which is why YOLOv2 fits them by k-means over the training boxes.
Anchor-free finders — FCOS, CenterNet, DETR — drop the ref boxes fully: FCOS predicts, for each point inside an object, the four distances to the box edges. CenterNet predicts centres as a heatmap. DETR predicts a fixed set of boxes and matches them to ground truth with a Hungarian set. They drop the anchor settings, a real simplification, but not the underlying set problem of deciding which guess owns which object. That is simply solved differently. And too many anchors is its own cost: each one is memory and work at each spot of each feature map. Past a point they share so heavily that extra ones add nothing but expense.
Where it goes wrong with Anchor Boxes
Default anchors on unusual data. Check the IoU distribution of your anchors against your ground-truth boxes before training.
Skipping the imbalance. Without focal loss or mining, the model predicts background all over and the loss looks fine.
Too many anchors. Each one costs memory and work at each spot of each feature map, and past a point they share so much that extra ones add nothing.
Questions people ask about Anchor Boxes
What is an anchor box? It is a predefined box of a fixed size and aspect ratio placed at every position on a feature map. The network predicts small offsets from each anchor instead of drawing a box from nothing.
Why use anchors instead of predicting boxes directly? Regressing a small correction to a sensible starting box is far easier to learn than regressing absolute coordinates. Anchors also let one network handle many objects at once.
How are anchors matched to ground-truth boxes? By intersection over union. An anchor above a high threshold, often 0.5 to 0.7, is positive, one below a low threshold is negative, and the ones between are ignored.
How do I choose anchor sizes and ratios? Cluster the box shapes in your training labels, as YOLO does with k-means, so anchors cover what your objects actually look like. Default COCO anchors suit general photos, not unusual datasets.
Why are there so many more negative anchors than positive ones? A typical image has a few objects and tens of thousands of anchors, so almost all of them match nothing. Hard negative mining or focal loss stops the background from drowning the loss.
Are anchor-free detectors replacing anchors? Detectors like FCOS and CenterNet predict from points or centres and skip anchors, which removes several hyperparameters. Anchor-based models remain common and work well.
Guessing boxes so the network only has to correct them
An anchor is a box the finder guesses before seeing the image. The network predicts a small correction instead of raw coords. This builds an anchor grid, measures which real objects it can and can't reach. Finds that the thing limiting coverage isn't always the thing you'd reach for first.
example_01.pyNumPy
import numpy as np
IMG = 64
def build(stride, scales, ratios):
shapes = [(s * np.sqrt(r), s / np.sqrt(r)) for s in scales for r in ratios]
n = IMG // stride
out = []
for i in range(n):
for j in range(n):
cy, cx = stride * (i + 0.5), stride * (j + 0.5)
for w, h in shapes:
out.append((cx - w/2, cy - h/2, cx + w/2, cy + h/2))
return np.array(out), shapes, n * n
def iou_many(box, boxes):
x1 = np.maximum(box[0], boxes[:, 0]); y1 = np.maximum(box[1], boxes[:, 1])
x2 = np.minimum(box[2], boxes[:, 2]); y2 = np.minimum(box[3], boxes[:, 3])
inter = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)
a = (box[2]-box[0])*(box[3]-box[1]); b = (boxes[:,2]-boxes[:,0])*(boxes[:,3]-boxes[:,1])
return inter / (a + b - inter)
A0, shapes0, nc0 = build(16, [16.0, 32.0], [0.5, 1.0, 2.0])
print("THE GRID: %dx%d image, one centre every 16 px, %d shapes each:" % (IMG, IMG, len(shapes0)))
print("%-24s %9s %9s %9s" % ("", "width", "height", "aspect"))
for (w, h), (s, r) in zip(shapes0,
[(s, r) for s in [16, 32] for r in [0.5, 1.0, 2.0]]):
print("%-24s %9.1f %9.1f %9.2f"
% ("scale %d, ratio %.1f" % (s, r), w, h, w / h))
print(" %d centres x %d shapes = %d anchors, all fixed before any image."
% (nc0, len(shapes0), len(A0)))
Output
The next block tests how well different anchor configurations cover a set of objects with varied shapes and sizes.
example_02.pyNumPy
import numpy as np
IMG = 64
def bld(st,sc,ra):
sh=[(s*np.sqrt(r),s/np.sqrt(r)) for s in sc for r in ra]; n=IMG//st
return np.array([(st*(j+.5)-w/2,st*(i+.5)-h/2,st*(j+.5)+w/2,st*(i+.5)+h/2) for i in range(n) for j in range(n) for w,h in sh]),sh,n*n
def iou_m(b,B):
i=np.clip(np.minimum(b[2],B[:,2])-np.maximum(b[0],B[:,0]),0,None)*np.clip(np.minimum(b[3],B[:,3])-np.maximum(b[1],B[:,1]),0,None)
return i/((b[2]-b[0])*(b[3]-b[1])+(B[:,2]-B[:,0])*(B[:,3]-B[:,1])-i)
OBJ=[("mid-sq",(20.,20.,50.,50.)),("wide",(10.,22.,54.,44.)),("tall",(26.,10.,42.,54.)),
("tiny",(30.,30.,38.,38.)),("6:1",(2.,30.,62.,40.))]
CFG=[("s16-6",16,[16.,32.],[.5,1.,2.]),("s8-6",8,[16.,32.],[.5,1.,2.]),
("s8-15",8,[8.,16.,32.],[.25,.5,1.,2.,4.])]
best = {}
for label,st,sc,ra in CFG:
A,_,_ = bld(st,sc,ra)
best[label] = {n: iou_m(np.array(b),A).max() for n,b in OBJ}
best[label]["__n"] = len(A)
print("%-12s %10s %10s %10s" % ("object","s16-6sh","s8-6sh","s8-15sh"))
for name,_ in OBJ:
print("%-12s %10.3f %10.3f %10.3f" % (name,best[CFG[0][0]][name],best[CFG[1][0]][name],best[CFG[2][0]][name]))
print("%-12s %10d %10d %10d" % ("anchors",best[CFG[0][0]]["__n"],best[CFG[1][0]]["__n"],best[CFG[2][0]]["__n"]))
c0,c1,c2 = CFG[0][0],CFG[1][0],CFG[2][0]
m0=[n for n,_ in OBJ if best[c0][n]>=0.5]
m1=[n for n,_ in OBJ if best[c1][n]>=0.5]
m2=[n for n,_ in OBJ if best[c2][n]>=0.5]
print(" matched at 0.5: %d, then %d, then %d of %d." % (len(m0),len(m1),len(m2),len(OBJ)))
Output
The next block analyses why a finer stride helps more than extra shapes: the first two columns use identical shapes, so all gains come from centre placement alone.
example_03.pyNumPy
import numpy as np
IMG = 64
def bld(st,sc,ra):
sh=[(s*np.sqrt(r),s/np.sqrt(r)) for s in sc for r in ra]; n=IMG//st
return np.array([(st*(j+.5)-w/2,st*(i+.5)-h/2,st*(j+.5)+w/2,st*(i+.5)+h/2) for i in range(n) for j in range(n) for w,h in sh]),sh,n*n
def iou_m(b,B):
i=np.clip(np.minimum(b[2],B[:,2])-np.maximum(b[0],B[:,0]),0,None)*np.clip(np.minimum(b[3],B[:,3])-np.maximum(b[1],B[:,1]),0,None)
return i/((b[2]-b[0])*(b[3]-b[1])+(B[:,2]-B[:,0])*(B[:,3]-B[:,1])-i)
OBJ=[("mid-sq",(20.,20.,50.,50.)),("wide",(10.,22.,54.,44.)),("tall",(26.,10.,42.,54.)),
("tiny",(30.,30.,38.,38.)),("6:1",(2.,30.,62.,40.))]
CFG=[("s16-6",16,[16.,32.],[.5,1.,2.]),("s8-6",8,[16.,32.],[.5,1.,2.]),
("s8-15",8,[8.,16.,32.],[.25,.5,1.,2.,4.])]
best = {}
for label,st,sc,ra in CFG:
A,_,_ = bld(st,sc,ra)
best[label] = {n: iou_m(np.array(b),A).max() for n,b in OBJ}
c0,c1,c2 = CFG[0][0],CFG[1][0],CFG[2][0]
gains = sorted(((best[c1][n]-best[c0][n],n) for n,_ in OBJ), reverse=True)
gain,who = gains[0]; wbox = dict(OBJ)[who]
print("SAME SHAPES, FINER GRID. %s gained %.3f from centres alone:" % (who,gain))
print(" stride 16: nearest centre %.0f px away." % min(abs((wbox[0]+wbox[2])/2-c) for c in (8,24,40,56)))
print(" stride 8: nearest centre %.0f px away." % min(abs((wbox[0]+wbox[2])/2-c) for c in range(4,64,8)))
print(" no shape can repair being in the wrong PLACE.")
newly = [n for n in [x for x,_ in OBJ if best[c2][x]>=0.5] if best[c1][n]<0.5]
print("WHERE EXTRA SHAPES HELP:")
if newly:
for n in newly:
print(" %-16s %.3f -> %.3f" % (n, best[c1][n], best[c2][n]))
print(" those needed a ratio the 6-shape set lacked.")
else:
print(" nothing new matched -- shapes not the binding constraint.")
still = [n for n,_ in OBJ if best[c2][n]<0.5]
if still:
print("STILL UNMATCHED: %s." % ", ".join(still))
print(" small objects suffer most: the grid offset is a much larger")
print(" relative error for an 8px box than a 45px one.")
print(" that is why feature pyramids use a fine stride for small objects.")
Output
The next block measures the positive-to-negative imbalance that results from a dense anchor grid, and how the network's predictions are encoded as offsets.
example_04.pyNumPy
import numpy as np
IMG = 64
def bld(st,sc,ra):
sh=[(s*np.sqrt(r),s/np.sqrt(r)) for s in sc for r in ra]; n=IMG//st
return np.array([(st*(j+.5)-w/2,st*(i+.5)-h/2,st*(j+.5)+w/2,st*(i+.5)+h/2) for i in range(n) for j in range(n) for w,h in sh]),sh,n*n
def iou_m(b,B):
i=np.clip(np.minimum(b[2],B[:,2])-np.maximum(b[0],B[:,0]),0,None)*np.clip(np.minimum(b[3],B[:,3])-np.maximum(b[1],B[:,1]),0,None)
return i/((b[2]-b[0])*(b[3]-b[1])+(B[:,2]-B[:,0])*(B[:,3]-B[:,1])-i)
OBJ=[("mid-sq",(20.,20.,50.,50.)),("wide",(10.,22.,54.,44.)),("tall",(26.,10.,42.,54.)),
("tiny",(30.,30.,38.,38.)),("6:1",(2.,30.,62.,40.))]
CFG=[("s16-6",16,[16.,32.],[.5,1.,2.]),("s8-6",8,[16.,32.],[.5,1.,2.]),
("s8-15",8,[8.,16.,32.],[.25,.5,1.,2.,4.])]
print("THE COST. every non-positive anchor is a negative:")
print("%-18s %8s %8s %10s" % ("config","anchors","pos","ratio"))
for label,st,sc,ra in CFG:
A,_,_ = bld(st,sc,ra)
pos = sum(int((iou_m(np.array(b),A)>=0.5).sum()) for _,b in OBJ)
print("%-18s %8d %8d %10s" % (label,len(A),pos,"1:%.0f"%(len(A)/max(pos,1))))
print(" standard repairs: hard neg mining, 1:3 sampling, focal loss.")
print()
print("WHAT THE NETWORK PREDICTS: offsets from its anchor:")
anchor=(16.,16.,48.,48.); truth=(20.,24.,52.,40.)
aw,ah = anchor[2]-anchor[0],anchor[3]-anchor[1]
acx,acy = anchor[0]+aw/2,anchor[1]+ah/2
gw,gh = truth[2]-truth[0],truth[3]-truth[1]
gcx,gcy = truth[0]+gw/2,truth[1]+gh/2
t = ((gcx-acx)/aw,(gcy-acy)/ah,np.log(gw/aw),np.log(gh/ah))
print(" anchor: %s truth: %s" % (str(anchor),str(truth)))
print(" targets: tx %.4f ty %.4f tw %.4f th %.4f" % t)
print(" tx,ty shift in anchor widths; tw,th are log size ratios.")
bcx,bcy = t[0]*aw+acx,t[1]*ah+acy; bw,bh = np.exp(t[2])*aw,np.exp(t[3])*ah
rec = (bcx-bw/2,bcy-bh/2,bcx+bw/2,bcy+bh/2)
print(" decoded: (%.1f,%.1f,%.1f,%.1f), error %.1e -- lossless." % (rec+(max(abs(a-b) for a,b in zip(rec,truth)),)))
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does the network predict for each anchor?
Regressing a small correction to a known reference box is far easier than producing coordinates from nothing, and it makes the output a fixed-size tensor.
Why do anchors with middling IoU get ignored rather than labelled?
An anchor at IoU 0.4 neither clearly covers the object nor clearly misses it. The ignore band keeps that ambiguity out of the loss.
Why does focal loss exist?
It down-weights examples the model already classifies correctly, which is what let one-stage detectors match two-stage ones.
Cheat sheet
Anchor Boxes
See how anchor boxes turn object detection into a small regression, and follow the IoU-based rule that assigns each ground truth to a box.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.