Three tasks that all colour in pixels, and differ in exactly two questions: do objects get separate ids, and does the background count.
Overview
Two questions
each segmentation task colours in pixels. What differs is the answer to two questions:
Do two objects of the same class get separate identities?
Does the background get labelled?
Switch between the three settings above and watch the two discs and the background. That is the entire taxonomy, visible in one scene.
Separate ids
Background labelled
Semantic
no
yes
Instance
yes
no
Panoptic
yes
yes
Semantic, Instance and Panoptic Segmentation
This module needs JavaScript: the
images are computed in the page rather than downloaded.
Worth knowing
Semantic: every pixel gets a class. Two adjacent cars are one region called 'car'.
Instance: every object gets its own id. The background is not labelled at all.
Panoptic: every pixel gets a class and objects keep separate ids. Both at once.
The distinction is things (countable objects) against stuff (sky, road, grass), which has no instances.
Semantic, Instance and Panoptic Segmentation
Three tasks, one scene, and the two questions that separate them.
Things and stuff
The vocab that makes this coherent is things against stuff.
Things are countable: people, cars, dogs, bottles. Asking "how many" makes sense, so instances make sense.
Stuff is uncountable: sky, road, grass, water. "How many skies" isn't a question. Stuff has extent but not instances.
Semantic segmentation treats all as stuff — it only ever assigns classes. Instance segmentation only handles things. It needs a thing to count. Panoptic segmentation covers both. This Is why it needed a name of its own.
Semantic
each pixel gets a class label. Two shared cars are one connected region labelled "car", and nothing in the output says there were two.
The standard architectures are fully convolutional: U-Net, DeepLab, SegFormer. The metric is mean IoU per class. This is the right task when what matters is area rather than count — how much of this field is diseased, which pixels are road, what fraction of the scan is tumour.
Instance
each object gets its own mask and identity, and pixels belonging to no object are left untagged.
Mask R-CNN is the canonical approach: detect boxes first, then predict a mask inside each. That sorting has a consequence. Masks can share, and a pixel can belong to two instances, because nothing forces a single answer per pixel.
This is the right task when counting matters: how many cells, how many people, which pallet is which.
Panoptic
each pixel gets just one class, and pixels belonging to things also get an instance id. Stuff regions get a class and no id.
The word means "all visible", and the point of the 2018 paper that named it was that the semantic and instance communities had been solving halves of the same problem with different metrics and different architectures.
The constraint that makes it harder than running both is just one label per pixel. Instance methods give shared masks and semantic methods skip instances, so mixing them needs resolving conflicts. Panoptic FPN and Mask2Former do it in one model instead.
Its metric, PQ, times a segmentation quality term (mean IoU over matched segments) by a spotting quality term (an F1 over whether segments were matched at all), which blocks a method from scoring well by getting the pixels roughly right while missing objects fully.
Choosing
Ask what the output is for.
Area, no counting — semantic. Land cover, tumour extent, drivable surface.
Counting or tracking individuals — instance. Cell counting, retail stock, people in a queue.
A complete scene description — panoptic. Autonomous driving needs both "where is the road" (stuff) and "which car is which" (things), and needs them steady.
PQ, worked
Panoptic quality is the one metric that spans both halves, and seeing it found shows why it resists the easy failures. It factors into segmentation quality times spotting quality: SQ is the mean IoU over *matched* segments, RQ is an F1 over whether segments were matched at all, TP / (TP + ½FP + ½FN).
Take a scene where 8 predicted segments matched ground truth at mean IoU 0.80, 1 guess matched nothing (a false plus), and 2 ground-truth objects were missed (false negatives):
term
value
SQ (mean IoU of matches)
0.80
RQ = 8 / (8 + 0.5 + 1)
0.84
PQ = SQ × RQ
0.67
The two factors guard different failures. A method that segments the pixels it finds beautifully but misses half the objects has a high SQ and a low RQ, so PQ falls. One that finds each object but traces them sloppily has the reverse. Only getting both right — the pixels *and* the object count — scores well, which is just the point of a metric built to end the split between the semantic community, who measured pixels with mIoU, and the instance community, who measured objects. It's also why a semantic mIoU and a panoptic PQ aren't close: they are answering different questions, and only one of them can tell two touching cars apart.
The taxonomy behind the metrics is two yes/no questions: do two objects of the same class get separate identities, and does the background get labelled. Semantic says no and yes — each pixel gets a class, two shared cars are one "car" region — so it's right when area matters and counting doesn't: tumour extent, drivable surface, land cover. Instance says yes and no. Each object gets its own mask, background is left untagged. Which is what counting needs, and because Mask R-CNN predicts a mask per detected box, its masks can share. Panoptic demands just one label per pixel *and* an instance id for the countable things, which is the constraint that makes it harder than running the other two and merging their outputs, since that merge needs a conflict rule.
The vocab that makes the split coherent is things against stuff: things are countable — people, cars — so instances make sense, while stuff is uncountable — sky, road — and has extent but no instances. Semantic treats all as stuff, instance handles only things, and panoptic covers both, which is just why it needed a name of its own.
Where it goes wrong
Using semantic segmentation to count. Two touching objects are one region. Post-hoc connected components will merge them.
Expecting instance segmentation to label the background. It doesn't. That isn't an oversight.
Checking mIoU with PQ. Different metrics on different tasks.
Skipping the one-label constraint. Merging separate semantic and instance outputs into a panoptic one needs a conflict rule, and the rule affects the score.
Questions people ask
What is the difference between semantic and instance segmentation? Semantic segmentation labels each pixel with a class, with no separation between objects of the same class. Instance segmentation also separates individual objects.
What is panoptic segmentation? It combines both. Countable things such as cars get instance IDs, and amorphous stuff such as sky or road gets a class label.
How is segmentation different from object detection? Detection outputs boxes, whereas segmentation outputs pixel-accurate masks. Segmentation costs more annotation effort but gives exact shapes.
Which metrics are used? Intersection over union per class, averaged as mIoU, and the Dice coefficient. Pixel accuracy is easily misled when the background dominates.
Which architectures are common? U-Net, DeepLab and FCN for semantic work, Mask R-CNN for instances, and transformer-based models such as SegFormer and Mask2Former.
Why do segmentation models use skip connections? Downsampling loses fine spatial detail. Skip connections pass high-resolution features straight to the decoder to sharpen the boundaries.
Three tasks that all look like colouring in
Semantic, instance and panoptic segmentation give pictures that look alike and answer different questions. This builds the same scene under all three, finds their metrics. Shows the exact query each one can't answer.
example_01.pyNumPy
import numpy as np
H, W = 8, 22
cls = np.zeros((H, W), int)
inst = np.zeros((H, W), int)
CLASSES = {0: "bg", 1: "person", 2: "car", 3: "road"}
cls[6:, :] = 3
cls[2:6, 2:6] = 1; inst[2:6, 2:6] = 1
cls[2:6, 7:11] = 1; inst[2:6, 7:11] = 2
cls[3:6, 14:21] = 2; inst[3:6, 14:21] = 3
def show(a, label, g):
print(" %s" % label)
for row in a:
print(" " + "".join(g.get(v, "?") for v in row))
show(cls, "SEMANTIC: one CLASS label per pixel", {0:".",1:"P",2:"C",3:"="})
print(" . bg P person C car = road")
show(inst, "INSTANCE: one OBJECT id per pixel, countable things only", {0:".",1:"1",2:"2",3:"3"})
print(" road is BLANK: it is stuff, not a countable thing.")
show(np.where(inst>0,inst,np.where(cls==3,9,0)), "PANOPTIC: every pixel gets both", {0:".",1:"1",2:"2",3:"3",9:"="})
print()
print("THE QUESTION EACH ONE CAN ANSWER:")
qs = [("what class is pixel (4, 3)?", "yes", "yes", "yes"),
("how many people are there?", "NO", "yes", "yes"),
("how much of the frame is road?", "yes", "NO", "yes"),
("which pixels belong to person B?", "NO", "yes", "yes"),
("is every pixel labelled?", "yes", "NO", "yes")]
print("%-34s %8s %8s %8s" % ("question","sem","inst","pan"))
for q in qs:
print("%-34s %8s %8s %8s" % q)
print(" pixel (4,3) is class %d (%s), instance %d." % (cls[4,3], CLASSES[cls[4,3]], inst[4,3]))
print(" semantic knows the class, not the instance.")
print(" %d pixels labelled person, but no field records how many" % int((cls==1).sum()))
print(" people that is. instance stores %d distinct ids." % len(set(inst[cls==1].tolist())))
print(" counting blobs in the semantic mask fails: two people shoulder")
print(" to shoulder form one blob -- the next section measures that.")
Output
The next block builds a prediction where the model finds both people but merges them into one blob, and scores that prediction with the semantic metric.
example_02.pyNumPy
import numpy as np
H, W = 8, 22
cls = np.zeros((H, W), int); inst = np.zeros((H, W), int)
CLASSES = {0: "bg", 1: "person", 2: "car", 3: "road"}
cls[6:, :] = 3
cls[2:6, 2:6] = 1; inst[2:6, 2:6] = 1
cls[2:6, 7:11] = 1; inst[2:6, 7:11] = 2
cls[3:6, 14:21] = 2; inst[3:6, 14:21] = 3
def show(a, label, g):
print(" %s" % label)
for row in a:
print(" " + "".join(g.get(v, "?") for v in row))
print("A PREDICTION: the model finds both people but MERGES them:")
pred_cls = cls.copy(); pred_cls[2:6, 6] = 1
pred_inst = np.zeros((H, W), int)
pred_inst[2:6, 2:11] = 1; pred_inst[3:6, 14:21] = 3
show(pred_cls, "predicted classes", {0:".",1:"P",2:"C",3:"="})
show(pred_inst, "predicted instances", {0:".",1:"1",3:"3"})
print()
print("SEMANTIC METRIC -- mean IoU, per class then averaged:")
print("%-12s %8s %8s %8s %8s" % ("class","truth","pred","inter","IoU"))
ious = []
for c in sorted(CLASSES):
t, p = (cls==c), (pred_cls==c)
inter = int((t&p).sum()); union = int((t|p).sum())
v = inter/float(union) if union else 1.0; ious.append(v)
print("%-12s %8d %8d %8d %8.4f" % (CLASSES[c],int(t.sum()),int(p.sum()),inter,v))
print("%-12s %35.4f" % ("mean IoU", float(np.mean(ious))))
print(" %.4f. the merge cost %d pixels out of %d, but semantic IoU" % (np.mean(ious), int((pred_cls==1).sum()-(cls==1).sum()), int((cls==1).sum())))
print(" is near perfect -- it asked 'which pixels are person?' and got")
print(" almost all right. it never asked how many people there were.")
Output
The next block scores the same prediction with the instance metric: match predicted instances to ground truth by IoU.
example_03.pyNumPy
import numpy as np
H, W = 8, 22
cls = np.zeros((H, W), int); inst = np.zeros((H, W), int)
cls[6:, :] = 3
cls[2:6, 2:6] = 1; inst[2:6, 2:6] = 1
cls[2:6, 7:11] = 1; inst[2:6, 7:11] = 2
cls[3:6, 14:21] = 2; inst[3:6, 14:21] = 3
pred_inst = np.zeros((H, W), int)
pred_inst[2:6, 2:11] = 1; pred_inst[3:6, 14:21] = 3
print("INSTANCE METRIC -- match predicted to true instances by IoU >= 0.5:")
true_ids, pred_ids = [1, 2, 3], [1, 3]
print("%-12s %-12s %8s %10s" % ("pred","best true","IoU","match?"))
matched = set()
for pid in pred_ids:
pm = (pred_inst == pid)
best, bid = 0.0, None
for tid in true_ids:
tm = (inst == tid)
u = int((pm | tm).sum())
v = int((pm & tm).sum()) / float(u) if u else 0.0
if v > best: best, bid = v, tid
ok = best >= 0.5 and bid not in matched
if ok: matched.add(bid)
print("%-12d %-12s %8.4f %10s" % (pid, "inst %d"%bid, best, "yes" if ok else "NO"))
print(" found %d of %d true instances." % (len(matched), len(true_ids)))
missed = [t for t in true_ids if t not in matched]
print(" missed: %s." % ", ".join("instance %d"%m for m in missed))
blob = (pred_inst == 1)
for tid in (1, 2):
tm = (inst == tid)
print(" merged blob IoU with true %d: %.4f" % (tid, int((blob&tm).sum())/float(int((blob|tm).sum()))))
print(" overlaps both, clears 0.5 against neither: BOTH people")
print(" missed AND the blob is a false positive. one merge, three errors.")
Output
The final block puts both metrics side by side for the same prediction and lists which task fits which problem.
example_04.pyNumPy
import numpy as np
H, W = 8, 22
cls = np.zeros((H, W), int); inst = np.zeros((H, W), int)
CLASSES = {0: "bg", 1: "person", 2: "car", 3: "road"}
cls[6:, :] = 3
cls[2:6, 2:6] = 1; inst[2:6, 2:6] = 1
cls[2:6, 7:11] = 1; inst[2:6, 7:11] = 2
cls[3:6, 14:21] = 2; inst[3:6, 14:21] = 3
pred_cls = cls.copy(); pred_cls[2:6, 6] = 1
pred_inst = np.zeros((H, W), int)
pred_inst[2:6, 2:11] = 1; pred_inst[3:6, 14:21] = 3
ious = []
for c in sorted(CLASSES):
t, p = (cls==c), (pred_cls==c)
inter = int((t&p).sum()); union = int((t|p).sum())
ious.append(inter/float(union) if union else 1.0)
true_ids = [1, 2, 3]; matched = set()
for pid in [1, 3]:
pm = (pred_inst==pid); best, bid = 0.0, None
for tid in true_ids:
tm = (inst==tid); u = int((pm|tm).sum())
v = int((pm&tm).sum())/float(u) if u else 0.0
if v > best: best, bid = v, tid
if best >= 0.5 and bid not in matched: matched.add(bid)
miou, irec = float(np.mean(ious)), len(matched)/float(len(true_ids))
print("%-28s %12s %12s" % ("","semantic mIoU","inst recall"))
print("%-28s %12.4f %12.4f" % ("this prediction", miou, irec))
print(" same prediction: %.2f and %.2f. neither is wrong -- they answer" % (miou, irec))
print(" different questions. '%.0f%% accurate' without naming the" % (100*miou))
print(" question tells you nothing.")
print()
print("WHICH ONE YOUR PROBLEM NEEDS:")
for task, ans in [("tumour area","semantic"),("count cells","instance"),
("drivable surface","semantic"),("track each car","instance"),
("replace sky","semantic"),("everything","panoptic")]:
print("%-28s %s" % (task, ans))
print(" they need different LABELS. a semantic dataset cannot train an")
print(" instance model: the info separating two touching people was")
print(" never recorded. more expensive than choosing the wrong architecture.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
What is the difference between semantic and panoptic segmentation?
Both label every pixel with a class. Only panoptic additionally distinguishes two adjacent cars as two objects rather than one 'car' region.
Why does instance segmentation not label the background?
Asking 'how many skies' is not a question. Instance methods need something countable, which is exactly why panoptic segmentation needed a name of its own.
What makes panoptic harder than running a semantic and an instance model together?
Mask R-CNN style methods can assign a pixel to two instances. Producing one consistent labelling requires resolving those conflicts, which is what panoptic architectures do directly.
Cheat sheet
Semantic, Instance and Panoptic Segmentation
Compare semantic, instance and panoptic segmentation on the same scene and see the two questions that separate each task from the others.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.