ROC Curve and AUC

Quick Context

A classifier does not really output a class. It outputs a score — a probability, usually — and something else turns that into a label by comparing it against a threshold. The confusion matrix describes one such threshold. Change the threshold and you get a different matrix from the very same model.

The ROC curve is what you get when you stop picking one and plot them all.

Two rates, one curve

Sweep the threshold from 1 down to 0. At each value compute:

• True Positive Rate = TP / (TP + FN) — of the actual positives, the share you caught. Also called recall or sensitivity.
• False Positive Rate = FP / (FP + TN) — of the actual negatives, the share you wrongly flagged.

Plot TPR against FPR and you have the ROC curve. The top-left corner is perfection: catch everything, flag nothing wrongly. The diagonal is a coin flip.

Both rates are computed within a true class — TPR divides by the positives, FPR by the negatives. That detail is what makes the curve independent of how many of each you happen to have, and it is the source of both its main strength and its main trap.

What AUC actually measures

The area under the curve has an interpretation worth memorising, because it is far more intuitive than "area":

AUC is the probability that a randomly chosen positive is scored higher than a randomly chosen negative.

So 0.5 means the model ranks no better than chance, 1.0 means every positive outranks every negative, and 0.9 means it gets the ordering right nine times out of ten. It says nothing about whether the scores are well calibrated, and nothing about which threshold you should use — only about the ranking.

Interactive Exploration Guide

1. Watch the point travel. Set the Threshold slider to 0.9. Very few things are predicted positive: FPR is near zero, but so is recall, and the operating point sits in the bottom-left of the curve. Now set the Threshold slider to 0.1 and it swings to the top-right — you catch nearly every positive, at the cost of flagging most negatives.Run this
2. Confirm what AUC ignores. Move the threshold anywhere you like and watch the AUC readout. It does not move. AUC is a property of the ranking; the threshold only chooses where you sit on the curve.
3. Make the model useless. Set the Class Separation slider to 0. The two distributions sit on top of each other, the curve collapses onto the diagonal, and AUC falls to about 0.5. There is no threshold that rescues a model that cannot rank.Run this
4. Make it perfect. Set the Class Separation slider to 4. The distributions barely touch, the curve hugs the top-left corner, and AUC approaches 1.Run this
5. Meet the imbalance trap. Set the Positive Rate slider to 0.02, so only 2% of the data is positive. AUC barely changes — the ranking is just as good — but watch precision collapse. Most of what you flag is now a false positive, because there are so many more negatives available to be wrongly flagged.Run this
6. Let the metric choose. With that imbalance still set, click Jump to Best F1 and then Jump to Best TPR − FPR. They land in different places, because they are optimising different things. Neither is "correct" without knowing what an error costs you.Run this

The imbalance trap, spelled out

Suppose 1% of a million people have a disease, and your test catches 90% of them at a 10% false positive rate. That is a respectable ROC point. In absolute numbers: 9,000 true positives, and 99,000 false positives. Over 90% of the people you flag are healthy.

The ROC curve does not show this, because FPR divides by the 990,000 negatives and 99,000 of them is only 10%. Precision divides by what you flagged, and that is where the problem appears. On heavily imbalanced problems the precision–recall curve is the more honest picture, and average precision the more honest summary.

Choosing a threshold

The curve tells you the options; it cannot tell you which to take. That depends on what each error costs:

• Youden's J (maximise TPR − FPR) is a reasonable default when the two errors matter about equally.
• Best F1 balances precision and recall, which is usually more appropriate when positives are rare.
• A fixed budget — "we can only follow up 500 cases a week" — means picking the threshold that produces 500 flags, whatever the rates work out to.
• Explicit costs, when you have them, beat all of the above: minimise the expected cost directly.

What usually goes wrong

• Reporting AUC on an imbalanced problem and stopping there. It can look excellent while the model is unusable in practice. Report precision at your operating point too.
• Comparing AUCs across different datasets. AUC depends on the difficulty of the data as much as the quality of the model. It is a fair comparison between two models on one dataset, not between one model on two.
• Treating AUC as accuracy. They are unrelated quantities. A model can have AUC 0.95 and terrible accuracy at the default threshold of 0.5, which usually means the scores need calibrating rather than the model replacing.
• Tuning the threshold on the test set. The threshold is a parameter like any other. Choose it on validation data, or the reported performance is optimistic.

Key Takeaway

The ROC curve plots true positive rate against false positive rate across every possible threshold, which separates two questions that accuracy tangles together: how well the model ranks, and where you choose to cut. AUC summarises only the first, and equals the probability that a random positive outscores a random negative — so it is unchanged by the threshold and nearly unchanged by class imbalance. That last property is exactly why it misleads on rare-event problems, where precision collapses while AUC looks fine; check precision at your actual operating point, and pick that point from what the two kinds of error really cost.