ROC Curve and AUC
Drag the threshold through two overlapping score distributions and watch the operating point travel along the ROC curve.
Controls
share of the data that is positive
Score Distributions
threshold 0.50ROC Curve
AUC
At This Threshold
Overview
A receiver operating characteristic (ROC) curve shows how the true-positive and false-positive rates trade off as a classifier threshold changes. Its area summarizes ranking across thresholds, but it does not select an operating threshold or show alert volume on its own.
Two rates across thresholds
- True-positive rate (TPR) = TP / (TP + FN), also called recall.
- False-positive rate (FPR) = FP / (FP + TN), the share of actual negatives incorrectly flagged.
Plot TPR against FPR as the threshold changes to get the ROC curve. A perfect classifier reaches the top-left corner; random ranking has expected AUC 0.5. ROC AUC measures how often a randomly chosen positive receives a higher score than a randomly chosen negative, with ties conventionally receiving half credit.
Follow one operating point
Set Threshold to 0.9, then lower it toward 0.1. The operating point moves along the curve: a lower threshold usually catches more positives but also flags more negatives. The AUC readout stays the same because changing the threshold does not change the score ranking.
Why prevalence still matters
TPR and FPR use separate class totals, so their values do not directly show how many alerts a system will generate. Suppose 1% of one million cases are positive. At TPR 90% and FPR 10%, the model catches 9,000 positives but also flags 99,000 negatives. Precision is 9,000 / (9,000 + 99,000), or about 8.3%; more than nine in ten alerts are false.
This is why a precision-recall curve can be especially informative when positives are rare: it shows precision alongside recall, and its baseline reflects prevalence. ROC remains useful for examining ranking and false-positive rate. Use both in the context of the task.
AUC is not a deployment decision
Two models can have similar AUC but different behavior in the threshold range that matters. Choose an operating threshold on validation data using error costs, a required recall or precision, or review capacity. Then report the resulting confusion matrix on untouched test data. AUC also says nothing about whether a score of 0.8 corresponds to an 80% event rate; assess calibration separately.
Common mistakes with ROC Curve and AUC
- Reading AUC as accuracy: it summarizes ranking across thresholds.
- Ignoring alert volume: a low FPR can still mean many false positives when negatives are numerous.
- Assuming 0.5 AUC is the only baseline: random rankings have expected AUC 0.5; class prevalence is the relevant no-skill baseline for precision-recall.
- Choosing a threshold on the test set: use validation data for that decision.
Sources and next steps
- scikit-learn: ROC metrics explains ROC curves and AUC.
- Continue with precision-recall versus ROC, threshold tuning, and probability calibration.