ROC Curve and AUC

Two rates, one curve

Sweep the threshold from 1 down to 0. At each value compute:

• True Positive Rate = TP / (TP + FN) — of the actual positives, the share you caught. Also called recall or sensitivity.
• False Positive Rate = FP / (FP + TN) — of the actual negatives, the share you wrongly flagged.

Plot TPR against FPR and you have the ROC curve. The top-left corner is perfection: catch everything, flag nothing wrongly. The diagonal is a coin flip.

Both rates are computed within a true class — TPR divides by the positives, FPR by the negatives. That detail is what makes the curve independent of how many of each you happen to have, and it is the source of both its main strength and its main trap.

What AUC actually measures

The area under that curve compresses the whole picture into one number between 0 and 1.

• 1.0 — perfect separation.
• 0.9 — strong.
• 0.7 — useful in many settings.
• 0.5 — no better than a coin.
• Below 0.5 — worse than chance, which usually means your labels or your sign convention are flipped.

There is an interpretation of AUC that is worth memorising, because it makes the number concrete: AUC is the probability that a randomly chosen positive example receives a higher score than a randomly chosen negative one. An AUC of 0.85 means that if you draw one fraudulent transaction and one legitimate one at random, the model scores the fraudulent one higher 85% of the time.

That framing makes clear what AUC is and is not. It is a measure of ranking quality. It says nothing about whether the predicted probabilities are calibrated, and nothing about which threshold you should use. A model can have excellent AUC and be badly calibrated, or have an AUC of 0.9 and still be useless at every threshold you can actually afford to operate at.

One model, every threshold

A classifier that outputs probabilities is not one classifier — it is a family of them, one for every threshold you might pick. Threshold at 0.9 and you get a cautious model; threshold at 0.1 and you get an eager one. Accuracy, precision and recall all describe a single member of that family.

An ROC curve describes the whole family at once. Sweep the threshold from 1 down to 0, and at each stop plot two numbers:

• True positive rate (recall, sensitivity) = TP / (TP + FN) — the share of real positives you caught.
• False positive rate = FP / (FP + TN) — the share of real negatives you wrongly flagged.

At a threshold of 1.0 the model flags nothing: both rates are 0, the bottom-left corner. At 0.0 it flags everything: both are 1, the top-right corner. In between, the curve traces the trade-off.

The diagonal line from corner to corner is what random guessing produces. Anything above it is better than chance; the further towards the top-left corner, the better. The top-left corner itself — catching every positive with no false alarms — is a perfect classifier.

The imbalance problem, and PR curves

Here is where ROC quietly misleads, and it happens on exactly the problems people care about most.

Take 10,000 transactions, 100 of them fraud. A model flags 500 transactions to catch 90 frauds.

• True positive rate = 90/100 = 0.90.
• False positive rate = 410/9,900 = 0.041.

Those numbers look superb, and the ROC curve will sit close to the top-left corner. But precision is 90/500 = 0.18 — four out of five alerts are false. The false positive rate is small only because the denominator, the 9,900 legitimate transactions, is enormous. ROC divides by a big number and flatters the result.

A precision–recall curve does not have this problem, because precision's denominator is the number of alerts, not the number of negatives. On imbalanced problems it tracks what a human reviewing the alerts will actually experience.

The rule of thumb: report ROC-AUC when classes are roughly balanced or when both classes matter equally, and PR-AUC when positives are rare and the cost of a false alarm is borne by a human.

One number that ignores your threshold

AUC is the probability that a random positive scores above a random negative. That is not a metaphor -- here it is, counted directly.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve, roc_auc_score

X, y = make_classification(n_samples=5000, n_features=15, n_informative=5,
                           weights=[0.8, 0.2], flip_y=0.10, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.35, random_state=0,
                                      stratify=y)

models = {
    "logistic regression": LogisticRegression(max_iter=2000),
    "random forest":       RandomForestClassifier(n_estimators=200, random_state=0),
}
probs = {}
for name, m in models.items():
    probs[name] = m.fit(Xtr, ytr).predict_proba(Xte)[:, 1]
    print("%-22s AUC %.4f" % (name, roc_auc_score(yte, probs[name])))

rng = np.random.default_rng(0)
print("%-22s AUC %.4f  <- coin flip" % ("random scores", roc_auc_score(yte, rng.random(len(yte)))))
print()

p = probs["random forest"]
pos, neg = p[yte == 1], p[yte == 0]
pairs = 200_000
a = pos[rng.integers(0, len(pos), pairs)]
b = neg[rng.integers(0, len(neg), pairs)]
print("what AUC literally means -- draw one positive and one negative at random:")
print("  the positive scored higher in %.4f of %d draws"
      % (((a > b) + 0.5 * (a == b)).mean(), pairs))
print("  roc_auc_score says              %.4f" % roc_auc_score(yte, p))
print()

fpr, tpr, thr = roc_curve(yte, p)
print("the curve is just every threshold at once:")
print("%12s %10s %10s" % ("threshold", "FPR", "TPR"))
for i in np.linspace(1, len(thr) - 2, 8).astype(int):
    print("%12.4f %10.4f %10.4f" % (thr[i], fpr[i], tpr[i]))
print()
print("AUC is the area under that curve. it never looks at a threshold, which")
print("is its strength when comparing models and its weakness when you have to")
print("ship one -- shipping means choosing a point on the curve.")

Guided tour

1. Watch the point travel. Set the Threshold slider to 0.9. Very few things are predicted positive: FPR is near zero, but so is recall, and the operating point sits in the bottom-left of the curve. Now set the Threshold slider to 0.1 and it swings to the top-right — you catch nearly every positive, at the cost of flagging most negatives.
2. Confirm what AUC ignores. Move the threshold anywhere you like and watch the AUC readout. It does not move. AUC is a property of the ranking; the threshold only chooses where you sit on the curve.
3. Make the model useless. Set the Class Separation slider to 0. The two distributions sit on top of each other, the curve collapses onto the diagonal, and AUC falls to about 0.5. There is no threshold that rescues a model that cannot rank.
4. Make it perfect. Set the Class Separation slider to 4. The distributions barely touch, the curve hugs the top-left corner, and AUC approaches 1.
5. Meet the imbalance trap. Set the Positive Rate slider to 0.02, so only 2% of the data is positive. AUC barely changes — the ranking is just as good — but watch precision collapse. Most of what you flag is now a false positive, because there are so many more negatives available to be wrongly flagged.
6. Let the metric choose. With that imbalance still set, click Jump to Best F1 and then Jump to Best TPR − FPR. They land in different places, because they are optimising different things. Neither is "correct" without knowing what an error costs you.

The imbalance trap, spelled out

Suppose 1% of a million people have a disease, and your test catches 90% of them at a 10% false positive rate. That is a respectable ROC point. In absolute numbers: 9,000 true positives, and 99,000 false positives. Over 90% of the people you flag are healthy.

The ROC curve does not show this, because FPR divides by the 990,000 negatives and 99,000 of them is only 10%. Precision divides by what you flagged, and that is where the problem appears. On heavily imbalanced problems the precision–recall curve is the more honest picture, and average precision the more honest summary.

Choosing a threshold

The curve tells you the options; it cannot tell you which to take. That depends on what each error costs:

• Youden's J (maximise TPR − FPR) is a reasonable default when the two errors matter about equally.
• Best F1 balances precision and recall, which is usually more appropriate when positives are rare.
• A fixed budget — "we can only follow up 500 cases a week" — means picking the threshold that produces 500 flags, whatever the rates work out to.
• Explicit costs, when you have them, beat all of the above: minimise the expected cost directly.

Traps worth knowing

• Reporting AUC on an imbalanced problem and stopping there. It can look excellent while the model is unusable in practice. Report precision at your operating point too.
• Comparing AUCs across different datasets. AUC depends on the difficulty of the data as much as the quality of the model. It is a fair comparison between two models on one dataset, not between one model on two.
• Treating AUC as accuracy. They are unrelated quantities. A model can have AUC 0.95 and terrible accuracy at the default threshold of 0.5, which usually means the scores need calibrating rather than the model replacing.
• Tuning the threshold on the test set. The threshold is a parameter like any other. Choose it on validation data, or the reported performance is optimistic.

Summing up

The ROC curve plots true positive rate against false positive rate across every possible threshold, which separates two questions that accuracy tangles together: how well the model ranks, and where you choose to cut. AUC summarises only the first, and equals the probability that a random positive outscores a random negative — so it is unchanged by the threshold and nearly unchanged by class imbalance. That last property is exactly why it misleads on rare-event problems, where precision collapses while AUC looks fine; check precision at your actual operating point, and pick that point from what the two kinds of error really cost.

Turning a curve into an operating point

An ROC curve does not choose a threshold for you. Three ways to make that choice, in increasing order of rigour:

Youden's J statistic picks the point that maximises TPR − FPR — the point furthest above the diagonal. It treats both error types as equally costly, which is rarely true but makes a defensible default.

A fixed constraint. "We must catch 95% of fraud" fixes the TPR, and you read the threshold off the curve at that height. Or "we can review 200 alerts a day" fixes the alert volume. Operations teams usually have one of these constraints already.

Expected cost. Assign a cost to a false negative and a false positive, compute the expected cost at each threshold, and take the minimum. This is the only method that gets the answer right when the costs are asymmetric, which they nearly always are.

from sklearn.metrics import roc_curve, roc_auc_score, precision_recall_curve, average_precision_score

probs = model.predict_proba(X_test)[:, 1]

fpr, tpr, thresholds = roc_curve(y_test, probs)
print("ROC-AUC:", roc_auc_score(y_test, probs))
print("PR-AUC :", average_precision_score(y_test, probs))

best = (tpr - fpr).argmax()          # Youden's J
print("threshold:", thresholds[best])

Note that predict() silently uses 0.5. If you have chosen a different operating point, you must apply it yourself — (probs >= chosen).astype(int) — or the threshold you carefully selected never reaches production.

Comparing models honestly

AUC is popular for model comparison because it does not depend on a threshold, so it compares the underlying ranking ability rather than a particular deployment decision. Three cautions:

A higher AUC does not guarantee a better model at your operating point. Two curves can cross: model A is better at low false positive rates, model B better at high ones. If you only ever operate in the low-FPR region, A is the better model regardless of total area. Always look at the curves, not only the numbers.

AUC differences within noise are not differences. On a test set of a few thousand rows, 0.842 versus 0.847 is nothing. Bootstrap the test set to get a confidence interval before declaring a winner.

Calibration is a separate question. If a downstream system multiplies your probability by a monetary value, ranking is not enough — check a reliability diagram and calibrate if needed. Calibration does not change AUC at all, since it is a monotonic transformation of the scores.

Questions people ask

What is a good AUC? Domain-dependent. Credit scoring lives around 0.7–0.8 and that is a mature, valuable industry. Some medical imaging tasks reach 0.99. Compare against the best existing model, not against 1.0.

Can AUC be below 0.5? Yes, and it almost always means an inverted convention — you passed the wrong column of predict_proba, or the positive label is not what you think.

Does AUC work for multi-class problems? By extension: one-vs-rest AUC per class, then macro- or weighted-averaged. Report the per-class numbers too, since averaging hides a class the model cannot rank at all.

Why is my AUC good but my accuracy poor? Because the ranking is good and the threshold is wrong. This is the most fixable problem in classification — move the threshold.

Does class imbalance change AUC? ROC-AUC is insensitive to prevalence, which is exactly why it can look good on data where a human would find the alerts useless. PR-AUC does move with prevalence and is the more honest measure there.

Should I optimise AUC directly during training? Rarely necessary. Train with a proper loss such as cross-entropy, then evaluate with AUC. Ranking-specific losses exist and are mostly used in learning-to-rank problems.

Recap in one screen

• An ROC curve shows every threshold at once, plotting recall against the false positive rate.
• AUC is the probability that a random positive outscores a random negative — a measure of ranking, not of calibration.
• 0.5 is chance, above 0.9 is strong, below 0.5 means something is inverted.
• On rare-positive problems ROC flatters the model; use precision–recall curves instead.
• The curve does not pick a threshold. Pick it from your constraints or your costs, and apply it explicitly.