A model that says 0.8 should be right 80% of the time. Most are not, and AUC will never tell you.
Overview
Two different questions
Discrimination asks whether the model ranks positives above negatives. AUC and ROC measure this.
Calibration asks whether the numbers mean what they say. Of all the cases where the model said 0.8, were about 80% really plus?
A model can be great at one and terrible at the other, and this isn't an edge case — it's the normal situation.
Probability Calibration
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
A reliability diagram bins predictions by confidence and plots the observed frequency in each bin.
Perfect calibration is the diagonal. Below it means over-confident; above it means under-confident.
ECE is the average gap between confidence and observed frequency, weighted by bin size.
Calibration and ranking are independent. The slider changes calibration dramatically and leaves AUC untouched.
Probability Calibration
Whether a model's trust means any of it. Why each ranking metric is blind to it.
The slider proves it
Drag the temperature control and watch the curve deform. That control performs temperature scaling: it divides the logit by a constant before the sigmoid. Cutting by a number less than one sharpens each score toward 0 or 1. Cutting by a number greater than one flattens them toward 0.5.
Critically, this is a monotonic change. It never changes the order of any two guesses. each ranking metric — AUC, mean precision, any of it based on sorting. Is the same at each setting of that slider, while the trust curve swings from badly too sure to badly too shy.
Which is the point. If you only ever look at AUC, you've no info about calibration at all.
Reading the diagram
guesses are binned by trust, and each bin plots mean trust against seen rate.
On the diagonal: calibrated.
Below the diagonal: over-confident. The model says 0.9 and is right 70% of the time. This is the common failure, and the damaging one.
Above the diagonal: under-confident. The model hedges when it should not.
ECE — expected calibration error — is the mean vertical distance, weighted by how many points fall in each bin. It's the number in the readout, and it's a summary that can hide compensating errors: a curve that is above the main-line in one region and below in another can report a small ECE. Look at the curve, not only the number.
Why models come out miscalibrated
Modern neural networks are too sure, and increasingly so as they get larger. Training to cut cross-entropy on data it can fit just right pushes outputs toward 0 and 1 long after the fit has stopped raising.
Support vector machines give distances from a hyperplane, not chances, and passing them through a sigmoid doesn't make them chances.
Naive Bayes is famously too sure because its freedom assumption times linked evidence as though it were free.
Random forests tend to be too shy at the extremes, because blending many trees pulls guesses toward the middle.
Logistic regression, fitted on well-given features, is often close to calibrated — it optimises just this.
Fixing it
All three methods fit a correction on a held-out set, never on the training data.
Platt scaling. Fit a logistic regression to map scores to chances. One or two params, works on small validation sets, assumes the distortion is sigmoid-shaped.
Temperature scaling. The single-param version, and the standard for neural networks. Because it changes no rankings, fit and AUC are ensured untouched — a rare free lunch.
Isotonic regression. Fits any monotonic mapping. More flexible, and needs a lot more data or it overfits the validation set.
When it matters
When a chance feeds a choice with costs. Expected value is chance times payoff, so a wrong chance is a wrong choice even when the ranking is perfect.
Medical and risk contexts. Here "30% chance" is communicated to a person.
When scores are mixed across models or over time.
When a threshold is set from a target rate.Choosing a threshold from calibrated chances gives the rate you asked for. From uncalibrated scores it doesn't.
If all you do is rank. Show the top 100 results — calibration is irrelevant.
Expected calibration error, worked
ECE puts a single number on "do the confidences mean any of it". Bin the guesses by stated trust, and in each bin compare the mean trust against the fraction really plus. ECE is the mean gap, weighted by bin size:
bin
mean confidence
observed accuracy
gap
high
0.90
0.70
0.20
mid
0.70
0.65
0.05
low
0.55
0.55
0.00
With 100 points in each bin the ECE is (0.20 + 0.05 + 0.00)/3 = 0.083 — the model is eight points miscalibrated on average, driven almost fully by the top bin, where it says 0.90 and is right 70% of the time. That over-trust is the common and damaging failure, and note what it does *not* touch: each one of those guesses could still rank right. So AUC and mean precision are unchanged. Temperature scaling — cutting the logit by a constant fitted on held-out data — pulls the top bin down toward its true fit without reordering any of it. This Is why it can fix calibration for free while leaving each ranking metric the same. The one caution is that ECE is an mean and can hide compensating errors, so a curve above the main-line in one region and below in another can post a small ECE. Read the trust curve, not only the number.
Why models miscalibrate is worth knowing, because it predicts which fix to reach for. Modern neural networks are too sure and increasingly so as they grow, because cutting cross-entropy on data they can fit just right keeps pushing outputs toward 0 and 1 long after fit has stopped raising. Temperature scaling is the standard remedy. Naive Bayes is too sure because its freedom assumption times linked evidence as if it were free. Random forests are often *under*-confident at the extremes, because blending many trees pulls guesses toward the middle. Logistic regression on well-given features tends to come out close to calibrated. That is just what it optimises.
Where it goes wrong with Probability Calibration
Calibrating on the training set. It will look perfect and generalise nothing.
Isotonic regression on a small validation set. It overfits.
given a good AUC implies good chances. The slider on this page exists to disprove just that.
Recalibrating without re-checking after data drift. A calibration fitted last year shows last year's spread.
Questions people ask about Probability Calibration
What does calibrated mean? Among predictions of 0.8, about 80 percent are positive. The output can be read as a real probability.
Can a model be accurate and miscalibrated? Yes. Accuracy depends on the ranking and the threshold, while calibration depends on whether the number matches the actual frequency.
How do I check calibration? Plot a reliability diagram, binning predictions and comparing the mean prediction with the observed rate. Brier score and expected calibration error give single numbers.
Platt scaling or isotonic regression? Platt scaling fits a sigmoid and works with small data. Isotonic is more flexible but can overfit below roughly a thousand samples.
Which models are usually miscalibrated? Random forests tend to be under-confident near 0 and 1, boosting and SVMs are distorted, and naive Bayes is often over-confident. Logistic regression is usually close.
When does calibration matter? When you use the probability itself, for example expected cost, risk scores or combining with other estimates. If you only rank or classify, it matters less.
When 0.9 does not mean 90%
A model can rank just right and still lie about its trust. This measures the gap, then closes it two different ways.
example_01.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.calibration import calibration_curve
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, brier_score_loss, log_loss
X, y = make_classification(n_samples=12000, n_features=15, n_informative=6,
flip_y=0.12, random_state=0)
Xtr, Xrest, ytr, yrest = train_test_split(X, y, test_size=0.5, random_state=0)
_, Xte, _, yte = train_test_split(Xrest, yrest, test_size=0.5, random_state=0)
models = {"logistic regression": LogisticRegression(max_iter=3000),
"random forest": RandomForestClassifier(n_estimators=200, random_state=0),
"naive bayes": GaussianNB(),
"SVM (rbf)": SVC(probability=True, random_state=0)}
print("%-22s %9s %9s %10s" % ("", "ROC AUC", "Brier", "log loss"))
probs = {}
for name, m in models.items():
probs[name] = m.fit(Xtr, ytr).predict_proba(Xte)[:, 1]
print("%-22s %9.4f %9.4f %10.4f"
% (name, roc_auc_score(yte, probs[name]),
brier_score_loss(yte, probs[name]), log_loss(yte, probs[name])))
print()
print("ROC AUC only cares about the ORDER. Brier and log loss care about")
print("the numbers themselves.")
print()
print("the reliability table -- group predictions by confidence:")
for name in ("logistic regression", "naive bayes"):
print(" %s" % name)
frac, mean_pred = calibration_curve(yte, probs[name], n_bins=8, strategy="quantile")
print(" %12s %14s %10s" % ("said", "actually was", "error"))
for mp, fp in zip(mean_pred, frac):
print(" %12.4f %14.4f %+10.4f" % (mp, fp, fp - mp))
print()
print("a perfectly calibrated model has zeros in that last column.")
Output
The next block fixes the miscalibration two different ways and shows the reliability table after.
example_02.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.naive_bayes import GaussianNB
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, brier_score_loss
X, y = make_classification(n_samples=12000, n_features=15, n_informative=6,
flip_y=0.12, random_state=0)
Xtr, Xrest, ytr, yrest = train_test_split(X, y, test_size=0.5, random_state=0)
Xcal, Xte, ycal, yte = train_test_split(Xrest, yrest, test_size=0.5, random_state=0)
raw = GaussianNB().fit(Xtr, ytr)
raw_p = raw.predict_proba(Xte)[:, 1]
print("two ways to fix it, both on a held-out calibration split:")
print(" %-26s Brier %.4f" % ("naive bayes, raw", brier_score_loss(yte, raw_p)))
for method in ("sigmoid", "isotonic"):
cal = CalibratedClassifierCV(raw, method=method, cv="prefit")
cal.fit(Xcal, ycal)
p = cal.predict_proba(Xte)[:, 1]
print(" %-26s Brier %.4f AUC %.4f"
% ("naive bayes + " + method, brier_score_loss(yte, p),
roc_auc_score(yte, p)))
print()
print("reliability table after sigmoid calibration:")
cal = CalibratedClassifierCV(raw, method="sigmoid", cv="prefit").fit(Xcal, ycal)
pc = cal.predict_proba(Xte)[:, 1]
frac, mean_pred = calibration_curve(yte, pc, n_bins=8, strategy="quantile")
print(" %12s %14s %10s" % ("said", "actually was", "error"))
for mp, fp in zip(mean_pred, frac):
print(" %12.4f %14.4f %+10.4f" % (mp, fp, fp - mp))
f0, m0 = calibration_curve(yte, raw_p, n_bins=8, strategy="quantile")
print(" worst error: %.4f before, %.4f after"
% (np.abs(f0 - m0).max(), np.abs(frac - mean_pred).max()))
print()
print("sigmoid (Platt) fits a one-parameter squash -- needs little data.")
print("isotonic fits any increasing function, more prone to overfit.")
print("the AUC barely moves: calibration is monotone, so it relabels")
print("confidence without changing the ranking.")
print()
print("this matters when the number leaves the model: thresholds, risk")
print("scores, expected-value calculations. argmax users can ignore it.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Why is AUC unchanged as the temperature slider moves?
Every ranking metric is identical at every setting, while calibration swings from badly over-confident to badly under-confident. Looking only at AUC tells you nothing about calibration.
A reliability curve sits below the diagonal. The model is:
It says 0.9 and is right 70% of the time. This is the common failure mode for modern neural networks, and the damaging one when probabilities feed decisions.
When does calibration not matter?
It matters whenever a probability feeds a decision with costs, because expected value is probability times payoff, so a wrong probability is a wrong decision.
Cheat sheet
Probability Calibration
Check whether predicted probabilities match observed frequencies. Apply Platt scaling and isotonic regression to close the gap.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.