0.5 is a default, not a decision. Set what each kind of mistake costs and watch the cheapest threshold move away from it.
Overview
Where 0.5 comes from
It comes from the library. predict() calls predict_proba() and cuts at 0.5, because a default has to be a thing and 0.5 is the least any-looking choice available.
It's best under two assumptions that are rarely true together: the classes are weighed, and the two kinds of mistake cost the same. Change either and 0.5 stops being the right cut.
Threshold Tuning
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
0.5 is where predict() happens to cut. It is not derived from your problem.
The optimal threshold depends on the cost of each error and on the class balance, neither of which the model knows.
Tune the threshold on validation data, not on the test set, or you have tuned on the thing you are reporting.
A model whose scores are not calibrated still ranks correctly, so threshold tuning works even when the probabilities are wrong.
Threshold Tuning
Where 0.5 came from, why it's almost never right. How to find the number that is.
Put a price on each mistake
The two sliders set what a false minus and a false plus cost. The curve is the total cost across each threshold, found over 1,500 cases with 20% positives. The grey line marks 0.5. The orange line marks the cheapest threshold.
Start with a miss costing 20 and a false alarm costing 1 — a screening problem, where investigating a healthy person is cheap and letting a sick one through isn't. The min sits well below 0.5. The model should flag generously, because the flags are cheap and the misses aren't.
Now reverse them: a miss at 1 and a false alarm at 50 — an automated account suspension, where a wrong suspension is costly and a missed offender is dealt with later. The min moves above 0.5.
Set them equal and, with these class proportions, the min lands near 0.5 but not just on it, because the balance is 20/80 rather than 50/50.
The arithmetic
The expected cost of predicting plus on a case with predicted chance *p* is (1 - p) x cost_FP. Predicting minus costs p x cost_FN. Flagging is worth it when
(1 - p) * cost_FP < p * cost_FN
which rearranges to
p > cost_FP / (cost_FP + cost_FN)
With equal costs that is 0.5. This Is where the default comes from. With a miss ten times as costly as a false alarm it's 1/11, about 0.09 — and a default of 0.5 is then five times too strict.
That formula is worth carrying around. It gives a defensible starting threshold from two numbers a domain expert can often supply, without any tuning at all.
When you cannot price the errors
Often no one can put a number on a miss. Three usable fallbacks:
Raise F1, or F-beta if one error matters more. This is the least opinionated option and it's what most people do.
Fix an operating constraint. "The team can investigate 200 cases a day" sets the threshold right: take the top 200 scores. "Precision must be at least 80%" does the same from the other side. These are often more honest than a metric, because they show what will really happen.
Choose from the curve. Plot precision and recall against threshold and pick the point where the trade-off turns. It's a judgement, but an informed one.
Two things that must be right
Tune on validation data. Choosing the threshold on the test set and then reporting test speed is the same category of error as tuning settings there — the reported number includes the choice you made to raise it.
Don't confuse this with calibration. Threshold tuning takes the scores as given and picks a cut. Calibration changes the scores so that a score of 0.8 really means 80% of such cases are plus.
They're free. A badly calibrated model can rank just right, and if the ranking is right, a tuned threshold works fine. But if the threshold is being derived from the cost formula above, the *p* in it must mean an actual chance — and then calibration matters. Platt scaling and isotonic regression are the usual tools.
The threshold the costs imply
The best cut isn't a matter of taste. It drops out of the two error costs. Flagging a case with predicted chance p is worth it when (1 - p) · cost_FP < p · cost_FN. This Rearranges to a threshold of cost_FP / (cost_FP + cost_FN):
a miss costs
a false alarm costs
optimal threshold
20
1 (screening)
0.048
1
1 (symmetric)
0.500
1
50 (auto-suspension)
0.980
The familiar 0.5 is just the middle row. The special case where the two mistakes cost the same. Move to a screening problem. Here Missing a sick patient is twenty times worse than a needless second look, and the right cut falls to 0.048: flag almost any of it. Flags are cheap and misses aren't, and the default of 0.5 is then more than ten times too strict. Reverse the costs and it climbs to 0.98. The formula needs only two numbers a domain expert can often supply, and it gives a defensible threshold with no tuning at all — but the p in it must be an actual chance. This Is why a cost-derived threshold is just the case where calibration stops being optional.
When no one can price the errors, three fallbacks are honest. Raise F1 (or F-beta if one error matters more). This Is the least opinionated option. Fix an operating constraint — "the team can investigate 200 cases a day" sets the threshold right as the top 200 scores, and often shows reality better than any metric. Or read the precision-recall trade-off off the curve and pick the knee. Whichever you use, tune it on validation data, never the test set: choosing the threshold to raise a number and then reporting that number folds the choice into the result, just as tuning settings on the test set does.
Where it goes wrong with Threshold Tuning
Reporting metrics at 0.5 and stopping. The most common test mistake after using fit.
Tuning on the test set. Optimistic by just the amount you gained.
Retuning per fold and blending. The thresholds aren't close across folds. Tune once on a held-out validation set.
Losing it drifts. The best threshold depends on the class balance, and that moves in production. It needs rechecking, not setting once.
Questions people ask about Threshold Tuning
Why is 0.5 not always the right threshold? It assumes both kinds of error cost the same and the classes are balanced. Neither is usually true.
How do I pick a threshold? Assign a cost to false positives and false negatives and choose the threshold that minimises expected cost. If you cannot price them, set a recall or precision target.
On which data should I tune the threshold? On validation data or cross-validated predictions, not the test set. Then report the final result on test data once.
Does threshold tuning change the model? No. It changes only how scores become labels, so ranking metrics like AUC stay the same.
Do I need calibrated probabilities? For thresholds derived from costs, yes. A cost-based threshold assumes the probabilities are real, so calibrate first.
What happens when class balance shifts in production? The best threshold moves. Monitor the positive rate and re-tune when the distribution changes.
0.5 is a default, not a decision
The threshold is the one setting you can change after training, in production, in a config file. Here is what it costs and what it buys.
example_01.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, roc_auc_score, precision_recall_curve, f1_score
X, y = make_classification(n_samples=8000, n_features=15, n_informative=6,
weights=[0.93, 0.07], flip_y=0.06, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.35, random_state=0, stratify=y)
prob = LogisticRegression(max_iter=3000).fit(Xtr, ytr).predict_proba(Xte)[:, 1]
pos = (yte == 1).sum()
print("%d test rows, %d positive (%.1f%%). AUC %.4f -- model is fine."
% (len(yte), pos, 100*pos/len(yte), roc_auc_score(yte, prob)))
print("the threshold is what is broken.")
print()
print("%10s %8s %8s %10s %9s %9s" % ("threshold", "TP", "FP", "FN", "prec", "recall"))
for t in (0.50, 0.30, 0.20, 0.10, 0.07, 0.03):
tn, fp, fn, tp = confusion_matrix(yte, (prob >= t).astype(int)).ravel()
print("%10.2f %8d %8d %10d %9.4f %9.4f"
% (t, tp, fp, fn, tp/max(tp+fp,1), tp/max(tp+fn,1)))
print()
print("at 0.50 the model catches %d of %d positives." %
(confusion_matrix(yte, (prob >= 0.5).astype(int)).ravel()[3], pos))
print("0.50 is not cautious -- it is wrong when only %.1f%% of rows are positive."
% (100*pos/len(yte)))
prec, rec, thr = precision_recall_curve(yte, prob)
f1 = 2*prec*rec / np.clip(prec+rec, 1e-12, None)
best = thr[np.nanargmax(f1[:-1])]
print("best-F1 threshold: %.4f (F1 %.4f vs %.4f at 0.50)"
% (best, f1_score(yte, (prob>=best).astype(int)),
f1_score(yte, (prob>=0.5).astype(int))))
Output
F1 is one way to pick a threshold, but a cost matrix is more honest. The next block shows how the same probabilities produce three different optimal thresholds depending on the price of each mistake.
example_02.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=8000, n_features=15, n_informative=6,
weights=[0.93, 0.07], flip_y=0.06, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.35, random_state=0, stratify=y)
prob = LogisticRegression(max_iter=3000).fit(Xtr, ytr).predict_proba(Xte)[:, 1]
print("put money on the two mistakes and the answer changes:")
for fp_cost, fn_cost, label in ((1, 1, "equal cost"),
(1, 20, "miss costs 20x false alarm"),
(20, 1, "false alarm costs 20x miss")):
costs = [(t, fp_cost*((prob>=t)&(yte==0)).sum()
+fn_cost*((prob<t)&(yte==1)).sum())
for t in np.linspace(0.01, 0.95, 300)]
t_best, c_best = min(costs, key=lambda kv: kv[1])
c_def = next(c for t, c in costs if abs(t-0.5) < 0.006)
print(" %-30s best %.3f, cost %6d (at 0.50: %6d)"
% (label, t_best, c_best, c_def))
print()
print("same model, three different right answers.")
print("the threshold is where business constraints enter the system.")
print("pick it on a validation split, not the test set. re-check it")
print("when the base rate moves -- that is what invalidates it.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Where does the 0.5 default threshold come from?
A default has to be something. Change the balance or the relative cost of the two mistakes and 0.5 stops being the right cut.
If a miss costs ten times a false alarm, roughly what threshold does the cost formula give?
cost_FP / (cost_FP + cost_FN) = 1/11, about 0.09. The default of 0.5 would be about five times too strict for that problem.
How does threshold tuning differ from calibration?
They are independent. A badly calibrated model can rank perfectly and a tuned threshold still works - but deriving the threshold from the cost formula needs p to be a real probability.
Cheat sheet
Threshold Tuning
Slide the classification threshold from zero to one and watch precision, recall and cost shift. Find the cutoff that fits your problem.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.