Put a price on each mistake
The two sliders set what a false negative and a false positive cost. The curve is the total cost across every threshold, computed over 1,500 cases with 20% positives. The grey line marks 0.5; the orange line marks the cheapest threshold.
Start with a miss costing 20 and a false alarm costing 1 — a screening problem, where investigating a healthy person is cheap and letting a sick one through is not. The minimum sits well below 0.5. The model should flag generously, because the flags are cheap and the misses are not.
Now reverse them: a miss at 1 and a false alarm at 50 — an automated account suspension, where a wrong suspension is expensive and a missed offender is dealt with later. The minimum moves above 0.5.
Set them equal and, with these class proportions, the minimum lands near 0.5 but not exactly on it, because the balance is 20/80 rather than 50/50.
The arithmetic
The expected cost of predicting positive on a case with predicted probability *p* is (1 - p) x cost_FP. Predicting negative costs p x cost_FN. Flagging is worth it when
(1 - p) * cost_FP < p * cost_FN
which rearranges to
p > cost_FP / (cost_FP + cost_FN)
With equal costs that is 0.5, which is where the default comes from. With a miss ten times as expensive as a false alarm it is 1/11, about 0.09 — and a default of 0.5 is then five times too strict.
That formula is worth carrying around. It gives a defensible starting threshold from two numbers a domain expert can usually supply, without any tuning at all.
When you cannot price the errors
Often nobody can put a number on a miss. Three usable fallbacks:
Maximise F1, or F-beta if one error matters more. This is the least opinionated option and it is what most people do.
Fix an operating constraint. "The team can investigate 200 cases a day" sets the threshold directly: take the top 200 scores. "Precision must be at least 80%" does the same from the other side. These are often more honest than a metric, because they describe what will actually happen.
Choose from the curve. Plot precision and recall against threshold and pick the point where the trade-off turns. It is a judgement, but an informed one.
Two things that must be right
Tune on validation data. Choosing the threshold on the test set and then reporting test performance is the same category of error as tuning hyperparameters there — the reported number includes the choice you made to maximise it.
Do not confuse this with calibration. Threshold tuning takes the scores as given and picks a cut. Calibration changes the scores so that a score of 0.8 really means 80% of such cases are positive.
They are independent. A badly calibrated model can rank perfectly, and if the ranking is right, a tuned threshold works fine. But if the threshold is being derived from the cost formula above, the *p* in it must mean an actual probability — and then calibration matters. Platt scaling and isotonic regression are the usual tools.
Where it goes wrong
Reporting metrics at 0.5 and stopping. The most common evaluation mistake after using accuracy.
Tuning on the test set. Optimistic by exactly the amount you gained.
Retuning per fold and averaging. The thresholds are not comparable across folds. Tune once on a held-out validation set.
Forgetting it drifts. The optimal threshold depends on the class balance, and that moves in production. It needs rechecking, not setting once.