Adjust the class distribution to see why Accuracy is a dangerously misleading metric when dealing with imbalanced datasets like fraud detection or rare diseases.
Imagine 100,000 transactions, 500 of which are fraudulent. A model that predicts “legitimate” every time gets 99.5% accuracy and catches no fraud. The score is correct as a fraction of all predictions, but it says little about the rare class the system is meant to find.
This is the accuracy paradox: high accuracy can coexist with poor performance on the minority class. It does not mean accuracy is always useless; it means the class distribution and decision costs matter when interpreting it.
No metric is universally best. A fraud team with limited review capacity may prioritize precision; a safety screen may accept more false alarms to improve recall. Report the confusion matrix, class prevalence, selected threshold, and the metric tied to the decision.
Adjust Class 0 (Normal) % and Data Overlap in the visualizer. As normal examples become more common, a model can achieve high overall accuracy without finding many anomalies. Increasing overlap makes the classes harder to separate even if their proportions stay fixed. These controls vary two different properties: prevalence and separability.
Suppose the 500 positive cases receive model scores. At one threshold, the system might raise 610 alerts and catch 280 positives: 330 alerts are false and 220 positives are missed. Lowering the threshold to raise 2,400 alerts and catch 420 positives increases recall from 56% to 84%, but precision falls from about 46% to 18%. The model’s ranking did not change; the operating point did.
These counts are illustrative. Estimate them on validation data and choose the threshold using error costs or real review capacity. Keep the final test set out of those choices.
About to run: set Class 0 (Normal) % to its maximum (99). Before it does — what happens to the readout?
Committing to an answer first is the point — the reveal runs the experiment on the visualisation above and reads the real value back, so nothing here is scripted.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Why accuracy can hide failure”?
What does this module say about “Read beyond the overall score”?
What does this module say about “Explore prevalence and overlap”?
See how a model scores 99% accuracy while missing every fraud case, and why imbalance breaks your intuition about metrics.