Why accuracy is the wrong number more often than it is the right one, and what to report instead.
Overview
Why accuracy misleads
Accuracy is the metric most people reach for first. It counts how many predictions were correct and divides by the total. That simplicity is appealing but misleading on any dataset where the classes are not evenly split.
Classification problems in practice are rarely balanced. Fraud accounts for a fraction of a percent of transactions. Disease affects a small share of screened patients. Equipment fails on a tiny share of cycles. In each case, a model that always predicts the majority class gets a high accuracy while catching none of the cases you actually care about.
The metrics that matter for imbalanced data are precision, recall and F1. Precision answers: of the cases the model flagged, how many were real. Recall answers: of the real cases, how many did the model find. F1 balances the two. Each tells you something accuracy hides.
This page starts with accuracy's failure mode, then covers precision, recall, F1 and the classification report. It also covers the averaging strategies that apply when there are more than two classes.
Fit is the proportion of guesses that were correct. It's the default that score() returns. On weighed data it's a reasonable summary. On imbalanced data it's close to worthless.
The editor above makes the case in five lines. With 99 negatives for each plus, a DummyClassifier that always predicts the majority scores 0.9867 and never predicts a single plus. It has learned nothing, it would be useless for any purpose. It beats plenty of real models on the metric.
That situation isn't exotic. Fraud, disease screening, equipment failure, click-through, churn, defect spotting. The notable class is rare in nearly each problem worth solving. Rarity is just what breaks fit. The rarer the thing you care about, the higher the score for skipping it.
So the first thing to do with any classification problem is count the classes. The second is fit a dummy. If the dummy's score is close to the model's, the model has added nothing even so of how high the number looks.
Worth knowing
Accuracy is the fraction predicted correctly, and it is meaningless when one class dominates.
Precision is how many of the flagged cases were real; recall is how many of the real cases were flagged.
F1 is their harmonic mean - a single number that punishes a low value on either side.
Always print classification_report: the per-class rows are where an ignored class shows up.
For several classes, macro averaging treats every class equally and weighted lets the big ones dominate.
DummyClassifier tells you what the metric scores for doing nothing - always the first comparison.
Classification Metrics: A Practical Guide
A model can be 99% accurate and fully useless. Which metric you report is a statement about which mistakes you're willing to make.
The 98% model that predicts nothing
A classifier that never says yes, on data where yes is rare - and the accuracy it reports.
example_01.pyscikit-learn
from sklearn.dummy import DummyClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# 99 rows of one class for every 1 of the other.
X, y = make_classification(n_samples=2000, n_features=10, weights=[0.99],
random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
random_state=0, stratify=y)
print("positives in the test set:", int(y_te.sum()), "of", len(y_te))
print()
lazy = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)
print("always predict the majority:",
round(accuracy_score(y_te, lazy.predict(X_te)), 4))
print("it never predicts a positive:", int(lazy.predict(X_te).sum()))
Output
Precision and recall, from their definitions
Two fractions with the same numerator and different denominators, which is the whole difference between them.
Two models on the same data: one flags almost nothing, the other flags almost everything.
example_03.pyscikit-learn
from sklearn.metrics import precision_score, recall_score
import numpy as np
y_true = np.array([1]*10 + [0]*90)
cautious = np.zeros(100, dtype=int)
cautious[:3] = 1 # flags 3, all correct
eager = np.zeros(100, dtype=int)
eager[:40] = 1 # flags 40, catching all 10
for label, pred in [("flags 3 cases", cautious), ("flags 40 cases", eager)]:
print("%-16s precision %.3f recall %.3f"
% (label, precision_score(y_true, pred), recall_score(y_true, pred)))
print()
print("you can have either one at the expense of the other;")
print("which you want is a question about the cost of each mistake.")
Output
classification_report, the one to print
Per-class precision, recall, f1 and support in a single call.
example_04.pyscikit-learn
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
X, y = make_classification(n_samples=2000, n_features=10, weights=[0.95],
random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
random_state=0, stratify=y)
model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print(classification_report(y_te, model.predict(X_te), digits=3))
Output
With three classes, the averaging matters
The rare class is never predicted correctly. Two of these three averages hide that completely.
example_05.pyscikit-learn
from sklearn.metrics import f1_score
import numpy as np
# three classes, very different sizes
y_true = np.array([0]*90 + [1]*8 + [2]*2)
y_pred = np.array([0]*90 + [1]*6 + [0]*2 + [0]*2)
for how in ("macro", "weighted", "micro"):
print("%-9s %.3f" % (how, f1_score(y_true, y_pred, average=how,
zero_division=0)))
print()
print("the rare class is predicted correctly zero times.")
print("only the macro average makes that visible.")
Output
balanced_accuracy, when you want one number
The average recall across classes, which the lazy model cannot game.
example_06.pyscikit-learn
from sklearn.metrics import balanced_accuracy_score, accuracy_score
import numpy as np
y_true = np.array([1]*10 + [0]*990)
majority = np.zeros(1000, dtype=int)
print("accuracy of always-say-no :",
round(accuracy_score(y_true, majority), 4))
print("balanced accuracy of the same thing:",
round(balanced_accuracy_score(y_true, majority), 4))
print()
print("balanced accuracy averages the recall of each class,")
print("so ignoring a class entirely scores 0.5 rather than 0.99.")
Output
Precision and recall
Both are fractions of the same numerator. The true positives, the cases you flagged that really were plus. And they differ in what they divide by.
Precision cuts by all you flagged. It answers: *when this model says yes, how often is it right?* Low precision means false alarms.
Recall cuts by all that was really plus. It answers: *of the cases that were really there, how many did we catch?* Low recall means misses.
The two names are unhelpful and the questions aren't. It's worth reading them as those two sentences until the meanings stop needing to be looked up.
Which one matters is decided fully by the cost of each kind of mistake. The costs are often wildly different. A cancer screen that misses cases is far worse than one that makes false alarms sent for a second test. Recall dominates. A spam filter that deletes real mail is far worse than one that lets some spam through. So Precision dominates. An email flagged wrongly is an annoyance. A tumour missed is a catastrophe. No metric knows that, and no default can.
They trade against each other
You can always have more of one by accepting less of the other. The third editor shows the extremes on the same data. A model flagging 3 cases has precision 1.000 and recall 0.300. One flagging 40 has precision 0.250 and recall 1.000.
Neither is better. They're different operating points on the same underlying model. Choosing between them is the choice the metric exists to inform.
That is why quoting one alone is misleading. A claim of "95% precision" says nothing without the recall beside it. A model that flags one obvious case and nothing else reaches it trivially. The pair together shows the action. Either alone can be manufactured.
F1 merges the pair into one number by taking their harmonic mean. This Is on purpose unforgiving. Unlike an plain mean, it stays low if either input is low. F1 of a model with precision 1.0 and recall 0.1 is 0.18, not 0.55. That property is what makes it handy as a single summary when you need to rank models. It's still a summary. Two models with the same F1 can behave very differently.
Print the report
classification_report gives precision, recall, F1 and support. The number of true instances. For each class, plus the means. It's one line and it should be the default thing you print after fitting a classifier.
The per-class rows are the point. An aggregate can look healthy while one class is being skipped fully. The row for that class shows it right away. Precision and recall both zero, with a support telling you how many cases that shows.
support deserves focus on its own. A class with 4 examples in the test set makes metrics that move in steps of 25%. A difference between two models on that class is noise. Reading the support column before believing a per-class number saves a lot of misplaced trust.
Averaging across several classes
With more than two classes, the per-class numbers have to be mixed in some way. The choice changes the answer a lot.
macro means the per-class scores, treating each class as equally key even so of size. A class with 2 samples counts as much as one with 900.
weighted means them in proportion to support, so large classes dominate.
micro pools all the guesses before finding. This For single-label classification makes it the same to fit.
The fifth editor shows how far apart they can be. Macro 0.612, weighted 0.949, micro 0.960, on guesses where the rarest class is never once spotted right. Only macro reflects that failure. Weighted and micro report the speed on the big classes and let the small one disappear.
The rule that follows: use macro when the rare classes matter. This Is often why they're notable. Use weighted when you truly care about total volume. Never report micro on imbalanced data and call it any of it other than fit.
balanced_accuracy, for a single honest number
balanced_accuracy_score means the recall of each class. It's fit with the class-size distortion removed.
Its virtue is that the lazy model can't game it. Always predicting the majority gives recall 1.0 on that class and 0.0 on the other, blending to just 0.5. Which is what "no skill" should look like. The editor above shows 0.99 against 0.5 for the same guesses.
It's the right default when you want one number for an imbalanced problem and don't want to think about precision and recall apart. It's still one number. It still hides the trade-off. It belongs alongside the report rather than instead of it.
Choosing, in practice
A short method that covers most cases.
Count the classes first. If they're roughly weighed, fit is fine and the rest of this page is optional. If they aren't, it isn't.
Fit a DummyClassifier and record its score on what metric you plan to use. That is the floor. A model that doesn't clear it convincingly hasn't earned any of it.
Decide which mistake is worse in the actual use. Say why in a sentence. If missing a case is worse, lead with recall. If a false alarm is worse, lead with precision. If they're close, F1 or weighed fit.
Print classification_report even so, and read the support column before trusting any per-class row.
And note that all of these are found from predict. This Applied a 0.5 threshold to a chance. Moving that threshold moves precision and recall in opposite directions without refitting any of it. Which is the subject of the next module. Often the cheapest improvement available.
Metrics that use the probability rather than the label
All above is found from predict. This Has already committed to a label. Two metrics use predict_proba instead. They measure a thing different. How well the model *ranks* cases, on its own of where the threshold sits.
ROC AUC is the chance that a at random chosen plus is scored higher than a at random chosen minus. 1.0 is perfect ranking, 0.5 is random. Its appeal is that it summarises each possible threshold at once. It checks models without committing to an operating point. Its weakness is that it's found across the whole range including thresholds no one would use. On heavily imbalanced data it stays flatteringly high. The huge number of true negatives dominates the false positive rate.
Mean precision, the area under the precision-recall curve, is the better choice when positives are rare. It skips true negatives fully. It can't be inflated by having a great many of them. It moves when the model's action on the class you care about changes.
The hands-on rule: report ROC AUC on roughly weighed problems, mean precision on imbalanced ones. Neither as a substitute for the metric that reflects the actual choice. A high AUC means the ranking is good. It doesn't mean any given threshold makes a handy precision and recall.
Multi-label and multi-class are different problems
Two situations get confused because both involve more than two labels.
Multi-class means each sample belongs to just one of a few classes. A flower is setosa or versicolor or virginica. All on this page applies, with the blending choice being the only addition.
Multi-label means each sample can carry a few labels at once. An article tagged both "politics" and "economics". Here y is a 2-D binary array rather than a 1-D vector, fit turns exact-set-match and is brutally strict. The blending choices gain a fourth option, samples. This Means per row rather than per class.
Confusing the two makes shape errors rather than silent wrongness. This Is fortunate. MultiLabelBinarizer is the tool for getting from a list of tag lists to the array the estimators want.
What is zero_division for? Precision is undefined when nothing was flagged. Recall when nothing was plus. The arg says what to return instead of raising - 0 is the usual choice. Seeing it fire tells you a class was never predicted.
Which metric should I optimise in a search? The one that reflects the choice. Passing scoring="f1" or scoring="balanced_accuracy" to a grid search changes which model wins. Leaving it at the default optimises fit whether or not that is what you want.
Can I weight the classes instead of changing the metric? Yes. Often you should do both. class_weight="weighed" changes what the model optimises during fitting. The metric changes what you measure afterwards.
Reporting a number somebody will act on
The metric that ends up in a slide is rarely the one that should drive the choice. Closing that gap is worth a paragraph.
Precision and recall are ratios, and ratios hide volume. "Recall of 0.6" sounds moderate. "we miss four hundred cases a month" is the same fact and provokes a different conversation. Timesing the rates back into counts. How many caught, how many missed, how many false alarms per week. Turns an abstract score into a thing a person with domain knowledge can judge.
That translation also exposes whether the model is worth deploying at all. A fraud model with precision 0.3 sounds poor until you work out that each investigation costs fifty pounds and each caught case saves two thousand, at which point flagging three cases to catch one is obviously correct. The reverse happens too. An impressive-looking model whose false alarms consume more analyst time than the catches save.
The habit worth forming is to state, alongside each metric, what it means in units of the thing the organisation cares about. It takes one extra line and it's the line that decides whether anyone uses the model.
Things to try with Classification Metrics
Run the first editor. 0.9867. Not one plus predicted. Sit with the number before reading on.
Change the balance. Set weights=[0.5] and re-run. Fit stops flattering the dummy right away.
Read the support column. In the fourth editor, note how few positives the metrics for class 1 are found from.
Break the macro mean. In the fifth editor, make the model get the rare class right once and watch macro move far more than weighted.
Questions people ask about Classification Metrics
Why is accuracy misleading? On imbalanced data a model that always predicts the majority class scores high while finding nothing. Precision and recall expose that.
What is the difference between precision and recall? Precision asks how many predicted positives were right. Recall asks how many actual positives were found.
What does F1 measure? It is the harmonic mean of precision and recall. It stays low unless both are reasonably high.
Macro, micro or weighted averaging? Macro treats every class equally, micro pools all samples, and weighted scales each class by its size. Macro is the one that notices a poor rare class.
When should I use ROC AUC versus average precision? ROC AUC suits balanced problems. Average precision is more informative when positives are rare.
What does classification_report show? Precision, recall, F1 and support for every class in one table, which is a good first look.
Where this leaves you
Fit for weighed problems only. Precision when false alarms cost, recall when misses cost, F1 or weighed fit when you need one number. classification_report printed each time so an skipped class can't hide behind an mean.
Check yourself
0 of 4
Answer without scrolling back up.
A classifier scores 99% accuracy on data that is 99% one class. What has it shown?
A DummyClassifier predicting the majority achieves it without learning anything. Fitting one is how you find out whether a score means anything.
What does recall measure?
Recall divides true positives by everything that was actually positive - it is about misses. Precision divides by everything flagged, and is about false alarms.
Why is F1 a harmonic mean rather than an ordinary one?
Precision 1.0 with recall 0.1 gives F1 of 0.18, not 0.55. An ordinary average would let one good half disguise a bad one.
Which averaging makes an ignored rare class visible?
Macro treats every class equally regardless of size. Weighted and micro are dominated by the large classes, and micro equals accuracy for single-label problems.
Cheat sheet
Classification Metrics
Choose between accuracy, precision, recall and F1 by asking which mistakes matter most, and see why 99 percent accuracy can mean nothing.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.