When the interesting class is 2% of the data, the default model ignores it and reports 96% accuracy. Four remedies, and the cheapest is usually the best.
Overview
What goes wrong
Class imbalance is present in most real classification problems. Fraud is rare. Equipment failure is rare. Disease is rare. Click-through is rare. The class you care about is almost always the minority.
A standard classifier trained on imbalanced data will mostly predict the majority class. It gets a high accuracy by ignoring the minority entirely. The accuracy number looks good, the model is useless, and nothing in the default output warns you.
The issue is not that the algorithm is broken. The algorithm is doing exactly what it was told: minimise the overall error rate. On data with 2% positives, predicting all negative gives 98% accuracy. That is the correct optimum of the wrong objective.
Fixing imbalance requires changing the objective, changing the data, or changing the threshold. scikit-learn supports all three. class_weight="balanced" adjusts the loss function. RandomOverSampler and RandomUnderSampler adjust the data. Moving the decision threshold adjusts the trade-off between false positives and false negatives.
This page covers the symptoms, the diagnostics and the remedies. It also covers why you should check whether imbalance is actually hurting before applying a fix.
When one class is rare, the model has almost no incentive to find it. Predicting the majority for all is right 98% of the time. The loss function. Which counts each row equally. Says that is great.
The first editor shows the result: 96% fit. 3 of 49 positives found. The confusion mat makes it concrete, with 46 misses and zero false alarms. That isn't a broken fit. It's a correct optimisation of the wrong goal.
The problems compound. Fit is uninformative, as the classification-metrics module showed. A random split can leave almost no positives in the test set. The metrics found from them are noise. And the default 0.5 threshold, chosen for weighed classes, is far from where it should be.
Worth knowing
Accuracy is meaningless here - always stratify the split and report per-class metrics.
class_weight="balanced" reweights the loss so minority errors count more; it raises recall and costs precision.
Tuning the threshold needs no refitting and often beats every other remedy - try it first.
Resampling must happen inside the folds, on the training part only; doing it first puts copies on both sides.
Oversampling duplicates rows and can overfit them; undersampling throws away real data. Both are blunter than reweighting.
Imbalance only hurts when the classes are also hard to separate - check before reaching for a remedy.
Class Imbalance: A Practical Guide
The notable class is often the rare one. Four remedies exist, they aren't equally good. The first thing to check is whether you need one at all.
What the default model does
It finds three of the forty-nine positives, and reports 96% accuracy for it.
example_01.pyscikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, recall_score
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=4000, n_features=10, n_informative=4,
weights=[0.98], flip_y=0.05, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
random_state=0, stratify=y)
m = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("positives in the test set:", int(y_te.sum()), "of", len(y_te))
print("accuracy:", round(m.score(X_te, y_te), 4))
print("recall :", round(recall_score(y_te, m.predict(X_te)), 4))
print()
print("confusion matrix (rows = actual):")
print(confusion_matrix(y_te, m.predict(X_te)))
Output
class_weight, and what it costs
Recall goes up nine-fold. Precision collapses, and accuracy with it.
example_02.pyscikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score, precision_score
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=4000, n_features=10, n_informative=4,
weights=[0.98], flip_y=0.05, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
random_state=0, stratify=y)
print("%-22s %9s %9s %9s" % ("", "accuracy", "precision", "recall"))
for label, cw in [("default", None), ("class_weight=balanced", "balanced")]:
m = LogisticRegression(max_iter=1000, class_weight=cw).fit(X_tr, y_tr)
p = m.predict(X_te)
print("%-22s %9.4f %9.4f %9.4f"
% (label, m.score(X_te, y_te),
precision_score(y_te, p, zero_division=0), recall_score(y_te, p)))
Output
The same model, thresholded
No refitting, no reweighting - the model already knew, and predict() was throwing it away.
example_03.pyscikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score, precision_score
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=4000, n_features=10, n_informative=4,
weights=[0.98], flip_y=0.05, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
random_state=0, stratify=y)
m = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
proba = m.predict_proba(X_te)[:, 1]
print("the same default model, thresholded differently:")
print("%10s %10s %8s" % ("threshold", "precision", "recall"))
for t in (0.5, 0.2, 0.1, 0.05):
p = (proba >= t).astype(int)
print("%10.2f %10.4f %8.4f"
% (t, precision_score(y_te, p, zero_division=0), recall_score(y_te, p)))
Output
Resampling before the split is a leak
Random labels, so nothing is learnable. One of these two numbers says otherwise.
example_04.pyscikit-learn
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import recall_score
rng = np.random.RandomState(0)
X = rng.normal(size=(600, 8))
y = (rng.rand(600) < 0.08).astype(int) # rare, and unrelated to X
def oversample(Xa, ya, seed):
r = np.random.RandomState(seed)
minority = np.flatnonzero(ya == 1)
extra = r.choice(minority, (ya == 0).sum() - len(minority), replace=True)
idx = np.concatenate([np.arange(len(ya)), extra])
return Xa[idx], ya[idx]
# WRONG: duplicate first, then split - copies land on both sides.
Xo, yo = oversample(X, y, 0)
wrong = []
for tr, te in StratifiedKFold(5, shuffle=True, random_state=0).split(Xo, yo):
m = LogisticRegression(max_iter=1000).fit(Xo[tr], yo[tr])
wrong.append(recall_score(yo[te], m.predict(Xo[te])))
# RIGHT: split first, oversample only the training part.
right = []
for tr, te in StratifiedKFold(5, shuffle=True, random_state=0).split(X, y):
Xr, yr = oversample(X[tr], y[tr], 0)
m = LogisticRegression(max_iter=1000).fit(Xr, yr)
right.append(recall_score(y[te], m.predict(X[te])))
print("labels are random, so honest recall is about 0.5 at best")
print("oversample then split :", round(np.mean(wrong), 3))
print("split then oversample :", round(np.mean(right), 3))
Output
Three remedies, one table
On F1, the cheapest of the three wins by a wide margin.
The same 2% positives, but classes the features can actually separate - and no remedy needed.
example_06.pyscikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
# Same 2% imbalance, but the classes are genuinely separable.
X, y = make_classification(n_samples=4000, n_features=10, n_informative=6,
weights=[0.98], flip_y=0.0, class_sep=2.0,
random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
random_state=0, stratify=y)
m = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("positives:", int(y_te.sum()), "of", len(y_te))
print("recall with no remedy at all:", round(recall_score(y_te, m.predict(X_te)), 4))
print()
print("imbalance is not the problem by itself. the problem is imbalance")
print("plus classes the features cannot separate.")
Output
Check whether you have a problem
Before reaching for a remedy, find out whether imbalance is really hurting.
The last editor keeps the same 2% positives but makes the classes truly separable. The default model. No weighting, no resampling, no threshold change. Recalls 70.8%. Nothing needed fixing.
Imbalance by itself isn't the trouble. The trouble is imbalance *plus* classes the features can't set apart. When the signal is strong, a rare class is found easily. When it's weak, the rarity means there's little evidence to learn from and the majority answer is truly hard to beat.
So the first step is a stratified split, a classification_report. A DummyClassifier for check. If recall on the minority is already acceptable, stop.
The remedies, cheapest first
Tune the threshold. No refitting, no new data, no risk of overfitting duplicates. The third editor takes the *same fitted model* from 6% recall to 41% by moving one number. The fifth editor checks the remedies on F1 and the threshold wins clearly: 0.38 against 0.12 for the default and 0.11 for class weighting.
This should almost always be tried first. It often makes the others unneeded. It's also the most controllable. You pick the operating point rather than accepting what a reweighting makes.
class_weight="weighed". Weights each class inversely to its rate. A minority error costs more during fitting. Supported by most classifiers as a constructor arg, and it's one word.
The second editor shows what it does: recall rises from 0.06 to 0.55, precision falls from 1.00 to 0.06. Fit from 0.96 to 0.65. That is a genuine trade rather than an improvement. Whether it's worth it depends fully on the costs. Note that on F1 it didn't beat the default at all.
class_weight can also take a dict. This Is how you express a exact cost ratio rather than accepting the inverse-rate default.
Resample. Oversample the minority by duplicating rows, or undersample the majority by discarding them, until the classes are weighed.
Both are blunter than the options. Oversampling shows the model the same rows again and again. This Pushes storing them. Undersampling throws away real data. This Is wasteful when the majority isn't huge. SMOTE and its relatives, from the imbalanced-learn package, make synthetic minority points between existing ones instead of duplicating. Better in principle. It invents data, which on a truly rare class can invent structure that isn't there.
Collect more of the minority. Rarely available and by far the best when it's.
Resampling has to be inside the folds
This is the mistake that makes the impressive result, and it's worth its own section.
Resampling before splitting means duplicated rows land on both sides. The model then sees, in training, exact copies of rows it will be tested on. The duplicate-row leak from the leakage module, added on purpose.
The fourth editor measures it on data where the labels are random and nothing is learnable. Oversampling before the split reports recall 0.632. Splitting first and oversampling only the training part reports 0.324. Neither is impressive. The first is a fabrication.
The same rule applies to cross-validation. Here The resampling must happen inside each fold. A plain Pipeline can't do this. Pipeline steps may not change the number of rows. Which is why imbalanced-learn gives its own Pipeline that can. That is the main hands-on reason to install it.
And what you do to the training data, never resample the test data. The test set must reflect the real spread, or the metrics show a world that doesn't exist.
What to measure
Fit is out. Beyond that, the choice follows the costs, as the metrics modules argued.
Recall when missing a case is costly: disease, fraud, safety. Precision when a false alarm is costly. F1 or weighed fit for a single summary. Mean precision rather than ROC AUC for the curve summary. ROC AUC stays flatteringly high when negatives dominate. 0.825 against an mean precision of 0.134 in the thresholds module.
Whichever you pick, pass it as scoring= to any cross-validation or search. Leaving it at the default tunes for fit. This On this data means tuning towards skipping the minority.
And report counts alongside rates. "Recall 0.55" is abstract. "we catch 27 of 49 cases and raise 400 false alarms" is a thing a person can judge.
Where the rare class actually comes from
The remedies above treat imbalance as a property of the dataset. Often it's a property of how the dataset was built. That is worth checking before treating it.
The window is too short. Events that are rare per day are common per year. Widening the period at times turns a 0.5% problem into a 5% one with no other change.
The unit of analysis is wrong. Predicting whether a *transaction* is fraudulent may be a 0.1% problem while predicting whether an *account* has ever been fraudulent is a 5% one. Aggregating to the entity often makes both the imbalance and the guess problem easier. And at times it's the question the business really wanted.
The negatives were over-collected. Some datasets include each minus available and only the positives someone bothered to label. Sampling the negatives down is then not undersampling in the statistical sense. It's undoing an artefact of collection.
The meaning is too narrow. A stricter positive label means fewer positives. Loosening it. "escalated" rather than "escalated and confirmed". Can give a learnable problem whose output is still handy.
None of these is always available. All of them are worth ten minutes before spending a week on SMOTE variants. A reframing that changes the base rate helps more than any resampling scheme.
What resampling does to the probabilities
A consequence that catches people after the model is working. Resampling changes the base rate the model believes in, and so its output chances.
Train on data weighed from 2% to 50% and the model learns a world where positives are half of all. Its predicted chances are calibrated to that world, not to yours. A "0.6" from it doesn't mean a 60% chance in production. It's systematically far too high.
For ranking and for thresholding that doesn't matter. Both are rule to a monotonic shift. It matters a great deal if the chance is used as a number. Expected value, risk scores shown to people, cost maths.
class_weight has a milder version of the same effect. Threshold tuning has none at all. It leaves the fitted model untouched. One more reason it's the remedy to try first.
If you do resample and need honest chances, recalibrate afterwards on data with the real class balance. This Is what CalibratedClassifierCV on an unresampled validation set does.
How imbalanced is too imbalanced? There's no threshold. A 1% problem with a strong signal is easy and a 30% problem with a weak one is hard. Measure recall on the minority rather than counting the ratio.
Is SMOTE worth installing imbalanced-learn for? Its pipeline is. It can resample inside cross-validation folds. This The scikit-learn one can't. SMOTE itself is often no better than threshold tuning.
Does class_weight work for regression? No - there are no classes. The equal is sample_weight at fit time. This Most estimators accept.
Should I resample the test set to make the metrics readable? No. The test set must reflect the real spread, or the numbers show a world that doesn't exist.
Which models suffer most
Imbalance doesn't affect each estimator equally, and knowing the pattern saves some experimentation.
any of it cutting an mean loss feels it most right. Logistic regression, SVMs, neural networks, grad boosting. each row adds equally to the goal. The rare class adds almost nothing and the fit skips it. All of them accept class_weight or sample_weight. This Is the direct fix.
Trees and forests are affected through the split criterion. A split that isolates a handful of minority rows barely reduces impurity. It's rarely chosen. min_samples_leaf can prevent it fully. A leaf size of 20 on a class with 40 members is a severe constraint no one intended. Forests also suffer at the bootstrap stage. Here A rare class can be almost absent from some samples. class_weight="balanced_subsample" reweights per bootstrap rather than globally. This Is the version to use there.
k-NN is affected geometrically: with 98% negatives, the nearest neighbours of almost any point are minus. A minority point has to be in a very pure pocket to be found. Lowering n_neighbors helps, and weights="distance" helps more.
Naive Bayes is fairly solid. It models each class apart and the prior can simply be adjusted.
The general rule: the more a model means across all rows at once, the more the rare class is drowned out. The more a weight helps.
Can I combine the remedies? Yes. It's easy to overshoot. class_weight="weighed" plus oversampling plus a lowered threshold corrects three times for the same thing and floods you with false positives. Apply one, measure, then decide.
What about multi-class imbalance? The same ideas: stratify, use macro blending so small classes count. Pass class_weight="weighed". This Handles any number of classes.
My minority class has 30 examples in total. What now? Very little, honestly. Thirty examples can't support a model of any cost. Each metric found from a handful of test cases moves in large steps. The productive answers are collecting more, widening the meaning, or treating it as outlier spotting rather than classification - IsolationForest and OneClassSVM model the majority and flag departures from it. This Needs no minority examples at all.
Why did class_weight make my fit worse? Because it's supposed to. Reweighting buys recall on the minority with precision and fit on the majority. If fit is what you're judged on, don't reweight - and reconsider whether fit is the right judgement.
Things to try with Class Imbalance
Read the confusion mat. The first editor's model misses 46 of 49 positives and reports 96% fit.
Compare the remedies. The fifth editor's threshold row beats both others on F1 by a factor of three.
Watch the leak. In the fourth editor, the labels are random. The first number isn't a result.
Make it separable. In the last editor, lower class_sep to 0.5 and watch the need for a remedy return.
Questions people ask about Class Imbalance
Why does accuracy fail with imbalanced classes? A model that always predicts the majority class gets a high score yet finds no positives. Use recall, precision or average precision.
What is the cheapest fix? Set class_weight='balanced', which reweights the loss without changing the data.
When should I use SMOTE or oversampling? When weights are not enough. Apply resampling inside the training folds only, using an imbalanced-learn pipeline.
Does resampling change predicted probabilities? Yes. They no longer match real-world rates, so recalibrate or move the threshold.
Should I resample the test set? No. The test set must keep the real class ratio so the numbers reflect production.
Which models suffer most from imbalance? Those trained to minimise overall error, like logistic regression and plain trees. Boosting with class weights usually copes better.
Where this leaves you
Stratify the split, drop fit, and check whether the model is really struggling before doing any of it. If it's, move the threshold first, reach for class_weight second, and resample only inside the folds. Never before the split, and never on the test set.
Check yourself
0 of 4
Answer without scrolling back up.
A model reports 96% accuracy on data with 2% positives. What should you check first?
In the first editor that model found 3 of 49. Accuracy cannot distinguish it from a model that ignores the class entirely.
Which remedy needs no refitting?
The probabilities are already there. In the editor it took the same fitted model from 6% recall to 41%, and it beat the other remedies on F1.
Where must oversampling happen?
Resampling first puts duplicate rows on both sides of the split. On random labels that reported recall of 0.632 against an honest 0.324.
Is imbalance always a problem?
With the same 2% positives but separable classes, the untreated default model recalled 70.8%. Check before reaching for a remedy.
Cheat sheet
Class Imbalance
Handle datasets where the interesting class is rare, compare four remedies and measure each one with the metric that actually fits.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.