When the interesting class is 2% of the data, the default model ignores it and reports 96% accuracy. Four remedies, and the cheapest is usually the best.
Overview
What goes wrong
When one class is rare, the model has almost no incentive to find it. Predicting the majority for everything is right 98% of the time, and the loss function — which counts every row equally — says that is excellent.
The first editor shows the result: 96% accuracy, and 3 of 49 positives found. The confusion matrix makes it concrete, with 46 misses and zero false alarms. That is not a broken fit; it is a correct optimisation of the wrong objective.
The problems compound. Accuracy is uninformative, as the classification-metrics module showed. A random split can leave almost no positives in the test set, so the metrics computed from them are noise. And the default 0.5 threshold, chosen for balanced classes, is far from where it should be.
Worth knowing
Accuracy is meaningless here - always stratify the split and report per-class metrics.
class_weight="balanced" reweights the loss so minority errors count more; it raises recall and costs precision.
Tuning the threshold needs no refitting and often beats every other remedy - try it first.
Resampling must happen inside the folds, on the training part only; doing it first puts copies on both sides.
Oversampling duplicates rows and can overfit them; undersampling throws away real data. Both are blunter than reweighting.
Imbalance only hurts when the classes are also hard to separate - check before reaching for a remedy.
Class Imbalance: A Practical Guide
The interesting class is usually the rare one. Four remedies exist, they are not equally good, and the first thing to check is whether you need one at all.
What the default model does
It finds three of the forty-nine positives, and reports 96% accuracy for it.
example_01.pyscikit-learn
Output
class_weight, and what it costs
Recall goes up nine-fold. Precision collapses, and accuracy with it.
example_02.pyscikit-learn
Output
The same model, thresholded
No refitting, no reweighting - the model already knew, and predict() was throwing it away.
example_03.pyscikit-learn
Output
Resampling before the split is a leak
Random labels, so nothing is learnable. One of these two numbers says otherwise.
example_04.pyscikit-learn
Output
Three remedies, one table
On F1, the cheapest of the three wins by a wide margin.
example_05.pyscikit-learn
Output
Imbalance alone is not the problem
The same 2% positives, but classes the features can actually separate - and no remedy needed.
example_06.pyscikit-learn
Output
Check whether you have a problem
Before reaching for a remedy, find out whether imbalance is actually hurting.
The last editor keeps the same 2% positives but makes the classes genuinely separable, and the default model — no weighting, no resampling, no threshold change — recalls 70.8%. Nothing needed fixing.
Imbalance by itself is not the difficulty. The difficulty is imbalance *plus* classes the features cannot distinguish. When the signal is strong, a rare class is found easily; when it is weak, the rarity means there is little evidence to learn from and the majority answer is genuinely hard to beat.
So the first step is a stratified split, a classification_report, and a DummyClassifier for comparison. If recall on the minority is already acceptable, stop.
The remedies, cheapest first
Tune the threshold. No refitting, no new data, no risk of overfitting duplicates. The third editor takes the *same fitted model* from 6% recall to 41% by moving one number. The fifth editor compares the remedies on F1 and the threshold wins clearly: 0.38 against 0.12 for the default and 0.11 for class weighting.
This should almost always be tried first, and it frequently makes the others unnecessary. It is also the most controllable: you pick the operating point rather than accepting whatever a reweighting produces.
class_weight="balanced". Weights each class inversely to its frequency, so a minority error costs more during fitting. Supported by most classifiers as a constructor argument, and it is one word.
The second editor shows what it does: recall rises from 0.06 to 0.55, precision falls from 1.00 to 0.06, and accuracy from 0.96 to 0.65. That is a genuine trade rather than an improvement, and whether it is worth it depends entirely on the costs. Note that on F1 it did not beat the default at all.
class_weight can also take a dictionary, which is how you express a specific cost ratio rather than accepting the inverse-frequency default.
Resample. Oversample the minority by duplicating rows, or undersample the majority by discarding them, until the classes are balanced.
Both are blunter than the alternatives. Oversampling shows the model the same rows repeatedly, which encourages memorising them. Undersampling throws away real data, which is wasteful when the majority is not enormous. SMOTE and its relatives, from the imbalanced-learn package, generate synthetic minority points between existing ones instead of duplicating — better in principle, and it invents data, which on a genuinely rare class can invent structure that is not there.
Collect more of the minority. Rarely available and by far the best when it is.
Resampling has to be inside the folds
This is the mistake that produces the impressive result, and it is worth its own section.
Resampling before splitting means duplicated rows land on both sides. The model then sees, in training, exact copies of rows it will be tested on — the duplicate-row leak from the leakage module, introduced deliberately.
The fourth editor measures it on data where the labels are random and nothing is learnable. Oversampling before the split reports recall 0.632; splitting first and oversampling only the training part reports 0.324. Neither is impressive, and the first is a fabrication.
The same rule applies to cross-validation, where the resampling must happen inside each fold. A plain Pipeline cannot do this, because pipeline steps may not change the number of rows — which is why imbalanced-learn provides its own Pipeline that can. That is the main practical reason to install it.
And whatever you do to the training data, never resample the test data. The test set must reflect the real distribution, or the metrics describe a world that does not exist.
What to measure
Accuracy is out. Beyond that, the choice follows the costs, as the metrics modules argued.
Recall when missing a case is expensive: disease, fraud, safety. Precision when a false alarm is expensive. F1 or balanced accuracy for a single summary. Average precision rather than ROC AUC for the curve summary, because ROC AUC stays flatteringly high when negatives dominate — 0.825 against an average precision of 0.134 in the thresholds module.
Whichever you pick, pass it as scoring= to any cross-validation or search. Leaving it at the default tunes for accuracy, which on this data means tuning towards ignoring the minority.
And report counts alongside rates. "Recall 0.55" is abstract; "we catch 27 of 49 cases and raise 400 false alarms" is something a person can judge.
Where the rare class actually comes from
The remedies above treat imbalance as a property of the dataset. Often it is a property of how the dataset was built, and that is worth checking before treating it.
The window is too short. Events that are rare per day are common per year. Widening the period sometimes turns a 0.5% problem into a 5% one with no other change.
The unit of analysis is wrong. Predicting whether a *transaction* is fraudulent may be a 0.1% problem while predicting whether an *account* has ever been fraudulent is a 5% one. Aggregating to the entity often makes both the imbalance and the prediction problem easier — and sometimes it is the question the business actually wanted.
The negatives were over-collected. Some datasets include every negative available and only the positives someone bothered to label. Sampling the negatives down is then not undersampling in the statistical sense; it is undoing an artefact of collection.
The definition is too narrow. A stricter positive label means fewer positives. Loosening it — "escalated" rather than "escalated and confirmed" — can produce a learnable problem whose output is still useful.
None of these is always available. All of them are worth ten minutes before spending a week on SMOTE variants, because a reframing that changes the base rate helps more than any resampling scheme.
What resampling does to the probabilities
A consequence that catches people after the model is working: resampling changes the base rate the model believes in, and therefore its output probabilities.
Train on data balanced from 2% to 50% and the model learns a world where positives are half of everything. Its predicted probabilities are calibrated to that world, not to yours, so a "0.6" from it does not mean a 60% chance in production — it is systematically far too high.
For ranking and for thresholding that does not matter, since both are invariant to a monotonic shift. It matters a great deal if the probability is used as a number: expected value, risk scores shown to people, cost calculations.
class_weight has a milder version of the same effect. Threshold tuning has none at all, because it leaves the fitted model untouched — one more reason it is the remedy to try first.
If you do resample and need honest probabilities, recalibrate afterwards on data with the real class balance, which is what CalibratedClassifierCV on an unresampled validation set does.
How imbalanced is too imbalanced? There is no threshold. A 1% problem with a strong signal is easy and a 30% problem with a weak one is hard. Measure recall on the minority rather than counting the ratio.
Is SMOTE worth installing imbalanced-learn for? Its pipeline is, because it can resample inside cross-validation folds, which the scikit-learn one cannot. SMOTE itself is often no better than threshold tuning.
Does class_weight work for regression? No - there are no classes. The equivalent is sample_weight at fit time, which most estimators accept.
Should I resample the test set to make the metrics readable? No. The test set must reflect the real distribution, or the numbers describe a world that does not exist.
Which models suffer most
Imbalance does not affect every estimator equally, and knowing the pattern saves some experimentation.
Anything minimising an average loss feels it most directly: logistic regression, SVMs, neural networks, gradient boosting. Every row contributes equally to the objective, so the rare class contributes almost nothing and the fit ignores it. All of them accept class_weight or sample_weight, which is the direct fix.
Trees and forests are affected through the split criterion. A split that isolates a handful of minority rows barely reduces impurity, so it is rarely chosen, and min_samples_leaf can prevent it entirely — a leaf size of 20 on a class with 40 members is a severe constraint nobody intended. Forests also suffer at the bootstrap stage, where a rare class can be almost absent from some samples; class_weight="balanced_subsample" reweights per bootstrap rather than globally, which is the version to use there.
k-NN is affected geometrically: with 98% negatives, the nearest neighbours of almost any point are negative, so a minority point has to be in a very pure pocket to be found. Lowering n_neighbors helps, and weights="distance" helps more.
Naive Bayes is relatively robust, because it models each class separately and the prior can simply be adjusted.
The general rule: the more a model averages across all rows at once, the more the rare class is drowned out, and the more a weight helps.
Can I combine the remedies? Yes, and it is easy to overshoot. class_weight="balanced" plus oversampling plus a lowered threshold corrects three times for the same thing and floods you with false positives. Apply one, measure, then decide.
What about multi-class imbalance? The same ideas: stratify, use macro averaging so small classes count, and pass class_weight="balanced", which handles any number of classes.
My minority class has 30 examples in total. What now? Very little, honestly. Thirty examples cannot support a model of any complexity, and every metric computed from a handful of test cases moves in large steps. The productive answers are collecting more, widening the definition, or treating it as anomaly detection rather than classification - IsolationForest and OneClassSVM model the majority and flag departures from it, which needs no minority examples at all.
Why did class_weight make my accuracy worse? Because it is supposed to. Reweighting buys recall on the minority with precision and accuracy on the majority. If accuracy is what you are judged on, do not reweight - and reconsider whether accuracy is the right judgement.
Things to try
Read the confusion matrix. The first editor's model misses 46 of 49 positives and reports 96% accuracy.
Compare the remedies. The fifth editor's threshold row beats both others on F1 by a factor of three.
Watch the leak. In the fourth editor, the labels are random. The first number is not a result.
Make it separable. In the last editor, lower class_sep to 0.5 and watch the need for a remedy return.
Where this leaves you
Stratify the split, drop accuracy, and check whether the model is actually struggling before doing anything. If it is, move the threshold first, reach for class_weight second, and resample only inside the folds — never before the split, and never on the test set.
Check yourself
0 of 4
Answer without scrolling back up.
A model reports 96% accuracy on data with 2% positives. What should you check first?
In the first editor that model found 3 of 49. Accuracy cannot distinguish it from a model that ignores the class entirely.
Which remedy needs no refitting?
The probabilities are already there. In the editor it took the same fitted model from 6% recall to 41%, and it beat the other remedies on F1.
Where must oversampling happen?
Resampling first puts duplicate rows on both sides of the split. On random labels that reported recall of 0.632 against an honest 0.324.
Is imbalance always a problem?
With the same 2% positives but separable classes, the untreated default model recalled 70.8%. Check before reaching for a remedy.
Cheat sheet
Class Imbalance
When the interesting class is 2% of the data, the default model ignores it and reports 96% accuracy. Four remedies, and the cheapest is usually the best.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.