Select a mitigation strategy to see how model parameters adapt to catch rare minority samples. Compare the standard biased boundary with the optimized one.
Standard training minimises average loss over the dataset. When one class holds 99% of the rows, that average is dominated by it: predicting the majority everywhere already achieves very low loss, so there is little gradient pressure to learn the minority class at all.
The model is not malfunctioning. It is optimising exactly what you asked it to, and what you asked for was the wrong thing — you wanted the rare class detected, and you told it to minimise overall error, which are different objectives whenever the classes are unequal.
When 99% of your rows share one label, a model that always predicts that label scores 99%. Fixing imbalance means changing the data, the loss, or the threshold - and being honest about which metric you are reading.
Resample the data. Oversampling duplicates minority rows — simple, and risks overfitting to the few examples you have. SMOTE improves on it by synthesising new minority points along the lines between existing neighbours rather than copying. Undersampling discards majority rows, which balances the classes and throws away real information; it is reasonable when the majority class is genuinely enormous.Reweight the loss. Give minority errors a larger weight, typically inversely proportional to class frequency, so one minority mistake costs as much as ninety-nine majority ones. This changes nothing about the data and is usually the first thing to try — class_weight="balanced" in scikit-learn, or pos_weight in a PyTorch loss. Focal loss goes further by down-weighting examples the model already classifies confidently, concentrating training on the hard cases.Move the threshold. Often the model’s probabilities are fine and only the 0.5 cutoff is wrong. Lowering it trades precision for recall without retraining anything, and choosing it from a precision-recall curve is frequently the cheapest real improvement available.
The instinct on seeing an imbalanced dataset is to resample it immediately. That is usually the third-best move. Two cheaper things come first, and often one of them is enough.
Change what you measure. If you are still reading accuracy, no training change will help, because you cannot see the problem. Switch to precision, recall, F1 and PR-AUC, and look at the confusion matrix directly. Sometimes the model was already fine and only the report was broken.
Change the threshold. A model that outputs probabilities has already learned a ranking. If it ranks fraud above legitimate transactions well, the only thing wrong is that 0.5 is too high a bar for a class that makes up 0.5% of the data. Sweep the threshold, pick the point that meets your recall target, and you may be finished — at no training cost at all.
Only after those two should you change the training procedure. And when you do, weighting is the gentler intervention: it changes how much each row counts, without inventing or deleting rows.
Class weights multiply each class's contribution to the loss by a factor, typically the inverse of its frequency. With 99,500 negatives and 500 positives, each positive counts 199 times as much, and the totals balance. No data is created or destroyed.
# scikit-learn: one argument
model = RandomForestClassifier(class_weight="balanced")
model = LogisticRegression(class_weight="balanced")
# boosting libraries: the ratio directly
xgb = XGBClassifier(scale_pos_weight=len(y[y==0]) / len(y[y==1]))Undersampling throws away majority rows until the ratio improves. Fast, and it makes training much cheaper on very large datasets, but you are discarding real information. Reasonable when you have millions of majority rows and can afford to lose some.
Oversampling duplicates minority rows. Keeps all the data, but exact duplicates encourage memorisation — the model can fit those specific rows rather than the pattern.
SMOTE creates new synthetic minority points along the lines between existing minority neighbours, instead of duplicating. It works best with continuous, well-scaled features and a moderate imbalance. It struggles with categorical features (the interpolated value is not a real category), with high dimensions (the interpolation is between points that were not really neighbours), and with noisy minority points (it amplifies them).
| Approach | Data changed | Cost | Main risk |
|---|---|---|---|
| Threshold shift | None | None | None — try it first |
| Class weights | None | None | Can over-correct; still tune |
| Undersampling | Majority removed | Cheap, faster training | Information thrown away |
| Oversampling | Minority duplicated | Slower training | Memorising duplicates |
| SMOTE | Synthetic rows added | Slower training | Unrealistic synthetic points |
Resampling belongs inside the cross-validation fold, applied only to the training portion, and never to validation or test data.
Get this wrong and the failure is spectacular but invisible: SMOTE applied before splitting creates synthetic points from rows that later land in the test set. The model then sees close relatives of test rows during training, scores 0.97, and collapses in production.
The safe pattern is an imbalanced-learn pipeline, which applies the sampler only during fit:
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
from sklearn.model_selection import cross_val_score, StratifiedKFold
pipe = Pipeline([
("smote", SMOTE(random_state=0)), # training folds only
("clf", RandomForestClassifier(n_jobs=-1)),
])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(pipe, X, y, cv=cv, scoring="average_precision")StratifiedKFold matters here too: without it a fold can end up with almost no positives, and its score becomes meaningless noise that drags the average around.
Class weights, oversampling, undersampling and threshold tuning, all applied to the same imbalanced problem so you can see what each one actually costs.
Imbalance is a mismatch between the average loss you are minimising and the rare-class detection you actually want. Fix it by reweighting the loss (usually first), resampling the training data (inside the fold, never before the split), or simply moving the decision threshold. Then change the metric too — accuracy cannot show you whether any of it worked.
Below roughly one positive in a thousand, classification starts to be the wrong framing altogether, and two alternatives become worth considering.
Anomaly detection learns what normal looks like from the majority class alone and flags departures from it. Isolation Forest, One-Class SVM and autoencoder reconstruction error are the usual tools. This works when the positives are heterogeneous — many different kinds of unusual — rather than one coherent pattern.
A two-stage pipeline. A cheap, high-recall model first reduces millions of candidates to thousands, and a more expensive, high-precision model then ranks those. This is how large-scale fraud and moderation systems are actually built, and it sidesteps the problem of asking one model to be both sensitive and precise.
Whichever route you take, the evaluation discipline is the same: a test set with the real prevalence, PR-AUC rather than ROC-AUC, and metrics reported at the operating point you will actually deploy.
Does class_weight="balanced" always help? No. It sometimes over-corrects, producing a model that flags far too much. Treat the weight as a hyperparameter and tune it rather than assuming the balanced setting is optimal.
Should I resample before or after splitting? After, always, and only the training part.
Is SMOTE better than plain oversampling? Often, on continuous numeric features. On categorical or very high-dimensional data it frequently is not, and plain oversampling with strong regularisation can do better. Test both.
What about generating synthetic data with a model? GANs and similar can produce minority-class examples, and this is used in some domains. It is far more effort than SMOTE and carries the same fundamental risk: the model learns the generator's idea of the class, not the world's.
Do neural networks need special handling? Yes, and focal loss is the standard answer — it down-weights easy, well-classified examples so training focuses on the hard ones. It was designed for object detection, where background boxes outnumber objects by a thousand to one.
How do I know if it worked? Compare confusion matrices before and after at the same threshold, on the real distribution. If recall rose and precision did not collapse, it worked. If both moved and you only looked at F1, you have not learned much.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Why imbalance breaks training”?
Standard training minimises average loss over the dataset. When one class holds 99% of the rows, that average is dominated by it: predicting the majority everywhere already achieves very low loss, so there is little gradient pressure to learn the minority class at all.
What does this module say about “The three families of fix”?
Resample the data. Oversampling duplicates minority rows — simple, and risks overfitting to the few examples you have. SMOTE improves on it by synthesising new minority points along the lines between existing neighbours rather than copying. Undersampling discards majority rows, which balances the classes and throws away real information; it is reasonable when the majority class is genuinely enormous.
What does this module say about “Start by changing nothing about the data”?
The instinct on seeing an imbalanced dataset is to resample it immediately. That is usually the third-best move. Two cheaper things come first, and often one of them is enough.
Standard training minimises average loss over the dataset. When one class holds 99% of the rows, that average is dominated by it: predicting the majority everywhere already achieves very low loss, so there is little gradient pressure to learn the minority class at all.