Fit the scaler before the split and the accuracy goes up. Nothing got better; the number just stopped meaning anything.
Overview
What it is
Leakage is any route by which info about the data you're testing on reaches the model before the test. The model then answers a question it was quietly shown the answer to, and the score you report measures nothing you can act on.
It has no error message. Nothing crashes, nothing warns, and the only symptom is that the number is better than you expected. Which isn't a symptom anyone investigates.
Data Leakage
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
Leakage is any path by which information about the evaluation data reaches the model before it is evaluated.
It never raises an error. The only symptom is a score that is better than it should be, which nobody investigates.
The commonest form is fitting a preprocessing step on the whole dataset. The mean and standard deviation are learned parameters.
The second commonest is a feature that could not exist at prediction time — a field filled in after the outcome was known.
Data Leakage
The mistake that makes your model look better and your results worthless.
Watch it happen, on data with nothing in it
The dataset behind the chart is 300 rows and a few hundred columns of pure noise. The labels are assigned on its own of each column. There's no signal whatsoever, and the only honest score is 50%.
Set the control to Once, dataset. Five columns are picked by correlating each one with the label across all 300 rows, and only then is the data cross-validated. The reported fit is well above 50%, on data that holds nothing.
Now set it to Inside each fold. The pick is redone within each fold's training rows, and the reported fit falls back to chance — slightly below it, which is worth a sentence of its own further down.
Drag the number of columns up. The inflation grows: 40 pick columns buy a few points, 600 buy a lot more. With more columns to search there's always some that correlate with the labels by chance, and the pick was allowed to see the labels of the rows it was later tested on.
Nothing about the model changed between those two settings. The only difference is whether the rows being scored had a say in which columns were used.
Why the honest number lands below 50%
The correct setting reports a thing like 45%, not 50%, and that isn't a bug in the demo.
A column is chosen inside a fold because it happened to separate the classes in those training rows. On pure noise that gap is fully accidental, and an accident large enough to win a search over hundreds of columns is an extreme one. Extremes regress: on the held-out rows, that column is as likely to lean the other way as this way, and the centroid built from the training rows then points slightly the wrong direction.
So selecting features on noise doesn't merely fail to help. It can actively mislead, and a model built on selected noise can score worse than one that skips the data fully. That is worth knowing on its own, and it's the strongest available arg for checking whether a feature-pick step is earning its place.
Why a scaler counts as a model
The instinct that makes this feel harmless is that scaling is "just prep". But StandardScaler has params — a mean and a standard deviation — and it *learns* them from data. any of it that learns params from data must learn them from training data only.
The same arg applies to each step that looks at the data as a whole:
imputing missing values with a column mean
selecting the top *k* features by link with the target
fitting a PCA rotation
learning target encodings for cat columns
finding TF-IDF weights
resampling to balance classes
each one of those is fitted, and each one leaks if fitted before the split.
They don't leak equally, and it's worth being straight about that. A scaler fitted on a few hundred well-behaved rows leaks a real but tiny amount: the mean barely moves when you add the validation rows to it, and the reported score may shift by a fraction of a per cent. Feature pick leaks enormously, which is why it's the demo above. The scaler case still matters, because a fraction of a per cent is enough to change which of two models you ship, and because the habit that blocks one blocks the other.
The other kind
The second family has nothing to do with order of ops. It's a feature that couldn't exist at guess time.
A model predicting whether a customer will churn, trained on a table that includes cancellation_reason. A model predicting loan default with a column for recovery_amount. A medical model given a field that is only filled in once a diagnosis has been made.
These are obvious when written out and very hard to spot in a warehouse table with three hundred columns, most of them undocumented. The diagnostic question isn't "is this related to the target" but "would this value be present, and correct, at the moment I need a guess?"
A near-perfect score is the strongest evidence of leakage there's. If a model reports 99% on a problem people find hard, the correct first response is suspicion, not celebration.
Leakage across time and groups
Two subtler cases are worth naming.
Time. A random split of time-series data puts future rows in the training set and past rows in the test set. The model is asked to predict backwards after being shown what happened. Split by time, always.
Groups. If one patient adds ten scans, a random split scatters them across train and test. The model can recognise the patient rather than the rule, and the score reflects that. Split by group.
The fix
Put each fitted step inside a pipeline and cross-validate the pipeline, not the model. Then each fold fits its own scaler, its own imputer and its own feature pick on that fold's training portion, which is just what you'd have to do in production.
This isn't a stylistic preference. It's the only setup in which the order of ops can't be got wrong by accident.
Where it goes wrong with Data Leakage
**Scaling or imputing before train_test_split.** The commonest form.
Selecting features on the full dataset. makes plausible scores from noise.
Random splits on time series or grouped data.
Believing a oddly good result. It's nearly always leakage, and finding out later is much more costly than checking now.
Questions people ask about Data Leakage
What is data leakage in plain terms? Information from outside the training set, usually from the test set or the future, sneaks into model fitting. The score looks great and then collapses in production.
How do I spot leakage? Be suspicious of scores that are too good to be true, or a single feature with huge importance. Check whether any feature would not be available at prediction time.
Is fitting a scaler before splitting really leakage? Yes. The scaler learns the mean and variance of the test rows, so the test set influences training. The effect is often small but it is still a leak, and it can be large for other steps like feature selection.
Does cross-validation prevent leakage? Only if every learned step runs inside each fold. Wrap preprocessing and the model in a pipeline so each fold fits its own transforms.
What is target leakage? A feature that encodes the answer, such as a refund flag when predicting whether a customer churns. It carries the label because it is recorded after the outcome.
How does time cause leakage? Random splits let the model train on rows from after the ones it is tested on. Use time-ordered splits, and group splits when rows from one entity repeat.
A model that scores well and knows nothing
Select features using the whole dataset, then cross-validate. The score is great and fully fake. The data is pure noise.
example_01.pyscikit-learn
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
# 3000 columns of noise, a coin-flip label. there is nothing to learn here.
X = rng.normal(size=(180, 3000))
y = rng.integers(0, 2, 180)
print("data: %d rows, %d columns, all noise. the label is a coin flip." % X.shape)
print("an honest model cannot beat 0.5.")
print()
# the wrong way: choose the 20 best columns using every row, then score
picked = SelectKBest(f_classif, k=20).fit(X, y).get_support()
leaked = cross_val_score(LogisticRegression(max_iter=2000), X[:, picked], y,
cv=StratifiedKFold(5, shuffle=True, random_state=0))
print("selecting features on the full dataset, then cross-validating:")
print(" fold scores", np.round(leaked, 4))
print(" mean %.4f <- from noise" % leaked.mean())
print()
# the right way: selection happens inside each fold, on training rows only
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=2000))
honest = cross_val_score(pipe, X, y, cv=StratifiedKFold(5, shuffle=True,
random_state=0))
print("selecting inside the pipeline, so each fold only sees its own training rows:")
print(" fold scores", np.round(honest, 4))
print(" mean %.4f <- the truth" % honest.mean())
print()
print("the gap is %.3f of pure illusion." % (leaked.mean() - honest.mean()))
print()
print("the leak: the selector looked at test-fold labels when choosing columns,")
print("so the 20 'best' columns were the ones that happened to correlate with")
print("the very rows the model was about to be scored on.")
print()
print("this is why every fit-like step belongs inside the pipeline. scalers,")
print("imputers, encoders and selectors all learn from data, and all of them")
print("leak if they learn from rows that are supposed to be unseen.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Why does fitting a StandardScaler before the train/test split count as leakage?
Anything that learns parameters from data must learn them from training data only. Otherwise the reported score stops being an estimate of performance on unseen data.
What is the right question to ask about a suspicious feature?
A cancellation reason correlates beautifully with churn and does not exist before the customer churns. Correlation is exactly what makes leaked features look valuable.
Why does a random split leak on time-series data?
Production always predicts forward from the past. A random split evaluates a situation that can never occur, so the score does not describe the deployed system.
Cheat sheet
Data Leakage
Spot data leakage in a pipeline before it inflates your metrics. See train-test contamination, target leakage, and preprocessing leaks with examples.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.