The failure that produces excellent scores and useless models, and the five shapes it comes in.
Overview
What it is
Data leakage is information from the future or from the test set reaching the model during training. It makes the model look better than it actually is. The score goes up, confidence goes up, and the model fails in production.
Leakage is harder to catch than other mistakes because it produces good numbers. A bug that crashes or gives a low score gets noticed quickly. A bug that gives a suspiciously high score gets celebrated. That asymmetry is why leakage causes more real-world damage than any other modelling error.
The most common form is preprocessing before splitting. Scaling the full dataset, imputing with the global median, encoding with frequencies computed on all rows. Each of these lets the test set influence the training process. The resulting score is optimistic by an amount you cannot measure without fixing the leak and rerunning.
A second common form is using features that carry information about the target that would not be available at prediction time. A hospital-readmission model that includes the discharge summary, which was written after the readmission decision, is an example. The feature is predictive precisely because it was created after the event.
This page covers both forms, shows how to detect them, and explains why cross-validation alone does not protect you.
Leakage is info reaching the model during training that it won't have when it's really used.
The defining property is the direction of the error. Overfitting, a bad metric, an unlucky split. These give results that look worse than the model is, or at least ambiguous. Leakage makes results that look *better*. So it doesn't trigger the instinct that a thing is wrong. It triggers satisfaction.
That skew is why it's the most costly mistake in the field. A model with a leak passes each check, gets deployed. Fails on the first real batch. At which point the test that approved it's worthless and no one knows why.
Worth knowing
Leakage is any information reaching the model that it would not have when actually used - and it always makes the score better.
Cross-validation does not detect it. A leak in the data is inside every fold.
Target leakage: a feature computed from, or after, the outcome. The most damaging and the least visible.
Train-test contamination: preprocessing fitted before the split, duplicate rows, or one entity's rows on both sides.
Temporal leakage: shuffling data that has an order, so the model trains on the future.
The signals are a suspiciously high score, a dominant feature nobody expected, and a model that collapses in production.
Data Leakage: A Practical Guide
Every other failure in machine learning makes the score worse. This one makes it better. This Is why it lives review and reaches production.
A column that was computed from the answer
The purest form: a feature that would not exist at prediction time.
example_01.pyscikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import numpy as np
rng = np.random.RandomState(0)
n = 400
X = rng.normal(size=(n, 4))
y = (X[:, 0] + rng.normal(scale=2.0, size=n) > 0).astype(int)
honest = cross_val_score(LogisticRegression(), X, y, cv=5).mean()
# A column built from the answer - a risk score filled in after the outcome
# was known, say, or a status field the database updates on closure.
leaked = np.column_stack([X, y + rng.normal(scale=0.01, size=n)])
cheating = cross_val_score(LogisticRegression(), leaked, y, cv=5).mean()
print("features that were available beforehand:", round(honest, 3))
print("with one column derived from the answer:", round(cheating, 3))
print()
print("cross-validation cannot catch this: the leak is inside every fold.")
Output
The same row on both sides
Random data, nothing to learn, and a perfect score - because every test row has a twin in the training set.
example_02.pyscikit-learn
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
import numpy as np
rng = np.random.RandomState(0)
X = rng.normal(size=(200, 6))
y = rng.randint(0, 2, size=200) # nothing to learn
clean = cross_val_score(KNeighborsClassifier(1), X, y, cv=5).mean()
# The same rows, twice - as happens after a bad join or a re-import.
Xd = np.vstack([X, X])
yd = np.concatenate([y, y])
duped = cross_val_score(KNeighborsClassifier(1), Xd, yd, cv=5).mean()
print("random data, no duplicates :", round(clean, 3))
print("every row duplicated :", round(duped, 3))
print()
print("the nearest neighbour of a test row is its own twin in the")
print("training set, so the model looks up the answer.")
Output
One entity, several rows, both sides of the split
The label is random per person. Anything above chance is the model recognising people rather than learning anything.
example_03.pyscikit-learn
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, GroupKFold, KFold
import numpy as np
rng = np.random.RandomState(0)
n_people, per_person = 40, 5
groups = np.repeat(np.arange(n_people), per_person)
# Each person has a measurable fingerprint, and a label that has nothing
# to do with it. Nothing here generalises to a new person.
fingerprint = rng.normal(size=n_people)
label = rng.randint(0, 2, size=n_people)
X = fingerprint[groups][:, None] + rng.normal(scale=0.01,
size=(n_people * per_person, 1))
y = label[groups]
model = RandomForestClassifier(random_state=0)
shuffled = cross_val_score(model, X, y,
cv=KFold(5, shuffle=True, random_state=0)).mean()
grouped = cross_val_score(model, X, y, cv=GroupKFold(5), groups=groups).mean()
print("shuffled folds - a person appears on both sides:", round(shuffled, 3))
print("GroupKFold - each person is kept whole :", round(grouped, 3))
Output
Shuffling a series that has an order
Interpolating between known points is a much easier job than the one the model will actually be given.
example_04.pyscikit-learn
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import cross_val_score, KFold, TimeSeriesSplit
import numpy as np
rng = np.random.RandomState(0)
n = 300
t = np.arange(n).reshape(-1, 1)
y = np.cumsum(rng.normal(size=n)) # a random walk over time
model = KNeighborsRegressor(3)
shuffled = cross_val_score(model, t, y,
cv=KFold(5, shuffle=True, random_state=0)).mean()
forward = cross_val_score(model, t, y, cv=TimeSeriesSplit(5)).mean()
print("shuffled folds (R2) :", round(shuffled, 3))
print("TimeSeriesSplit (R2) :", round(forward, 3))
print()
print("shuffling lets the model interpolate between points it has seen")
print("on both sides. Predicting forward is the job it will actually have.")
Output
An encoding that used the target
Target encoding is powerful and leaks by default, because each row helps compute the number that describes it.
example_05.pyscikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import numpy as np
rng = np.random.RandomState(0)
n, n_cats = 300, 50
cat = rng.randint(0, n_cats, size=n)
y = rng.randint(0, 2, size=n) # random - unrelated to the category
# Target encoding computed over the whole dataset: each row's own label
# has contributed to the number that describes it.
means = np.array([y[cat == c].mean() if (cat == c).any() else 0.5
for c in range(n_cats)])
X_leaky = means[cat].reshape(-1, 1)
print("target-encoded on all rows:",
round(cross_val_score(LogisticRegression(), X_leaky, y, cv=5).mean(), 3))
print("the truth (labels are random): 0.5")
print()
print("with six rows per category, a row is a sixth of its own encoding.")
Output
Noticing it
There is no detector. There is a baseline, a suspicion, and two questions.
example_06.pyscikit-learn
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import numpy as np
rng = np.random.RandomState(0)
X = rng.normal(size=(300, 5))
y = (X[:, 0] > 0).astype(int)
X = np.column_stack([X, y]) # the answer, as a column
base = cross_val_score(DummyClassifier(strategy="most_frequent"), X, y, cv=5).mean()
real = cross_val_score(LogisticRegression(), X, y, cv=5).mean()
print("dummy baseline :", round(base, 3))
print("the model :", round(real, 3))
print()
print("a perfect score is not a triumph, it is a symptom.")
print("the checks: is any feature unavailable at prediction time?")
print(" was any column computed after the outcome was known?")
Output
Cross-validation will not save you
Worth stating plainly, because cross-validation is or else the answer to almost all on this track.
Cross-validation protects against one exact thing: a score that depends on which rows happened to be held out. If the leak is in the *data*. A column derived from the answer. Then each fold holds it, each fold is inflated. The mean of five inflated numbers is an inflated number with a reassuring standard deviation.
The first editor shows it: an honest 0.643 turns 1.000, across all five folds, with no warning of any kind.
The five shapes
Target leakage. A feature that codes the answer, often because it was recorded after the outcome. A case_closed_date predicting whether a case closed. A discount_given predicting churn, when the discount was offered *because* the customer was leaving. A diagnosis code entered also as the diagnosis. These are the hardest to spot because the column is legitimately in the DB and nothing about its name says when it was written.
prep before the split. A scaler, imputer, encoder or feature selector fitted on all the data has seen the test rows. The cross-validation module measured this. Feature pick outside the folds reported 0.825 on pure noise.
Duplicate rows. After a bad join, a re-import, or truly repeated records. A duplicate split across train and test means the model has the answer stored. The second editor scores 1.000 on random data for just this reason.
Group leakage. a few rows showing one entity. A patient, a customer, a device, a doc. Landing on both sides. The model learns to recognise the entity rather than the pattern. The third editor scores 0.87 where the honest answer is chance.
Temporal leakage. Shuffling data that has an order. The training set holds the future. The fourth editor goes from R² of 0.977 to −2.526 when the split respects time. The second number is the one showing the job the model would really have.
Target encoding deserves its own warning
It's the most handy high-cardinality encoding and the most solid way to leak.
Replacing a category with the mean target for that category is compact, powerful. Found from y. Do it over the whole dataset and each row has contributed to the number that shows it. With a hundred rows per category the part is small. With six, as in the fifth editor, it's a sixth. And a feature built from random labels predicts those labels at 0.71.
The fix is to find the encoding inside the folds, with the row's own part excluded. Which is what scikit-learn's TargetEncoder does, using an inner cross-fitting scheme. Hand-rolled target encoding is almost always leaky. The leak is proportional to how rare the categories are. This Is just when the encoding is most tempting.
How to notice
There's no finder. There's a set of habits.
Be suspicious of a good score. Not sceptical of a good model. Suspicious of a *oddly* good one. If the problem is hard and the number is great, the first hypothesis should be a leak, not a breakthrough. Checking against a dummy makes the size of the claim visible.
Look at feature importances. A single feature dominating, above all one no one expected to matter, is the classic signature. Ask what it's, when it's written, and by what process.
Ask the timing question for each column. *Would this value be known, in this form, at the moment the guess has to be made?* This is the single most effective check available. It needs domain knowledge rather than code. A column that is only populated after the event is a leak even so of how innocent it looks.
Ask who made the row. If a few rows can come from one entity, group the split. If rows can be duplicated, check. If the data has a timestamp, respect it.
Test on truly later data where possible. A model that scores well in cross-validation and badly on the next month is telling you a thing a random split couldn't.
The habits that prevent it
Split first, before any of it that learns. Put each transformer in a pipeline so it's refitted inside each fold. Use GroupKFold when rows share an entity and TimeSeriesSplit when they're sorted. Check for duplicates before splitting. Hold out a final test set and touch it once.
None of these is costly, and together they close each mechanical route. What they can't close is target leakage. That is a fact about the meaning of a column rather than about the code. Which is why the timing question has to be asked of each feature, by someone who knows what the columns mean.
The famous cases
Leakage isn't a beginner's mistake. It has invalidated published research, contest results and deployed systems. The examples are worth knowing because they're all plausible.
A widely cited early result on detecting a disease from chest scans turned out to be partly detecting *which hospital* the scan came from. Sicker patients were concentrated at one site. The scanner left a signature in the image. The model was great at the task it was really solving and useless at the one it was believed to be solving.
Contest datasets have again and again leaked through row sorting. Here The file happened to be sorted by the target. Through id columns that linked with when a record was created. Both give a model a thing real to learn that has nothing to do with the problem.
A recurring one in industry: a churn model trained on a table where the "customer contacted support" flag was updated during the cancellation call. The feature was truly predictive and truly down at the moment a guess was needed.
The common thread is that none of these look like errors in the code. each one is a fact about how the data came to exist. This Is why reading the code more carefully never finds them and asking where a column comes from often does.
The cost of being too careful
The opposite failure exists and is worth naming. Leakage anxiety can become its own problem.
Excluding each feature that has any tie with the target drops the signal along with the leak. A feature that is predictive because it's truly causally upstream is just what you want. Dropping it because it's "too predictive" makes the model worse for no reason.
Grouping a split when the rows are truly free throws away detail and gives an needlessly pessimistic guess. Refusing to use any aggregate feature because target encoding can leak rules out a whole class of handy ones that are found right.
The gap isn't how predictive a feature is. It's whether the value would exist, in that form, at the moment the guess has to be made. A feature that passes that test is allowed to be as predictive as it likes.
Is a highly predictive feature always suspicious? No. It's worth one question - would this be known at guess time - and if the answer is yes, use it.
How do I check for duplicates?df.duplicated().sum() before splitting. Consider near-duplicates too. Rows differing only in an id or a timestamp are functionally the same row.
Does leakage matter if I only care about ranking? Yes. A leaked feature ranks well in test and is absent in production. The ranking you deploy isn't the one you measured.
Can I fix a leak by dropping the column? often. Check what else was derived from it. Aggregates, ratios and encodings built on a leaky column carry the leak with them.
A checklist before you trust a number
Six questions, in the order they're cheapest to answer. Running through them takes a few minutes and catches the large majority of leaks.
Did any of it fit before the split? Scaler, imputer, encoder, selector, vectoriser. If it learned from data and it ran before train_test_split, the number is compromised. The fix is a pipeline.
Are there duplicate rows? Count them, including near-duplicates that differ only in an id or a timestamp.
Can one entity give a few rows? If so, the split must group them. This is the one people most often get wrong on data that looks tabular and free.
Does the data have an order? Timestamps, sequence numbers, any of it where later rows couldn't have informed earlier ones. Then the split is forward-only.
For each feature: would it exist at guess time? Asked column by column, with someone who knows the source system. This is the slow one and the one that catches target leakage. This Nothing else will.
Is the score plausible? Checked against a dummy, against published results on similar problems. Against what a domain expert would guess. Surprise is the signal.
A model that lives all six isn't ensured honest. The left routes are narrow and unusual. A model that hasn't been through them has an unknown chance of being wrong in the most costly direction.
What if I find a leak after deploying? Retrain without the leaking feature and re-measure honestly before deciding whether the model is still worth running. The old score should be treated as never having existed rather than as a target to recover.
Is leakage always accidental? Mostly, and there's a well-known range that isn't. Tuning against the test set until the number is satisfactory. That is leakage through the researcher rather than through the data, and it makes the same inflated result.
Things to try with Leakage in a scikit-learn Workflow
Run the first editor. 0.643 turns 1.000 from one extra column, and each fold agrees.
Vary the duplication. In the second editor, duplicate only a quarter of the rows and see how much of the inflation lives.
Change the group size. In the third editor, set per_person=2 and watch the gap narrow. The more rows per entity, the worse the leak.
Make the categories rarer. In the fifth editor, raise n_cats to 150 and watch the leaked score climb.
Questions people ask about Leakage in a scikit-learn Workflow
What is the most common cause of leakage? Fitting a scaler, imputer or encoder on the full dataset before splitting, which lets test information shape training.
Does cross-validation prevent leakage? Only if preprocessing happens inside each fold. A pipeline passed to cross_val_score does this correctly.
How can I spot leakage? A score that looks too good to be true is the main sign. Also check for features that would not exist at prediction time.
What is target leakage? A feature that contains or is derived from the outcome, such as a refund flag when predicting churn. The model looks great and fails in production.
Can duplicate rows leak? Yes. Identical or near-identical rows in train and test inflate the score, so deduplicate or group them before splitting.
Is leakage possible in time series? Yes. Shuffling lets the model see the future, so split by time and build features only from earlier data.
Where this leaves you
The failure that raises your metrics. Cross-validation can't see it, a good score is evidence for it rather than against it. The only solid defence is asking, of each column, whether it would exist at the moment the guess is needed.
Check yourself
0 of 4
Answer without scrolling back up.
What makes leakage harder to catch than other failures?
Every other failure makes results look worse or ambiguous. Leakage makes them look excellent, so it passes review.
Does cross-validation protect against a leaked column?
Cross-validation guards against a score depending on which rows were held out. A column derived from the target inflates all of them equally.
Rows from one patient appear in both train and test. What is the effect?
With a label that is random per person, shuffled folds scored 0.87 and GroupKFold 0.335. The first number was recognition, not learning.
What is the single most effective check for target leakage?
Target leakage is a fact about what a column means and when it is written, so it needs domain knowledge rather than code.
Cheat sheet
Leakage in a scikit-learn Workflow
Spot data leakage before it inflates your score: learn the patterns that let test information leak into training and how pipelines prevent them.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.