The failure that produces excellent scores and useless models, and the five shapes it comes in.
Overview
What it is
Leakage is information reaching the model during training that it will not have when it is actually used.
The defining property is the direction of the error. Overfitting, a bad metric, an unlucky split — these produce results that look worse than the model is, or at least ambiguous. Leakage produces results that look *better*. So it does not trigger the instinct that something is wrong; it triggers satisfaction.
That asymmetry is why it is the most expensive mistake in the field. A model with a leak passes every check, gets deployed, and fails on the first real batch — at which point the evaluation that approved it is worthless and nobody knows why.
Worth knowing
Leakage is any information reaching the model that it would not have when actually used - and it always makes the score better.
Cross-validation does not detect it. A leak in the data is inside every fold.
Target leakage: a feature computed from, or after, the outcome. The most damaging and the least visible.
Train-test contamination: preprocessing fitted before the split, duplicate rows, or one entity's rows on both sides.
Temporal leakage: shuffling data that has an order, so the model trains on the future.
The signals are a suspiciously high score, a dominant feature nobody expected, and a model that collapses in production.
Data Leakage: A Practical Guide
Every other failure in machine learning makes the score worse. This one makes it better, which is why it survives review and reaches production.
A column that was computed from the answer
The purest form: a feature that would not exist at prediction time.
example_01.pyscikit-learn
Output
The same row on both sides
Random data, nothing to learn, and a perfect score - because every test row has a twin in the training set.
example_02.pyscikit-learn
Output
One entity, several rows, both sides of the split
The label is random per person. Anything above chance is the model recognising people rather than learning anything.
example_03.pyscikit-learn
Output
Shuffling a series that has an order
Interpolating between known points is a much easier job than the one the model will actually be given.
example_04.pyscikit-learn
Output
An encoding that used the target
Target encoding is powerful and leaks by default, because each row helps compute the number that describes it.
example_05.pyscikit-learn
Output
Noticing it
There is no detector. There is a baseline, a suspicion, and two questions.
example_06.pyscikit-learn
Output
Cross-validation will not save you
Worth stating plainly, because cross-validation is otherwise the answer to almost everything on this track.
Cross-validation protects against one specific thing: a score that depends on which rows happened to be held out. If the leak is in the *data* — a column derived from the answer — then every fold contains it, every fold is inflated, and the mean of five inflated numbers is an inflated number with a reassuring standard deviation.
The first editor shows it: an honest 0.643 becomes 1.000, across all five folds, with no warning of any kind.
The five shapes
Target leakage. A feature that encodes the answer, usually because it was recorded after the outcome. A case_closed_date predicting whether a case closed. A discount_given predicting churn, when the discount was offered *because* the customer was leaving. A diagnosis code entered at the same time as the diagnosis. These are the hardest to spot because the column is legitimately in the database and nothing about its name says when it was written.
Preprocessing before the split. A scaler, imputer, encoder or feature selector fitted on all the data has seen the test rows. The cross-validation module measured this: feature selection outside the folds reported 0.825 on pure noise.
Duplicate rows. After a bad join, a re-import, or genuinely repeated records. A duplicate split across train and test means the model has the answer memorised. The second editor scores 1.000 on random data for exactly this reason.
Group leakage. Several rows describing one entity — a patient, a customer, a device, a document — landing on both sides. The model learns to recognise the entity rather than the pattern. The third editor scores 0.87 where the honest answer is chance.
Temporal leakage. Shuffling data that has an order, so the training set contains the future. The fourth editor goes from R² of 0.977 to −2.526 when the split respects time, and the second number is the one describing the job the model would actually have.
Target encoding deserves its own warning
It is the most useful high-cardinality encoding and the most reliable way to leak.
Replacing a category with the mean target for that category is compact, powerful, and computed from y. Do it over the whole dataset and every row has contributed to the number that describes it. With a hundred rows per category the contribution is small; with six, as in the fifth editor, it is a sixth — and a feature built from random labels predicts those labels at 0.71.
The fix is to compute the encoding inside the folds, with the row's own contribution excluded — which is what scikit-learn's TargetEncoder does, using an internal cross-fitting scheme. Hand-rolled target encoding is almost always leaky, and the leak is proportional to how rare the categories are, which is exactly when the encoding is most tempting.
How to notice
There is no detector. There is a set of habits.
Be suspicious of a good score. Not sceptical of a good model — suspicious of a *surprisingly* good one. If the problem is hard and the number is excellent, the first hypothesis should be a leak, not a breakthrough. Comparing against a dummy makes the size of the claim visible.
Look at feature importances. A single feature dominating, especially one nobody expected to matter, is the classic signature. Ask what it is, when it is written, and by what process.
Ask the timing question for every column. *Would this value be known, in this form, at the moment the prediction has to be made?* This is the single most effective check available, and it requires domain knowledge rather than code. A column that is only populated after the event is a leak regardless of how innocent it looks.
Ask who produced the row. If several rows can come from one entity, group the split. If rows can be duplicated, check. If the data has a timestamp, respect it.
Test on genuinely later data where possible. A model that scores well in cross-validation and badly on the next month is telling you something a random split could not.
The habits that prevent it
Split first, before anything that learns. Put every transformer in a pipeline so it is refitted inside each fold. Use GroupKFold when rows share an entity and TimeSeriesSplit when they are ordered. Check for duplicates before splitting. Hold out a final test set and touch it once.
None of these is expensive, and together they close every mechanical route. What they cannot close is target leakage, because that is a fact about the meaning of a column rather than about the code — which is why the timing question has to be asked of every feature, by someone who knows what the columns mean.
The famous cases
Leakage is not a beginner's mistake. It has invalidated published research, competition results and deployed systems, and the examples are worth knowing because they are all plausible.
A widely cited early result on detecting a disease from chest scans turned out to be partly detecting *which hospital* the scan came from — sicker patients were concentrated at one site, and the scanner left a signature in the image. The model was excellent at the task it was actually solving and useless at the one it was believed to be solving.
Competition datasets have repeatedly leaked through row ordering, where the file happened to be sorted by the target, and through id columns that correlated with when a record was created. Both give a model something real to learn that has nothing to do with the problem.
A recurring one in industry: a churn model trained on a table where the "customer contacted support" flag was updated during the cancellation call. The feature was genuinely predictive and genuinely unavailable at the moment a prediction was needed.
The common thread is that none of these look like errors in the code. Every one is a fact about how the data came to exist, which is why reading the code more carefully never finds them and asking where a column comes from usually does.
The cost of being too careful
The opposite failure exists and is worth naming, because leakage anxiety can become its own problem.
Excluding every feature that has any relationship with the target removes the signal along with the leak. A feature that is predictive because it is genuinely causally upstream is exactly what you want, and dropping it because it is "too predictive" makes the model worse for no reason.
Grouping a split when the rows are genuinely independent throws away resolution and gives an unnecessarily pessimistic estimate. Refusing to use any aggregate feature because target encoding can leak rules out a whole class of useful ones that are computed correctly.
The distinction is not how predictive a feature is. It is whether the value would exist, in that form, at the moment the prediction has to be made. A feature that passes that test is allowed to be as predictive as it likes.
Is a highly predictive feature always suspicious? No. It is worth one question - would this be known at prediction time - and if the answer is yes, use it.
How do I check for duplicates?df.duplicated().sum() before splitting, and consider near-duplicates too: rows differing only in an id or a timestamp are functionally the same row.
Does leakage matter if I only care about ranking? Yes. A leaked feature ranks well in evaluation and is absent in production, so the ranking you deploy is not the one you measured.
Can I fix a leak by removing the column? Usually, and check what else was derived from it. Aggregates, ratios and encodings built on a leaky column carry the leak with them.
A checklist before you trust a number
Six questions, in the order they are cheapest to answer. Running through them takes a few minutes and catches the large majority of leaks.
Did anything fit before the split? Scaler, imputer, encoder, selector, vectoriser. If it learned from data and it ran before train_test_split, the number is compromised. The fix is a pipeline.
Are there duplicate rows? Count them, including near-duplicates that differ only in an id or a timestamp.
Can one entity produce several rows? If so, the split must group them. This is the one people most often get wrong on data that looks tabular and independent.
Does the data have an order? Timestamps, sequence numbers, anything where later rows could not have informed earlier ones. Then the split is forward-only.
For every feature: would it exist at prediction time? Asked column by column, with someone who knows the source system. This is the slow one and the one that catches target leakage, which nothing else will.
Is the score plausible? Compared against a dummy, against published results on similar problems, and against what a domain expert would guess. Surprise is the signal.
A model that survives all six is not guaranteed honest, but the remaining routes are narrow and unusual. A model that has not been through them has an unknown probability of being wrong in the most expensive direction.
What if I find a leak after deploying? Retrain without the leaking feature and re-measure honestly before deciding whether the model is still worth running. The old score should be treated as never having existed rather than as a target to recover.
Is leakage always accidental? Mostly, and there is a well-known variety that is not: tuning against the test set until the number is satisfactory. That is leakage through the researcher rather than through the data, and it produces the same inflated result.
Things to try
Run the first editor. 0.643 becomes 1.000 from one extra column, and every fold agrees.
Vary the duplication. In the second editor, duplicate only a quarter of the rows and see how much of the inflation survives.
Change the group size. In the third editor, set per_person=2 and watch the gap narrow — the more rows per entity, the worse the leak.
Make the categories rarer. In the fifth editor, raise n_cats to 150 and watch the leaked score climb.
Where this leaves you
The failure that improves your metrics. Cross-validation cannot see it, a good score is evidence for it rather than against it, and the only reliable defence is asking, of every column, whether it would exist at the moment the prediction is needed.
Check yourself
0 of 4
Answer without scrolling back up.
What makes leakage harder to catch than other failures?
Every other failure makes results look worse or ambiguous. Leakage makes them look excellent, so it passes review.
Does cross-validation protect against a leaked column?
Cross-validation guards against a score depending on which rows were held out. A column derived from the target inflates all of them equally.
Rows from one patient appear in both train and test. What is the effect?
With a label that is random per person, shuffled folds scored 0.87 and GroupKFold 0.335. The first number was recognition, not learning.
What is the single most effective check for target leakage?
Target leakage is a fact about what a column means and when it is written, so it needs domain knowledge rather than code.
Cheat sheet
Leakage in a scikit-learn Workflow
Every other failure in machine learning makes the score worse. This one makes it better, which is why it survives review and reaches production.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.