Most estimators refuse to fit with a gap in the data. Filling it is easy; filling it without throwing away what the gap was telling you is the part worth learning.
Overview
Why it has to be handled
Almost every estimator in scikit-learn raises ValueError: Input X contains NaN rather than guessing. That refusal is deliberate — there is no universally correct fill, and silently choosing one would hide a decision that changes the model.
The exceptions are worth knowing before you reach for an imputer at all. HistGradientBoostingClassifier and its regressor sibling handle missing values natively: at each split, the tree learns which side the missing rows should go, which is a genuinely better answer than filling in a number, because it lets the missingness itself influence the prediction without inventing data. On tabular problems with gaps, starting there and skipping imputation entirely is often the right move.
For everything else, something has to fill the gap.
Worth knowing
Most estimators raise on NaN; the HistGradientBoosting pair handle it natively and often better than any imputation.
SimpleImputer learns one number per column - statistics_ - so it must be fitted on the training data only.
median over mean for anything skewed or with outliers; most_frequent or a constant for categorical columns.
add_indicator=True keeps a flag for where the value was missing, which is free and frequently the most useful feature added.
KNNImputer and IterativeImputer use the other columns, which beats a column constant when features are correlated.
Dropping rows is only safe when values are missing at random and there are few of them - otherwise it biases the data.
Missing Values: A Practical Guide
A gap in the data is two problems: the estimator will not run, and the gap itself may have been telling you something.
Most estimators simply refuse
And one family does not, which is worth knowing before reaching for an imputer at all.
example_01.pyscikit-learn
Output
The four simple strategies
Note what the outlier does to the mean, and what it does not do to the median.
example_02.pyscikit-learn
Output
Keeping the fact that it was missing
add_indicator appends a column recording where the gaps were.
example_03.pyscikit-learn
Output
The fill value is learned, so it obeys the split
The test rows get the training median, not their own.
example_04.pyscikit-learn
Output
Using the other columns to guess
When features are correlated, the column mean is a poor guess and the neighbouring rows are a good one.
example_05.pyscikit-learn
Output
When the gap is the signal
The value is missing precisely when the answer is yes. Imputing it away destroys the most useful feature in the data.
example_06.pyscikit-learn
Output
The simple strategies, and choosing between them
SimpleImputer replaces missing values with one number per column, learned during fit and stored in statistics_.
mean is the default and the wrong default for most real data. The editor above shows why: with values of 1, 2 and 100, the mean is 34.33 — a number nothing like any of the observations. Any skew or outlier drags it.
median is the safer choice for numeric columns and should probably be your habit. It is unaffected by the tail and always lands on a plausible value.
most_frequent works for categorical columns and for numeric ones with few distinct values.
constant fills with fill_value — useful when there is a meaningful default, and useful with a sentinel like -1 when you want the model to be able to see the imputed rows. For categorical data, fill_value="missing" makes absence an explicit category, which is often exactly right.
The fill value is learned
This is the part that connects back to the rest of the track: statistics_ is computed from data, so an imputer fitted on everything has seen the test set.
The effect is usually small — a median shifts a little — but the principle is the same as everywhere else, and there is a stronger practical reason. In production, the median of the future data is not available. The model has to fill gaps with a number computed at training time, so that is the number the evaluation should use.
The editor above shows the training median and the test median differing in the third decimal. The test rows get the training one, which is correct.
SimpleImputer inside a Pipeline handles this automatically, refitting on each training fold. As with scaling, there is no good reason to do it by hand.
Missingness is information
The most common mistake is not choosing the wrong fill value. It is discarding the fact that a value was missing.
Data is rarely missing at random. A blank income field on a loan application, an absent test result, an unrecorded satisfaction score — in each case the absence usually correlates with something. People decline to answer questions for reasons, tests are not ordered for reasons, and fields go unfilled for reasons.
add_indicator=True appends a binary column marking where each gap was. It costs one column per affected feature and it hands the model the pattern of absence as a feature in its own right.
The last editor makes the size of the effect concrete. With a value missing mostly when the answer is yes, imputing alone scores 0.77; imputing with an indicator scores 0.987. The imputer filled in a plausible number and erased the most informative thing in the dataset; the indicator put it back.
The taxonomy is worth knowing by name. Missing completely at random — the gap has no relationship to anything — is the only case where plain imputation loses nothing, and it is the rarest. Missing at random, where the gap depends on other observed features, is common and partly recoverable by a model-based imputer. Missing not at random, where the gap depends on the unobserved value itself, is both the most common and the one where the indicator matters most.
Using the other columns
When features are correlated, a column constant is a poor guess. Two better options use the rest of the row.
KNNImputer finds the most similar complete rows and averages their values for the missing feature. The editor above shows it recovering 8.0 where the column mean gives 5.5, because the two columns move together and the neighbours know it. It needs the features scaled, since it measures distance, and it is expensive on large data because it searches for neighbours.
IterativeImputer models each column with missing values as a function of the others, in rounds, until the estimates settle. It is the most capable of the three and the slowest, and it is still marked experimental — importing it requires from sklearn.experimental import enable_iterative_imputer first, which catches everyone once.
Both are worth trying when imputation quality matters. Neither is worth the complexity when the missingness is light and the model is a gradient booster that could have handled it natively.
Dropping, and when it is allowed
dropna() is the tempting one-liner and it is only safe under conditions worth checking.
Dropping rows is defensible when the values are genuinely missing at random and the affected rows are few. It is a mistake when the missingness is informative, because you are deleting exactly the rows that carry the pattern — and the bias it introduces is invisible in every subsequent metric.
Dropping a column is defensible when most of it is missing and no indicator would help. A column that is 95% empty rarely carries enough to be worth imputing, though the 5% pattern occasionally does — which is what the indicator would tell you.
The asymmetry to remember: you can drop rows from the training data, but you cannot drop them at prediction time. A model that has never seen a missing value in a column will still be handed one eventually, and something has to happen. Deciding what, during training, is the whole point of having an imputer in the pipeline.
Categorical columns need a different answer
Everything above assumes numeric data. Text columns have gaps too, and the arithmetic strategies do not apply.
SimpleImputer(strategy="most_frequent") works and quietly asserts that the missing rows are like the majority, which is often exactly the claim that is false. strategy="constant", fill_value="missing" is usually better: it creates an explicit category, the encoder gives it its own column, and the model can learn whatever that column is worth. Absence becomes a value rather than a guess.
OneHotEncoder also handles NaN directly, treating it as its own category without any imputation step, which is the shortest route to the same result.
The reason the explicit category usually wins is the same as for the indicator: in real data, "not stated" is a fact about the record rather than a hole where a fact should be. A blank employer field, an unselected optional dropdown, a survey question skipped — each tells you something, and folding it into the most common value asserts the opposite.
How much is too much
A quick rule for triage, applied per column rather than to the dataset as a whole.
Under about 5% missing. Almost any reasonable imputation works and the choice barely matters. Use the median, add the indicator, move on.
Between 5% and 30%. The choice starts to matter. Compare a couple of strategies by cross-validation, and treat the indicator as compulsory rather than optional.
Over 30%. The column is mostly absent, and imputing it means most of it is invented. Consider keeping only the indicator and discarding the values, which sometimes performs better than either the column or its removal.
Over 80%. Usually drop the column, unless the small present portion is known to be highly informative.
These are rules of thumb rather than thresholds, and the honest way to settle any of them is to try both inside a pipeline and compare cross-validated scores. That takes two lines, which is less effort than arguing about it.
Why does IterativeImputer need a special import? It is still marked experimental, so from sklearn.experimental import enable_iterative_imputer has to run first. The import looks unused and is not.
Should I impute the target? No. Rows with a missing target carry no information for supervised learning - drop them, and check why they are missing before you do.
Does KNNImputer need scaling? Yes. It measures distance between rows, so a column in thousands dominates the neighbour search exactly as it would for k-NN itself.
Can I impute before splitting if I only use the median? No. The median is learned from data, so it carries information from the test rows into training. The effect is small and the habit is what matters.
Finding out what is actually missing
Before choosing a strategy, two minutes of counting decides most of it.
Count per column rather than in total. df.isna().sum() gives the picture, and it is usually uneven — one or two columns carry nearly all the gaps while the rest are complete. That changes the problem from "how do I impute this dataset" to "what do I do about these two columns", which is a much easier question.
Then look at whether the gaps coincide. Columns that are missing together usually come from the same source: a form section that was skipped, a system that was down, a join that failed for a subset. If three columns are missing on exactly the same rows, they are one piece of information absent once, and one indicator covers all three rather than three separate ones.
Finally, compare the target between rows with and without the gap. A difference in the target rate between present and missing is direct evidence that the missingness is informative, and it takes one line to check. When it is large, the indicator is not optional and imputation alone will lose most of the signal — which is what the last editor on this page demonstrates at 0.77 against 0.987.
None of this requires a plot or a library. Three counts and a comparison, before any decision about strategy.
What about values that are not NaN but mean missing? Sentinels like -999, 0, or an empty string are extremely common in exported data, and no imputer recognises them. Convert them to NaN first, or the model treats -999 as a very small number.
Does the order of imputation and encoding matter? Yes. Impute categorical columns before encoding them, or the encoder treats NaN as its own category - which is sometimes what you want and should be a decision rather than an accident.
Things to try
Compare the strategies. In the second editor, note that mean fills 34.33 into a column whose other values are 1, 2 and 100.
Recover the correlation. The fifth editor has KNNImputer land on 8.0 exactly. Change the neighbours to 1 and see what happens.
Turn the indicator off. The last editor drops from 0.987 to 0.77. That gap is the value of one extra column.
Skip imputation entirely. Replace the pipeline in the last editor with HistGradientBoostingClassifier() on the raw data with gaps.
Where this leaves you
Median rather than mean for numeric columns, an explicit category for categorical ones, add_indicator=True almost always, the imputer inside a pipeline so the fill value obeys the split, and a check of whether a gradient booster could have handled the gaps without any of it.
Check yourself
0 of 4
Answer without scrolling back up.
What does SimpleImputer learn during fit?
One number per column, computed from the data it was fitted on - which is why it must be fitted on the training half only.
Why prefer median over mean for numeric columns?
With values of 1, 2 and 100 the mean is 34.33 - a number nothing like any observation. The median is 2.
What does add_indicator=True do?
It preserves the fact of absence, which is frequently informative. In the last editor it took the score from 0.77 to 0.987.
Which estimator handles NaN without an imputer?
It learns which side of each split the missing rows belong on, which uses the missingness rather than filling over it.
Cheat sheet
Missing Values
Most estimators refuse to fit with a gap in the data. Filling it is easy; filling it without throwing away what the gap was telling you is the part worth learning.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.