Most estimators refuse to fit with a gap in the data. Filling it is easy; filling it without throwing away what the gap was telling you is the part worth learning.
Overview
Why it has to be handled
Real datasets almost always have missing values. A sensor drops a reading, a survey respondent skips a question, a join produces nulls where records do not match. The gaps are part of the data, and handling them is part of the modelling work.
scikit-learn does not fill missing values for you. Most estimators raise an error when they encounter a NaN. That design is deliberate. There is more than one way to fill a gap, and the choice affects the model. Filling with the mean preserves the column's average but shrinks its variance. Filling with the median is less affected by outliers. Filling with a constant like -1 creates a new category that the model can split on.
The right strategy depends on why the data is missing and what the model will do with it. A column where missingness itself is informative -- for example, income is missing because the respondent refused to answer -- should not be filled with the median. The fact that it is missing carries signal.
This page covers the imputation strategies scikit-learn provides, the tree-based models that handle NaN natively, and the common mistake of imputing before splitting.
Almost each estimator in scikit-learn raises ValueError: Input X holds NaN rather than guessing. That refusal is deliberate. There's no universally correct fill. Silently choosing one would hide a choice that changes the model.
The exceptions are worth knowing before you reach for an imputer at all. HistGradientBoostingClassifier and its regressor sibling handle missing values natively. At each split, the tree learns which side the missing rows should go. This Is a truly better answer than filling in a number. It lets the missingness itself sway the guess without inventing data. On tabular problems with gaps, starting there and skipping imputation fully is often the right move.
For all else, a thing has to fill the gap.
Worth knowing
Most estimators raise on NaN; the HistGradientBoosting pair handle it natively and often better than any imputation.
SimpleImputer learns one number per column - statistics_ - so it must be fitted on the training data only.
median over mean for anything skewed or with outliers; most_frequent or a constant for categorical columns.
add_indicator=True keeps a flag for where the value was missing, which is free and frequently the most useful feature added.
KNNImputer and IterativeImputer use the other columns, which beats a column constant when features are correlated.
Dropping rows is only safe when values are missing at random and there are few of them - otherwise it biases the data.
Missing Values: A Practical Guide
A gap in the data is two problems: the estimator won't run. The gap itself may have been telling you a thing.
Most estimators simply refuse
And one family does not, which is worth knowing before reaching for an imputer at all.
example_01.pyscikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
import numpy as np
X = np.array([[1.0, 2.0], [np.nan, 3.0], [3.0, 4.0], [4.0, 5.0]])
y = np.array([0, 0, 1, 1])
try:
LogisticRegression().fit(X, y)
except ValueError as e:
print("LogisticRegression:", str(e).split("\n")[0][:80])
HistGradientBoostingClassifier(max_iter=10).fit(X, y)
print()
print("HistGradientBoosting: fitted, NaN handled natively")
Output
The four simple strategies
Note what the outlier does to the mean, and what it does not do to the median.
example_02.pyscikit-learn
from sklearn.impute import SimpleImputer
import numpy as np
X = np.array([[1.0], [2.0], [100.0], [np.nan]])
for strategy in ("mean", "median", "most_frequent", "constant"):
imp = SimpleImputer(strategy=strategy, fill_value=-1)
out = imp.fit_transform(X).ravel()
print("%-14s filled with %8.2f -> %s" % (strategy, out[-1], out))
Output
Keeping the fact that it was missing
add_indicator appends a column recording where the gaps were.
example_03.pyscikit-learn
from sklearn.impute import SimpleImputer
import numpy as np
X = np.array([[1.0], [2.0], [np.nan], [4.0]])
imp = SimpleImputer(strategy="mean", add_indicator=True)
out = imp.fit_transform(X)
print("columns out:", out.shape[1], "- the value, and a was-it-missing flag")
print(out)
print()
print("the flag lets the model use the fact of absence, which is often")
print("informative in its own right.")
Output
The fill value is learned, so it obeys the split
The test rows get the training median, not their own.
example_04.pyscikit-learn
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
import numpy as np
rng = np.random.RandomState(0)
X = rng.normal(loc=10.0, size=(200, 1))
X[rng.rand(200) < 0.2] = np.nan
X_tr, X_te = train_test_split(X, test_size=0.5, random_state=0)
imp = SimpleImputer(strategy="median").fit(X_tr)
print("median learned from the training half:", round(imp.statistics_[0], 4))
print("median of the test half (not used) :",
round(float(np.nanmedian(X_te)), 4))
print()
print("the test rows are filled with the training median, because that")
print("is the number that would be available in production.")
Output
Using the other columns to guess
When features are correlated, the column mean is a poor guess and the neighbouring rows are a good one.
example_05.pyscikit-learn
from sklearn.impute import SimpleImputer, KNNImputer
import numpy as np
# Two correlated columns; the second is missing on one row.
X = np.array([[1.0, 2.0], [2.0, 4.0], [3.0, 6.0], [4.0, np.nan], [5.0, 10.0]])
simple = SimpleImputer(strategy="mean").fit_transform(X)
knn = KNNImputer(n_neighbors=2).fit_transform(X)
print("true value would be about 8.0")
print("SimpleImputer (column mean):", round(simple[3, 1], 3))
print("KNNImputer (similar rows) :", round(knn[3, 1], 3))
Output
When the gap is the signal
The value is missing precisely when the answer is yes. Imputing it away destroys the most useful feature in the data.
example_06.pyscikit-learn
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
import numpy as np
rng = np.random.RandomState(0)
n = 600
X = rng.normal(size=(n, 3))
y = (X[:, 0] > 0).astype(int)
# Missing NOT at random: the value is absent mostly when y is 1.
Xm = X.copy()
Xm[(y == 1) & (rng.rand(n) < 0.7), 0] = np.nan
plain = make_pipeline(SimpleImputer(), LogisticRegression())
flagged = make_pipeline(SimpleImputer(add_indicator=True), LogisticRegression())
print("impute only :", round(cross_val_score(plain, Xm, y, cv=5).mean(), 3))
print("impute + indicator :", round(cross_val_score(flagged, Xm, y, cv=5).mean(), 3))
print()
print("when absence itself carries the signal, throwing it away costs you.")
Output
The simple strategies, and choosing between them
SimpleImputer replaces missing values with one number per column, learned during fit and stored in statistics_.
mean is the default and the wrong default for most real data. The editor above shows why. With values of 1, 2 and 100, the mean is 34.33. A number nothing like any of the readings. Any skew or outlier drags it.
median is the safer choice for numeric columns and should probably be your habit. It's unaffected by the tail and always lands on a plausible value.
most_frequent works for cat columns and for numeric ones with few distinct values.
constant fills with fill_value — handy when there's a real default. Handy with a sentinel like -1 when you want the model to be able to see the imputed rows. For cat data, fill_value="missing" makes lack an clear category. This Is often just right.
The fill value is learned
This is the part that connects back to the rest of the track: statistics_ is found from data. An imputer fitted on all has seen the test set.
The effect is often small. A median shifts a little — but the principle is the same as all over else. There's a stronger hands-on reason. In production, the median of the future data isn't available. The model has to fill gaps with a number found at training time. That is the number the test should use.
The editor above shows the training median and the test median differing in the third decimal. The test rows get the training one, which is correct.
SimpleImputer inside a Pipeline handles this automatically, refitting on each training fold. As with scaling, there's no good reason to do it by hand.
Missingness is information
The most common mistake isn't choosing the wrong fill value. It's discarding that a value was missing.
Data is rarely missing at random. A blank income field on a loan use, an absent test result, an unrecorded satisfaction score. In each case the lack often correlates with a thing. People decline to answer questions for reasons, tests aren't sorted for reasons. Fields go unfilled for reasons.
add_indicator=True appends a binary column marking where each gap was. It costs one column per affected feature and it hands the model the pattern of lack as a feature in its own right.
The last editor makes the size of the effect concrete. With a value missing mostly when the answer is yes, imputing alone scores 0.77. Imputing with an indicator scores 0.987. The imputer filled in a plausible number and erased the most helpful thing in the dataset. The indicator put it back.
The taxonomy is worth knowing by name. Missing fully at random. The gap has no tie to any of it. Is the only case where plain imputation loses nothing. It's the rarest. Missing at random. Here The gap depends on other seen features, is common and partly recoverable by a model-based imputer. Missing not at random. Here The gap depends on the unobserved value itself, is both the most common and the one where the indicator matters most.
Using the other columns
When features are linked, a column constant is a poor guess. Two better options use the rest of the row.
KNNImputer finds the most similar complete rows and means their values for the missing feature. The editor above shows it getting back 8.0 where the column mean gives 5.5. The two columns move together and the neighbours know it. It needs the features scaled. It measures distance. It's costly on large data because it searches for neighbours.
IterativeImputer models each column with missing values as a function of the others, in rounds, until the guesses settle. It's the most capable of the three and the slowest. It's still marked experimental. Importing it needs from sklearn.experimental import enable_iterative_imputer first. This Catches everyone once.
Both are worth trying when imputation quality matters. Neither is worth the cost when the missingness is light and the model is a grad booster that could have handled it natively.
Dropping, and when it is allowed
dropna() is the tempting one-liner and it's only safe under rules worth checking.
Dropping rows is defensible when the values are truly missing at random and the affected rows are few. It's a mistake when the missingness is helpful. You're deleting just the rows that carry the pattern. And the bias it adds is invisible in each later metric.
Dropping a column is defensible when most of it's missing and no indicator would help. A column that is 95% empty rarely carries enough to be worth imputing, though the 5% pattern at times does. Which is what the indicator would tell you.
The skew to note: you can drop rows from the training data. You can't drop them at guess time. A model that has never seen a missing value in a column will still be handed one eventually. A thing has to happen. Deciding what, during training, is the whole point of having an imputer in the pipeline.
Categorical columns need a different answer
All above assumes numeric data. Text columns have gaps too. The math strategies don't apply.
SimpleImputer(strategy="most_frequent") works and quietly asserts that the missing rows are like the majority. This Is often just the claim that is false. strategy="constant", fill_value="missing" is often better. It creates an clear category, the encoder gives it its own column. The model can learn what that column is worth. Lack turns a value rather than a guess.
OneHotEncoder also handles NaN right, treating it as its own category without any imputation step. This Is the shortest route to the same result.
The reason the clear category often wins is the same as for the indicator. In real data, "not stated" is a fact about the record rather than a hole where a fact should be. A blank employer field, an unselected optional dropdown, a survey question skipped. Each tells you a thing. Folding it into the most common value asserts the opposite.
How much is too much
A quick rule for triage, applied per column rather than to the dataset as a whole.
Under about 5% missing. Almost any reasonable imputation works and the choice barely matters. Use the median, add the indicator, move on.
Between 5% and 30%. The choice starts to matter. Compare a couple of strategies by cross-validation. Treat the indicator as compulsory rather than optional.
Over 30%. The column is mostly absent. Imputing it means most of it's invented. Consider keeping only the indicator and discarding the values. This At times performs better than either the column or its removal.
Over 80%. often drop the column, unless the small present portion is known to be highly helpful.
These are rules of thumb rather than thresholds. The honest way to settle any of them is to try both inside a pipeline and compare cross-validated scores. That takes two lines. This Is less effort than arguing about it.
Why does IterativeImputer need a special import? It's still marked experimental. from sklearn.experimental import enable_iterative_imputer has to run first. The import looks unused and isn't.
Should I impute the target? No. Rows with a missing target carry no info for guided learning - drop them. Check why they're missing before you do.
Does KNNImputer need scaling? Yes. It measures distance between rows. A column in thousands dominates the neighbour search just as it would for k-NN itself.
Can I impute before splitting if I only use the median? No. The median is learned from data. It carries info from the test rows into training. The effect is small and the habit is what matters.
Finding out what is actually missing
Before choosing a strategy, two minutes of counting decides most of it.
Count per column rather than in total. df.isna().sum() gives the picture. It's often uneven. One or two columns carry nearly all the gaps while the rest are complete. That changes the problem from "how do I impute this dataset" to "what do I do about these two columns". This Is a much easier question.
Then look at whether the gaps coincide. Columns that are missing together often come from the same source. A form section that was skipped, a system that was down, a join that failed for a subset. If three columns are missing on just the same rows, they're one piece of info absent once. One indicator covers all three rather than three separate ones.
Finally, compare the target between rows with and without the gap. A difference in the target rate between present and missing is direct evidence that the missingness is helpful. It takes one line to check. When it's large, the indicator isn't optional and imputation alone will lose most of the signal. Which is what the last editor on this page shows at 0.77 against 0.987.
None of this needs a plot or a library. Three counts and a check, before any choice about strategy.
What about values that aren't NaN but mean missing? Sentinels like -999, 0, or an empty string are extremely common in exported data. No imputer recognises them. Convert them to NaN first, or the model treats -999 as a very small number.
Does the order of imputation and encoding matter? Yes. Impute cat columns before encoding them, or the encoder treats NaN as its own category - which is at times what you want and should be a choice rather than an accident.
Things to try with Missing Values
Compare the strategies. In the second editor, note that mean fills 34.33 into a column whose other values are 1, 2 and 100.
Recover the link. The fifth editor has KNNImputer land on 8.0 just. Change the neighbours to 1 and see what happens.
Turn the indicator off. The last editor drops from 0.987 to 0.77. That gap is the value of one extra column.
Skip imputation fully. Replace the pipeline in the last editor with HistGradientBoostingClassifier() on the raw data with gaps.
Questions people ask about Missing Values
Should I drop rows or fill them in? Dropping is fine when few rows are affected and the loss is random. Otherwise impute, and keep an indicator if missingness could carry signal.
Which SimpleImputer strategy should I use? Median for skewed numbers, mean for roughly symmetric ones, and most_frequent for categories. Constant works when you want an explicit placeholder.
Why fit the imputer on training data only? Because the fill value is learned. Learning it from test rows leaks information.
What does add_indicator do? It appends binary columns marking which values were missing, so the model can use the pattern of missingness.
When is KNNImputer or IterativeImputer worth it? When columns are related and a simple fill would ignore that. They are slower, so test whether they actually improve the score.
Do all models need imputation? No. HistGradientBoosting models handle NaN natively, but most other estimators raise an error.
Where this leaves you
Median rather than mean for numeric columns, an clear category for cat ones, add_indicator=True almost always, the imputer inside a pipeline so the fill value obeys the split. A check of whether a grad booster could have handled the gaps without any of it.
Check yourself
0 of 4
Answer without scrolling back up.
What does SimpleImputer learn during fit?
One number per column, computed from the data it was fitted on - which is why it must be fitted on the training half only.
Why prefer median over mean for numeric columns?
With values of 1, 2 and 100 the mean is 34.33 - a number nothing like any observation. The median is 2.
What does add_indicator=True do?
It preserves the fact of absence, which is frequently informative. In the last editor it took the score from 0.77 to 0.987.
Which estimator handles NaN without an imputer?
It learns which side of each split the missing rows belong on, which uses the missingness rather than filling over it.
Cheat sheet
Missing Values
Fill gaps in your data with SimpleImputer, understand when the gap itself carries information, and keep imputation inside a pipeline.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.