Drop them, fill them with the mean, fill them with the median — and watch what each choice does to the distribution it came from.
Overview
Why, before what
each tutorial answers "what do I do about NaN". The question that decides the answer is why the value is absent, and there are three cases with three different consequences.
Missing fully at random. The lack has nothing to do with any of it. A form field lost to a transmission error. This is the harmless case, and it's rare.
Missing at random. The lack depends on other columns you've. Younger respondents skip the income question more often, and you've their age. This is recoverable, because the info needed to model the lack is present.
Missing not at random. The lack depends on the missing value itself. High earners decline to state their income. This is the hard case, and it's also the common one.
The data in the visualisation is on purpose the third kind: the largest values are the ones that failed. Look at the two histograms. The grey spread is what existed. The orange one is what each strategy leaves you with.
Handling Missing Values
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
The first question is never which strategy. It is why the values are missing.
Mean or median imputation puts a spike at one value, which shrinks the variance and weakens every correlation the column had.
Dropping rows is only safe when the missingness is unrelated to anything you care about. It rarely is.
A missing-value indicator column keeps the fact that it was missing, which is often the most predictive thing about it.
Handling Missing Values
Four strategies, and why choosing between them starts with a question about the world rather than about the data.
Drop the rows
Select Drop. The right tail is gone, and the mean in the readout has fallen well below the true mean.
Nothing about dropping is wrong in principle. It's unbiased when the values are missing fully at random, and with a handful of affected rows out of many it is a just right reasonable choice.
Here it's a disaster, because the rows removed aren't a random sample — they're the large ones. each statistic found afterwards shows a group that excludes them.
The other cost is math. Dropping any row with any missing value across ten columns each 5% missing drops about 40% of the data, though 95% of each single column is present.
Fill with the mean, or the median
Select Mean, then Median. Both put a spike at a single value in the middle of the spread, and both leave the mean below the truth.
They differ in strength. The mean is dragged by extreme values. The median is not. For a skewed column. Income, house prices, time-to-event — the median is the better default, and for a roughly even column the two are close enough that it rarely matters.
What neither can do is invent info. Imputation makes the dataset rectangular so the model will accept it. It doesn't restore what was lost, and it adds two distortions worth knowing:
Variance shrinks. A pile of the same values has none.
Links weaken. The imputed rows carry no tie to any other column, so they dilute each tie the column really had.
Keep the fact that it was missing
Select Median + indicator. The imputation is the same. What changes is that a second column is added, holding 1 where the value was missing and 0 or else.
This is often the best simple answer, because in the third case — missing not at random — the lack is itself helpful. If high earners decline to answer, then "declined to answer" predicts high earnings, and plain imputation throws that signal away while an indicator preserves it.
It costs one column and no assumptions. It's hard to do worse than the options with it.
Strategy
Keeps rows
Keeps distribution
Keeps the signal in absence
Drop
no
only if MCAR
no
Mean
yes
no, spike at centre
no
Median
yes
no, spike at centre
no
Median + indicator
yes
no
yes
Model-based (kNN, MICE)
yes
better
only with an indicator
More sophisticated options
k-NN imputation fills a value from similar rows. MICE models each column from the others, iteratively. Both preserve ties better than a constant and both cost far more work, and both must be fitted on training data only — they're models, so imputing before the split is leakage.
Also worth knowing: some models handle missing values natively. LightGBM and XGBoost learn a default direction at each split for rows whose value is absent, which is often better than any of it you'd impute by hand.
Why "why" changes the answer
The strategy matters far less than the reason the value is absent, and the same 30%-missing column shows it. When the missing values are the *largest* ones (missing not at random — high earners decline to state income), each method that skips the lack inherits the bias:
values 1-100, top 30 missing
resulting mean
true mean
50.5
after dropping the missing rows
35.5
after mean/median imputation
~35.5 (a spike at the low centre)
Dropping or filling both land near 35.5, understating the truth by fifteen, because the rows removed or flattened are just the high ones. No math on the values that remain can recover values that were never seen. The one strategy that helps adds a *missingness indicator*: a second column marking which rows were absent. In the not-at-random case the lack is itself the signal — "declined to answer" predicts high income. So keeping that flag lets the model use what plain imputation throws away. It costs one column and no assumptions. And dropping carries a second, math cost: with ten columns each 5% missing, 1 - 0.95^10 = 40% of rows have *some* gap. Row-dropping discards 40% of the data though 95% of each column is present.
Where it goes wrong with Handling Missing Values
Imputing before the split. The column mean is found from the test rows too.
Filling with zero without thinking. Zero is a real value in most columns. A temperature of zero isn't a missing temperature.
Dropping rows across many columns at once. A little missingness all over drops a lot of data.
Never checking why. Ten minutes finding out what caused the gap is worth more than any choice of strategy.
Questions people ask about Handling Missing Values
Should I just drop rows with missing values? Only when few rows are affected and the values are missing at random. Otherwise you lose data and can bias the sample.
Mean or median imputation? Median is safer with skewed data or outliers. Mean is fine for symmetric distributions.
Why add a missing-value indicator? The fact that a value is absent often carries signal, such as an unreported income. A binary flag lets the model use that.
Do I impute before or after splitting? After. Learn the fill values from the training set only, ideally inside a pipeline, to avoid leakage.
When should I use model-based imputation? When many features are missing and related to each other. KNN or iterative imputation can recover the relationships, at higher cost and with more risk of leakage.
Which models handle missing values natively? Gradient-boosted trees like HistGradientBoosting, XGBoost and LightGBM do. Most linear models, SVMs and neural networks need imputation first.
Dropping, filling, and the flag that saves you
Whether a gap can be safely filled depends fully on WHY it's missing. Two datasets with the same 30% hole rate. Only one of them is fixable by imputation.
example_01.pyscikit-learn
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
n = 700
X = rng.normal(size=(n, 6))
y = 5 * X[:, 0] + 2 * X[:, 1] - X[:, 2] + rng.normal(0, 2.0, n)
print("feature 0 is the dominant predictor. with no gaps at all: R2 %.4f"
% cross_val_score(Ridge(), X, y, cv=5).mean())
print()
print("CASE 1 -- missing completely at random (a sensor dropped packets).")
Xa = X.copy()
Xa[rng.random(Xa.shape) < 0.30] = np.nan
keep = ~np.isnan(Xa).any(axis=1)
print(" 30%% of cells gone, which costs %d of the %d rows entirely."
% (n - keep.sum(), n))
print(" drop incomplete rows : R2 %7.4f (on %d rows)"
% (cross_val_score(Ridge(), Xa[keep], y[keep], cv=5).mean(), keep.sum()))
for name, imp in (("fill with mean ", SimpleImputer(strategy="mean")),
("fill with median", SimpleImputer(strategy="median")),
("k-NN imputer ", KNNImputer(n_neighbors=5))):
print(" %s : R2 %7.4f"
% (name, cross_val_score(make_pipeline(imp, Ridge()), Xa, y, cv=5).mean()))
print(" HistGradientBoosting : R2 %7.4f <- takes NaN directly"
% cross_val_score(HistGradientBoostingRegressor(random_state=0),
Xa, y, cv=5).mean())
print()
print(" dropping rows is not WRONG here -- the surviving rows are a fair")
print(" sample. it is just wasteful, and it gets worse as columns multiply:")
for cols in (6, 12, 25, 50):
print(" %2d columns at 30%% missing -> %.1f%% of rows survive intact"
% (cols, 100 * 0.7 ** cols))
Output
The next block shows what happens when the data is missing for a reason, and why a missing-value indicator flag is the only thing that recovers the lost signal.
example_02.pyscikit-learn
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
n = 700
X = rng.normal(size=(n, 6))
y = 5 * X[:, 0] + 2 * X[:, 1] - X[:, 2] + rng.normal(0, 2.0, n)
print("CASE 2 -- missing for a reason (high earners skip the income question).")
Xb = X.copy()
hide = X[:, 0] > np.percentile(X[:, 0], 65)
Xb[hide, 0] = np.nan
print(" feature 0 is missing on exactly the %d rows where it is high." % hide.sum())
print(" mean of the values that survived: %7.4f" % np.nanmean(Xb[:, 0]))
print(" mean of the values that vanished: %7.4f" % X[hide, 0].mean())
print(" imputation fills the first number into rows that deserve the second.")
print()
print(" drop incomplete rows : R2 %7.4f (on %d rows, all low earners)"
% (cross_val_score(Ridge(), X[~hide], y[~hide], cv=5).mean(), (~hide).sum()))
print(" mean imputation : R2 %7.4f"
% cross_val_score(make_pipeline(SimpleImputer(), Ridge()), Xb, y, cv=5).mean())
flag = np.column_stack([Xb, np.isnan(Xb[:, 0]).astype(float)])
print(" plus a was-missing flag: R2 %7.4f"
% cross_val_score(make_pipeline(SimpleImputer(), Ridge()), flag, y, cv=5).mean())
print()
print(" the flag is the only thing that lets the model say 'this row is one")
print(" of the hidden ones'. the gap itself carried information, and filling")
print(" it in without a flag throws that information away.")
print(" SimpleImputer(add_indicator=True) adds those columns for you.")
print()
print("so the order of operations is: ask why it is missing, then choose. and")
print("whatever you choose, put it inside the pipeline -- an imputer fitted on")
print("the full dataset has already read the test rows.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does 'missing not at random' mean?
It is the hard case and the common one: dropping such rows removes a biased sample, and imputing a central value pulls every statistic toward the middle.
Why does mean imputation weaken a column's correlations?
A pile of identical values has no variance and no covariance with anything, so it drags every measured relationship toward zero.
Why add a missing-value indicator column?
When absence depends on the value - high earners declining to answer - 'declined to answer' predicts the answer. Plain imputation throws that away for free.
Cheat sheet
Handling Missing Values
Learn four strategies for handling missing data. Watch each method change predictions on the same dataset and pick the one that fits.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.