NaN, None and pd.NA - finding them, counting them, and the choice that matters more than the code.
Overview
Three markers
pandas has more than one representation of "missing", for historical reasons.
np.nan is a float. It is what you get in numeric columns, and it is why an integer column with a gap becomes float64.
None is Python's null. In an object column it stays None; in a float column it is converted to NaN on the way in.
pd.NA is the newer, dtype-agnostic marker used by the nullable extension types (Int64, boolean, string).
You rarely need to care which one you have, because isna() and notna() handle all three. Use them rather than == None or == np.nan, neither of which works — NaN is not equal to anything, including itself.
The one place the difference bites: comparisons with pd.NA return pd.NA rather than False, so filtering a nullable column can behave differently from filtering a float one.
Worth knowing
NaN, None and pd.NA are three markers; isna() is the one test that catches all of them.
Count before deciding: isna().sum() per column, isna().mean() as a share, and isna().any(axis=1).sum() for rows you would lose.
dropna() drops a row if any value is missing. how, thresh and subset loosen that.
Filling with zero moves the mean; filling with the mean keeps it and shrinks the variance. Neither is neutral.
ffill/bfill carry neighbouring values and take a limit; interpolate fits between them.
Missingness is often informative — an indicator column can carry more signal than any fill value.
Missing Data
Finding it, counting it, and the choice that matters more than the code.
Three markers, one test
isna catches all of them, whatever the dtype.
example_01.pypandas
Output
Counting before deciding
Per column, as a share, and how many rows you would lose.
example_02.pypandas
Output
dropna, and how much it takes
The default is stricter than people expect.
example_03.pypandas
Output
fillna, and what each choice costs
Filling is never neutral - it puts numbers in that were not measured.
example_04.pypandas
Output
Filling from neighbours
For ordered data, the value before or after is often the honest guess.
example_05.pypandas
Output
For ordered data, adjacency carries meaning and the neighbouring value is often the best guess.
ffill() carries the last known value forward. This is the standard treatment for a sensor reading or a price that is only recorded when it changes.
bfill() carries the next value backward.
Both leave an edge unfilled — ffill cannot fill a leading gap, bfill cannot fill a trailing one — which is worth checking rather than assuming the column is now complete.
limit=n caps how far a value travels. Without it, a single reading can propagate across a gap of any length, which turns one measurement into a hundred fabricated ones.
interpolate() fits between the surrounding values rather than repeating one, and takes a method for the shape of the fit. It needs a meaningful order, so it belongs on time series and not on arbitrary rows.
Note that fillna(method="ffill") is deprecated; ffill() is the current spelling.
Missing is sometimes the signal
Recording that a value was absent can matter more than replacing it.
example_06.pypandas
Output
Count first
Before choosing a strategy, get three numbers:
df.isna().sum() — how many are missing in each column.
df.isna().mean() — the same as a proportion, which is far easier to judge.
df.isna().any(axis=1).sum() — how many rows would be lost to dropna(). This is usually much larger than any single column's count, because different rows are missing different things.
Those three numbers make the decision. Dropping 2% of rows is a rounding error; dropping 40% changes what the dataset is. A column that is 60% empty probably should not be used at all, and no amount of imputation fixes that.
dropna
The default is stricter than most people expect: df.dropna() drops a row if any value in it is missing.
how="all" only drops rows that are entirely empty, which is the right tool for trailing junk from a spreadsheet export.
thresh=n keeps rows with at least n non-missing values.
subset=["a", "b"] only considers those columns, which is usually what you actually want — you care that the key fields are present, not that every optional field is.
axis=1 drops columns rather than rows, which is the blunt way to remove mostly-empty fields.
fillna
The code is easy. The choice is a statistical decision, and it is not neutral.
Zero shifts the mean toward zero. For a count where missing genuinely means none, it is correct. For a measurement where missing means unknown, it invents data and biases every summary.
The mean preserves the mean by construction and shrinks the variance, because you are adding points with no spread. Anything downstream that cares about dispersion — a standard deviation, a confidence interval, a model that weights by variance — is then quietly wrong. The fourth editor measures this.
The median is more robust to outliers and has the same variance problem.
A sentinel like -1 is honest for categorical codes but must be documented, or someone will average it.
Group-wise fills — filling with the mean of the same city, or the same product — are usually better than a global fill, because they use information you actually have.
Missingness as information
The most commonly missed point: why a value is absent often matters more than what you replace it with.
If a value is missing at random, dropping or imputing is defensible. If it is missing *because of what it would have been* — a sensor that fails at extremes, an income question people skip when the answer is embarrassing, a test not run because the patient was too ill — then both dropping and imputing bias the result, in opposite directions.
The last editor shows the extreme case: a column that is missing exactly when another column takes a particular value. Filling it with the mean would erase the most informative thing in the data.
Adding an indicator column — df["x_missing"] = df["x"].isna().astype(int) — keeps that information while still allowing a fill. It costs one column and is frequently the single most useful feature in a model built on messy data.
A working order
Count. Look at *which* rows are missing, not just how many. Ask whether the missingness looks random. Add an indicator if it does not. Then choose a fill, and write down which one you used — because by the time someone asks, the code will have moved on.
Where missing values come from
Knowing the source usually tells you what to do.
Not collected — the question was not asked, the sensor was not installed. Often missing at random; dropping is defensible.
Not applicable — a spouse's name for an unmarried person. Not missing at all; a sentinel or a separate flag is more honest than NaN.
Failed — a sensor error, a timeout. Frequently correlated with the value, and therefore not missing at random.
Created by an operation — a join that did not match, an alignment mismatch, a shift, a pct_change on the first row, a value outside a cut range. These are the ones people mistake for data problems when they are really code problems.
That last category is worth checking first. If a column gained missing values partway through a pipeline, the cause is upstream in the code, not in the source data.
Missing values in operations
Most aggregations skip missing values by default: sum, mean, max all use skipna=True.
That is different from NumPy, where a single NaN poisons the whole reduction, and it is worth knowing because it changes the denominator. df["x"].mean() divides by the number of present values, not the row count.
skipna=False makes them propagate, which is occasionally what you want when a missing value should invalidate the result.
sum() of an all-missing column returns 0, not NaN, which is defensible and surprising. min_count=1 makes it return NaN instead.
groupbydrops rows whose key is missing, unless dropna=False.
merge treats NaN keys as non-matching in most cases.
Sorting puts them last regardless of direction.
Each of these is reasonable in isolation; together they mean missing data quietly changes several numbers at once, and the totals stop reconciling.
Filling within groups
A global fill uses the least information available. A group fill uses more:
The part that is easy to skip and matters most later.
Record, in the code and in whatever the output is:
How many values were missing per column, before you touched them.
What you did — dropped, filled with what, interpolated how.
Whether you added an indicator.
The reason is that a filled value is indistinguishable from a measured one once it is in the frame. Six months later, nobody can tell which numbers were observed and which were invented, and the analysis cannot be reproduced or corrected.
A frame with an x_imputed boolean column carries that information with the data, which is more robust than a comment.
The order to work in
Count them. Look at which rows they are in. Decide whether the missingness looks random. Add an indicator if it does not. Choose a strategy for each column separately — there is no reason the same one suits every column. Record what you did. Then proceed.
Missing values that pandas created
Worth a separate checklist, because these are code problems rather than data problems and the fix is upstream.
A join that did not match — the added columns are NaN for unmatched rows. indicator=True confirms it.
Alignment on assignment — a Series with different labels assigned to a column fills the unmatched rows.
shift, diff, pct_change — the first row of each has no predecessor.
rolling — leading rows before the window fills.
cut or qcut — values outside the bin edges.
reindex or unstack — combinations that did not occur in the data.
resample upsampling — periods with no observation.
If a column gained missing values partway through a pipeline, one of these is the cause, and filling them is treating a symptom.
Interpolation options
interpolate() defaults to method="linear", which treats the values as evenly spaced regardless of the index.
method="time" uses the actual time gaps, which is what you want on an irregular time series — without it, a value after a three-week gap is weighted the same as one after an hour.
method="nearest", "polynomial", "spline" fit other shapes; the last two need an order.
limit_direction controls whether leading and trailing gaps are filled. By default interpolate fills forward only, so a leading NaN survives.
limit_area="inside" restricts filling to gaps between real observations, which is usually the honest choice — interpolating beyond the ends of your data is extrapolation, and it should be a deliberate decision rather than a default.
Choosing per column
There is no reason one strategy suits every column, and treating them uniformly is usually wrong.
An identifier with gaps is a data problem; the rows probably cannot be used.
A category can take an explicit "unknown" level, which is honest and keeps the rows.
A count where missing means none takes zero.
A measurement takes a group mean, an interpolation, or nothing — depending on why it is missing.
A timestamp rarely takes a fill at all; a missing date usually means the event did not happen.
Writing the decision down per column, in code, is more maintainable than a single fillna(0) across the frame — which is the most common and least defensible choice.
A summary
isna() finds all three markers; equality never works.
Count per column, as a share, and count the rows you would lose, before deciding anything.
dropna() drops on any missing value; subset= is usually what you want.
Zero shifts the mean; the mean shrinks the variance; neither is neutral.
ffill/bfill for ordered data, with a limit.
interpolate(method="time") for irregular series.
Group fills use information a global fill discards.
Add an indicator when missingness might be informative — it often is.
And record what you did, because a filled value is indistinguishable from a measured one the moment it enters the frame.
A closing note
Handling missing data is the part of this track where the code is easiest and the decisions are hardest.
isna, dropna and fillna take a few minutes to learn. What they cannot tell you is whether the values are missing at random, whether the rows you are about to drop share something, or whether filling with a mean is defensible for this particular column.
Those are questions about the data rather than about pandas, and the honest answer is often that missingness is informative — that a value is absent precisely because of what it would have been. In that case both dropping and imputing bias the result, and an indicator column preserves the signal that filling would erase.
The practical minimum is: count them per column and as a share, look at which rows they are in, choose per column rather than uniformly, and record what you did — because once a filled value is in the frame, nothing distinguishes it from a measured one.
Check yourself
0 of 4
Answer without scrolling back up.
Which test finds NaN, None and pd.NA alike?
NaN is not equal to anything including itself, so equality tests never work. isna() handles all three markers.
What does the default `df.dropna()` do?
Stricter than most people expect. how='all', thresh= and subset= loosen it, and subset is usually what you actually want.
What does filling with the column mean do to the variance?
The mean is preserved by construction, so it looks harmless - but anything downstream relying on dispersion is quietly wrong.
Why add an `x_missing` indicator column?
When a value is missing because of what it would have been, both dropping and imputing bias the result. The indicator keeps that signal.
Cheat sheet
Missing Data
You rarely need to care which one you have, because isna() and notna() handle all three. Use them rather than == None or == np.nan, neither of which works — NaN is not equal to anything, including itself.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.