NaN, None and pd.NA - finding them, counting them, and the choice that matters more than the code.
Overview
Three markers
Missing data appears in almost every real dataset. A sensor drops a reading, a survey question goes unanswered, a join brings in rows that have no match. How you handle those gaps shapes every downstream result. An uncounted NaN in a mean, a silent drop in a merge, a column that turned float because of one absent value -- each of these is a missing-data problem wearing a different face.
pandas uses more than one marker for "missing" because of how Python and NumPy interact. NumPy has NaN for floats but nothing for integers or booleans. Python has None, which works in object columns but not in numeric ones. The newer nullable types use pd.NA, which works everywhere but behaves differently in boolean logic. Knowing which marker you have matters less than knowing that isna catches all three.
The first question to answer is how much data is missing and where. A column with five percent NaN needs a different plan from a column with eighty percent NaN. A column where gaps cluster in one time period tells a different story from one where gaps are scattered at random.
Before choosing a fill strategy, you need those numbers. This module starts with counting, then covers the three main options: dropping rows, filling with a fixed value, and interpolating. Each has a correct use case and a common misuse. Dropping rows when the remaining data is biased gives biased results. Filling with the mean when the data has groups gives a mean that belongs to no group.
The goal is not to eliminate every NaN. It is to make a deliberate choice about each one, knowing how that choice affects the analysis.
pandas has more than one form of "missing", for historical reasons.
np.nan is a float. It's what you get in numeric columns. It's why an int column with a gap turns float64.
None is Python's null. In an object column it stays None. In a float column it's converted to NaN on the way in.
pd.NA is the newer, dtype-agnostic marker used by the nullable extension types (Int64, boolean, string).
You rarely need to care which one you've. isna() and notna() handle all three. Use them rather than == None or == np.nan, neither of which works. NaN isn't equal to anything, including itself.
The one place the gap bites: checks with pd.NA return pd.NA rather than False. Filtering a nullable column can behave differently from filtering a float one.
Worth knowing
NaN, None and pd.NA are three markers; isna() is the one test that catches all of them.
Count before deciding: isna().sum() per column, isna().mean() as a share, and isna().any(axis=1).sum() for rows you would lose.
dropna() drops a row if any value is missing. how, thresh and subset loosen that.
Filling with zero moves the mean; filling with the mean keeps it and shrinks the variance. Neither is neutral.
ffill/bfill carry neighbouring values and take a limit; interpolate fits between them.
Missingness is often informative — an indicator column can carry more signal than any fill value.
Missing Data
Finding it, counting it, and the choice that matters more than the code.
Three markers, one test
isna catches all of them, whatever the dtype.
example_01.pypandas
import pandas as pd
import numpy as np
f = pd.Series([1.0, np.nan, 3.0])
o = pd.Series(["a", None, "c"])
n = pd.Series([1, None, 3], dtype="Int64")
for name, s in [("float + NaN", f), ("object + None", o), ("Int64 + NA", n)]:
print("%-14s dtype %-8s isna: %s" % (name, str(s.dtype), list(s.isna())))
print()
print("None became NaN in the float column:", repr(f[1]))
print("None stayed None in the object column:", repr(o[1]))
print("Int64 uses pd.NA:", repr(n[1]))
print()
print("isna() is the one test that works for all three.")
Output
Counting before deciding
Per column, as a share, and how many rows you would lose.
example_02.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"a": [1, np.nan, 3, 4, 5],
"b": [np.nan, np.nan, np.nan, 4, 5],
"c": [1, 2, 3, 4, 5],
})
print("count per column:"); print(df.isna().sum())
print()
print("share per column:"); print((df.isna().mean() * 100).round(0))
print()
rows_lost = int(df.isna().any(axis=1).sum())
print("rows with any missing :", rows_lost, "of", len(df),
"(%.0f%%)" % (100 * rows_lost / len(df)))
print()
print("Dropping rows loses 40% here, and column b is 60% empty.")
print("Those two numbers decide the strategy, not a rule of thumb.")
Output
dropna, and how much it takes
The default is stricter than people expect.
example_03.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"a": [1, np.nan, 3],
"b": [np.nan, 2, 3],
})
print("original:"); print(df)
print()
print("dropna() drops a row if ANY value is missing:")
print(df.dropna())
print()
print("how='all' only drops rows that are entirely missing:")
print(df.dropna(how="all"))
print()
print("thresh=1 keeps rows with at least 1 real value:")
print(df.dropna(thresh=1))
print()
print("subset limits which columns count:")
print(df.dropna(subset=["a"]))
Output
fillna, and what each choice costs
Filling is never neutral - it puts numbers in that were not measured.
example_04.pypandas
import pandas as pd
import numpy as np
s = pd.Series([10.0, np.nan, np.nan, 40.0, 50.0])
print("original :", list(s))
print("mean of real :", round(float(s.mean()), 2))
print()
zero = s.fillna(0)
mean = s.fillna(s.mean())
print("fillna(0) :", list(zero), " mean now", round(float(zero.mean()), 2))
print("fillna(mean):", [round(v, 1) for v in mean],
" mean now", round(float(mean.mean()), 2))
print()
print("std of real :", round(float(s.std()), 2))
print("std after mean-fill:", round(float(mean.std()), 2), "<- shrunk")
print()
print("Zero moves the mean. The mean keeps it and shrinks the spread.")
Output
Filling from neighbours
For ordered data, the value before or after is often the honest guess.
example_05.pypandas
import pandas as pd
import numpy as np
s = pd.Series([1.0, np.nan, np.nan, 4.0, np.nan],
index=pd.date_range("2024-01-01", periods=5))
print("original :", [None if pd.isna(v) else v for v in s])
print("ffill :", list(s.ffill()))
print("bfill :", list(s.bfill()))
print("interpolate:", [round(v, 2) for v in s.interpolate()])
print()
print("ffill leaves a leading gap unfilled, bfill a trailing one:")
print(" ffill first value:", s.ffill().iloc[0])
print()
print("limit caps how far a fill travels:")
print(" ffill(limit=1):", [None if pd.isna(v) else v for v in s.ffill(limit=1)])
Output
For sorted data, adjacency carries meaning and the neighbouring value is often the best guess.
ffill() carries the last known value forward. This is the standard treatment for a sensor reading or a price that is only recorded when it changes.
bfill() carries the next value backward.
Both leave an edge unfilled. ffill can't fill a leading gap, bfill can't fill a trailing one — which is worth checking rather than given the column is now full.
limit=n caps how far a value travels. Without it, a single reading can pass across a gap of any length. This Turns one reading into a hundred fabricated ones.
interpolate() fits between the nearby values rather than repeating one. Takes a method for the shape of the fit. It needs a real order. It belongs on time series and not on any rows.
Note that fillna(method="ffill") is deprecated. ffill() is the current spelling.
Missing is sometimes the signal
Recording that a value was absent can matter more than replacing it.
example_06.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"income": [100.0, np.nan, 300.0, np.nan, 500.0],
"responded": [1, 0, 1, 0, 1],
})
df["income_missing"] = df["income"].isna().astype(int)
print(df)
print()
corr = df["income_missing"].corr(df["responded"])
print("correlation between 'was missing' and 'responded': %.2f" % corr)
print()
print("-1.00 is a perfect NEGATIVE correlation: income is missing")
print("exactly when the person did not respond. Filling it with the")
print("mean would erase the most informative thing in the column.")
Output
Count first
Before choosing a plan, get three numbers:
df.isna().sum() — how many are missing in each column.
df.isna().mean() — the same as a proportion. This Is far easier to judge.
df.isna().any(axis=1).sum() — how many rows would be lost to dropna(). This is often much larger than any single column's count. Other rows are missing other things.
Those three numbers make the choice. Dropping 2% of rows is a rounding error. Dropping 40% changes what the dataset is. A column that is 60% empty probably shouldn't be used at all. No amount of imputation fixes that.
dropna
The default is stricter than most people expect: df.dropna() drops a row if any value in it's missing.
how="all" only drops rows that are fully empty. This Is the right tool for trailing junk from a spreadsheet export.
thresh=n keeps rows with at least n non-missing values.
subset=["a", "b"] only considers those columns. This Is often what you really want. You care that the key fields are present, not that each optional field is.
axis=1 drops columns rather than rows. This Is the blunt way to drop mostly-empty fields.
fillna
The code is easy. The choice is a statistical choice, and it isn't neutral.
Zero shifts the mean toward zero. For a count where missing truly means none, it's correct. For a reading where missing means unknown, it invents data and biases each summary.
The mean preserves the mean by build and shrinks the variance. You're adding points with no spread. Anything downstream that cares about dispersion. A standard deviation, a trust interval, a model that weights by variance. Is then quietly wrong. The fourth editor measures this.
The median is more solid to outliers and has the same variance problem.
A sentinel like -1 is honest for cat codes but must be documented, or someone will mean it.
Group-wise fills — filling with the mean of the same city, or the same product — are often better than a global fill. They use data you really have.
Missingness as information
The most often missed point: why a value is absent often matters more than what you replace it with.
If a value is missing at random, dropping or imputing is defensible. If it's missing *because of what it would have been*. A sensor that fails at extremes, an income question people skip when the answer is embarrassing, a test not run because the patient was too ill. Then both dropping and imputing bias the result, in opposite directions.
The last editor shows the extreme case: a column that is missing just when another column takes a given value. Filling it with the mean would erase the most helpful thing in the data.
Adding an indicator column — df["x_missing"] = df["x"].isna().astype(int) — keeps that data while still allowing a fill. It costs one column and is often the single most handy feature in a model built on messy data.
A working order
Count. Look at *which* rows are missing, not just how many. Ask whether the missingness looks random. Add an indicator if it doesn't. Then choose a fill. Write down which one you used. Because by the time someone asks, the code will have moved on.
Where missing values come from
Knowing the source often tells you what to do.
Not collected — the question wasn't asked, the sensor wasn't installed. Often missing at random. Dropping is defensible.
Not applicable — a spouse's name for an unmarried person. Not missing at all. A sentinel or a split flag is more honest than NaN.
Failed — a sensor error, a timeout. often linked with the value. So not missing at random.
Created by an step — a join that didn't match, an alignment mismatch, a shift, a pct_change on the first row, a value outside a cut range. These are the ones people mistake for data problems when they're really code problems.
That last group is worth checking first. If a column gained missing values partway through a pipeline, the cause is upstream in the code, not in the source data.
Missing values in operations
Most aggregations skip missing values by default: sum, mean, max all use skipna=True.
That is other from NumPy. Here A single NaN poisons the whole reduction. It's worth knowing because it changes the denominator. df["x"].mean() cuts by the number of present values, not the row count.
skipna=False makes them pass. This Is at times what you want when a missing value should invalidate the result.
sum() of an all-missing column returns 0, not NaN. This Is defensible and surprising. min_count=1 makes it return NaN instead.
groupbydrops rows whose key is missing, unless dropna=False.
merge treats NaN keys as non-matching often.
Sorting puts them last even so of direction.
Each of these is fair in alone. Together they mean missing data quietly changes a few numbers at once, and the totals stop merging.
Filling within groups
A global fill uses the least data on hand. A group fill uses more:
The part that is easy to skip and matters most later.
Record, in the code and in what the output is:
How many values were missing per column, before you touched them.
What you did — dropped, filled with what, interpolated how.
Whether you added an indicator.
a filled value is the same from a measured one once it's in the frame. Six months later, no one can tell which numbers were seen and which were invented. The analysis can't be copied or corrected.
A frame with an x_imputed boolean column carries that data with the data. This Is more solid than a comment.
The order to work in
Count them. Look at which rows they're in. Decide whether the missingness looks random. Add an indicator if it doesn't. Choose a plan for each column apart. There's no reason the same one suits each column. Record what you did. Then proceed.
Missing values that pandas created
Worth a split checklist, because these are code problems rather than data problems and the fix is upstream.
A join that didn't match. The added columns are NaN for unmatched rows. indicator=True confirms it.
Alignment on set — a Series with other labels assigned to a column fills the unmatched rows.
**shift, diff, pct_change** — the first row of each has no predecessor.
**rolling** — leading rows before the window fills.
**cut or qcut** — values outside the bin edges.
**reindex or unstack** — mixes that didn't occur in the data.
**resample upsampling** — periods with no finding.
If a column gained missing values partway through a pipeline, one of these is the cause. Filling them is treating a symptom.
Interpolation options
interpolate() defaults to method="linear". This Treats the values as evenly spaced even so of the index.
method="time" uses the actual time gaps. This Is what you want on an irregular time series. Without it, a value after a three-week gap is weighted the same as one after an hour.
method="nearest", "polynomial", "spline" fit other shapes. The last two need an order.
limit_direction controls whether leading and trailing gaps are filled. By default interpolate fills forward only. A leading NaN lives.
limit_area="inside" restricts filling to gaps between real findings. This Is often the honest choice. Interpolating beyond the ends of your data is extrapolation. It should be a deliberate choice rather than a default.
Choosing per column
There's no reason one plan suits each column, and treating them uniformly is often wrong.
An identifier with gaps is a data problem. The rows probably can't be used.
A group can take an clear "unknown" level. This Is honest and keeps the rows.
A count where missing means none takes zero.
A reading takes a group mean, an blending, or nothing. Based on why it's missing.
A timestamp rarely takes a fill at all. A missing date often means the event didn't happen.
Writing the choice down per column, in code, is more maintainable than a single fillna(0) across the frame. Which is the most common and least defensible choice.
A summary
isna() finds all three markers. Equality never works.
Count per column, as a share, and count the rows you'd lose, before deciding anything.
dropna() drops on any missing value. subset= is often what you want.
Zero shifts the mean. The mean shrinks the variance. Neither is neutral.
ffill/bfill for sorted data, with a limit.
interpolate(method="time") for irregular series.
Group fills use data a global fill discards.
Add an indicator when missingness might be helpful — it often is.
And record what you did. A filled value is the same from a measured one the moment it enters the frame.
Questions people ask about Missing Data
What is the difference between NaN, None and NaT? NaN is the float missing marker, NaT is the missing marker for datetimes, and None is Python's null in object columns. pandas treats all of them as missing in isna().
Should I drop or fill missing values? Drop when rows with gaps are few and random. Fill when dropping would lose too much data, and choose a fill that fits the meaning, such as zero for counts or the median for measurements.
Why does df == np.nan not find the missing values? NaN is not equal to anything, including itself, so the comparison is always False. Use df.isna().
How do I fill missing values using the previous value? Use df['col'].ffill() for forward-fill, or bfill() for the next value. These suit ordered data such as time series.
How does a missing value affect sum and mean? By default pandas skips missing values in sum, mean and similar reductions. Pass skipna=False to get NaN instead.
How can I fill with each group's own average? Use df['col'].fillna(df.groupby('g')['col'].transform('mean')). transform returns a value for every row, so the fill lines up.
A closing note on Missing Data
Handling missing data is the part of this track where the code is easiest and the choices are hardest.
isna, dropna and fillna take a few minutes to learn. What they can't tell you is whether the values are missing at random, whether the rows you're about to drop share a thing, or whether filling with a mean is defensible for this given column.
Those are questions about the data rather than about pandas. The honest answer is often that missingness is helpful. That a value is absent just because of what it would have been. In that case both dropping and imputing bias the result. An indicator column preserves the signal that filling would erase.
The hands-on min is: count them per column and as a share, look at which rows they're in, choose per column rather than uniformly. Record what you did. Because once a filled value is in the frame, nothing sets apart it from a measured one.
Check yourself
0 of 4
Answer without scrolling back up.
Which test finds NaN, None and pd.NA alike?
NaN is not equal to anything including itself, so equality tests never work. isna() handles all three markers.
What does the default `df.dropna()` do?
Stricter than most people expect. how='all', thresh= and subset= loosen it, and subset is usually what you actually want.
What does filling with the column mean do to the variance?
The mean is preserved by construction, so it looks harmless - but anything downstream relying on dispersion is quietly wrong.
Why add an `x_missing` indicator column?
When a value is missing because of what it would have been, both dropping and imputing bias the result. The indicator keeps that signal.
Cheat sheet
Missing Data
Find, count and handle missing values in a pandas DataFrame with isna, dropna and fillna. Pick a strategy based on why the data is missing.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.