object, category and the nullable types - and why one text column can cost more than the rest of the frame.
Overview
Why this matters early
A frame that does not fit in memory cannot be analysed at all, and the usual reason a frame is larger than expected is not the number of rows. It is one or two columns stored in the widest possible type.
Fixing dtypes is often the difference between a dataset that loads and one that does not, and it takes one line per column.
Worth knowing
An object column holds pointers to Python objects, so memory_usage() undercounts it badly — use deep=True.
category stores each distinct value once plus a small code per row, and pays off only when values repeat.
int64 is the default and usually far wider than needed; pd.to_numeric(..., downcast=...) narrows it.
Nullable Int64 (capital I) holds integers and missing values together, using pd.NA rather than NaN.
A plain integer column with one missing value becomes float64, which is why ids print as 1001.0.
astype raises on bad input; pd.to_numeric(s, errors="coerce") turns the failures into NaN so you can find them.
Dtypes and Memory
object, category and the nullable types.
Every column has a dtype
And the ones inferred from messy data are often not the ones you want.
example_01.pypandas
Output
object columns are expensive
The default memory report does not tell you the truth about them.
example_02.pypandas
Output
category, for repeated text
Store each distinct value once and keep small integer codes per row.
example_03.pypandas
Output
Downcasting numbers
int64 is the default and is usually far wider than the data needs.
example_04.pypandas
Output
int64 holds numbers up to about 9.2 quintillion. A column of ages does not need that.
pd.to_numeric(s, downcast="integer") picks the narrowest signed type that fits; downcast="unsigned" does the same for non-negative data, and downcast="float" gives float32.
The saving is up to 8x for integers and 2x for floats.
The caution is the same as NumPy's: narrow integer types wrap silently on overflow. Downcast when the range is genuinely bounded — ages, day-of-month, a small enum — and leave counters and identifiers alone. An int8 column that later receives 200 does not give you 200.
Nullable dtypes hold integers AND missing
The capital-I Int64 is a different type from int64.
example_05.pypandas
Output
Converting safely
astype raises on bad input; to_numeric can be told what to do instead.
example_06.pypandas
Output
astype(int) raises on the first value it cannot convert, and the message does not tell you which row.
pd.to_numeric(s, errors="coerce") converts what it can and turns the rest into NaN. That is usually the better tool for real data, because it lets you find the offenders:
converted = pd.to_numeric(s, errors="coerce")
bad = s[converted.isna() & s.notna()]
Those are the rows that were present and unconvertible — the stray "N/A", the number with a thousands separator, the value with a trailing space.
errors="raise" is the default and is right when the data is supposed to be clean and you want to know immediately if it is not.
object is the expensive one
A column of strings has dtype object. That means an array of pointers, each leading to a separate Python string somewhere else in memory.
Two costs follow. The obvious one is memory: a pointer plus a Python string object plus the characters, against a few bytes for a number. The less obvious one is speed — operations on object columns fall back to per-element Python, which is the same reason apply is slow.
df.memory_usage() reports 8 bytes per pointer and stops there. For a text column that is a wild underestimate. df.memory_usage(deep=True) follows the pointers and reports the truth, and the gap between the two numbers is often a factor of ten.
df.info(memory_usage="deep") gives the same accounting alongside the dtypes.
Always use deep=True when looking at a frame with text in it. The shallow number is close to meaningless there.
category
When a text column has few distinct values relative to its length — a city, a status, a product code — category is the fix.
It stores the distinct values once in a lookup, and one small integer code per row. A column of 30,000 rows drawn from four cities goes from storing 30,000 strings to storing four strings and 30,000 one-byte codes.
The savings are large, and they compound: group-by and comparison on a category are faster too, because they operate on the codes.
The condition is repetition. A column where every value is distinct — an email address, a UUID — costs more as a category, because you store the codes in addition to all the original values. The rule of thumb is that it pays when the number of distinct values is well under half the row count, and it pays enormously when it is a tiny fraction.
Two behaviours to know. Categories carry an order if you give them one, which is what makes sorting by a grade or a size work properly. And operations that produce a value outside the category set give NaN rather than extending the set, which is occasionally surprising and generally the safer default.
Nullable dtypes
The oldest wart in pandas is that a NumPy integer array cannot represent a missing value, so an integer column with one gap becomes float64 and identifiers start printing as 1001.0.
The nullable extension types fix this. Int64 with a capital I is a different dtype from int64, and it holds integers and missing values together. The missing marker is pd.NA rather than np.nan.
Float64, boolean and string are the equivalents for the other kinds. The nullable boolean is genuinely useful, because plain bool columns also collapse to object when they gain a missing value.
They are not the default, and there are corners where a library downstream expects NumPy types and does not handle them. But for identifiers and counts that must not become floats, they are the right answer.
pd.NA propagates through arithmetic like NaN, and comparisons with it return pd.NA rather than False — a difference from NaN that matters if you are filtering on such a column.
A practical routine
After loading anything:
df.info(memory_usage="deep")
Then, for each object column, nunique() against len(df). Anything with heavy repetition becomes a category. Anything numeric-looking gets to_numeric. Anything that is an identifier gets str at read time so its leading zeros survive.
That is usually a handful of lines, and on a large frame it routinely cuts memory by more than half.
Seeing where the memory goes
df.memory_usage(deep=True).sort_values(ascending=False) ranks the columns. On most real frames one or two text columns dominate, and everything else is noise.
That ranking tells you where to spend effort. Converting a column that holds 2% of the memory is not worth the risk of changing a dtype.
df.info(memory_usage="deep") gives the same information alongside dtypes and null counts, which is usually the more convenient single call.
Categories in more detail
astype("category") builds the category set from the values present.
Two consequences follow from the set being fixed.
Assigning a value outside the set fails or produces NaN, depending on the operation. Adding a new city to a categorical column requires cat.add_categories first.
Combining two categoricals with different sets gives object unless the sets match. This bites when concatenating frames from different files, each of which saw a different subset of the categories. pd.api.types.union_categoricals handles it, or convert after concatenating rather than before.
cat.set_categories([...], ordered=True) gives an order, which is what makes sorting by size or grade work correctly rather than alphabetically. cat.as_ordered() and comparison operators then behave as you would expect.
cat.remove_unused_categories() shrinks the set after filtering, which otherwise keeps every original category and can leave empty groups in a group-by.
Group-by on a categorical includes every category by default, even absent ones, producing rows of zeros. observed=True restricts it to categories actually present, and it is worth passing deliberately since the default has changed across versions.
What float32 costs
float32 halves memory and keeps about seven significant digits.
For data that came from a sensor with three, that is ample. For accumulating a long sum it is not, and the fix is to accumulate in a wider type: s.sum(dtype="float64") reads a narrow column and adds in a wide accumulator.
float16 exists and is rarely a good idea outside deep learning — three significant digits is not much, and most operations upcast it anyway.
Nullable types in practice
The extension types — Int64, Float64, boolean, string — fix the missing-value gaps in the NumPy-backed dtypes.
Three practical notes.
pd.NA propagates through comparisons as pd.NA rather than False. Filtering a nullable column therefore needs care: df[df["x"] > 1] drops the missing rows either way, but a mask containing pd.NA cannot always be used directly, and .fillna(False) on the mask is the fix.
Some libraries downstream expect NumPy dtypes and do not handle extension types. Converting back with astype("float64") at the boundary is sometimes necessary.
convert_dtypes() converts a whole frame to the best available nullable types in one call, which is a quick way to see what it would look like.
A conversion routine
After loading, and before anything else:
for c in df.select_dtypes("object"):
if df[c].nunique() / len(df) < 0.5:
df[c] = df[c].astype("category")
for c in df.select_dtypes("integer"):
df[c] = pd.to_numeric(df[c], downcast="integer")
Two loops, run once, that routinely halve the memory of a real frame.
The threshold is a judgement rather than a rule. Well under half distinct is a clear win; near half is marginal; mostly distinct is a loss.
And the general principle from the numpy track applies here too: set the dtype as early as possible, ideally at read time, because every conversion afterwards allocates a second copy of the column.
A diagnosis routine
When a frame is unexpectedly large or slow, four lines find the cause:
The first two find which columns dominate. The third says whether they are candidates for category — low cardinality — or identifiers that will not benefit. The fourth shows average string length, which explains a column that is large despite having few rows.
Almost always the answer is one or two object columns, and almost always one of them should be a category.
dtype mistakes that cause wrong answers
Memory is the visible cost. These are the ones that change results.
An identifier read as an integer loses leading zeros, and two different ids can collide once the zeros are gone.
A numeric column stored as object compares as strings: "100" < "20" is True. Sorting, comparison and max all silently give the wrong answer, and nothing raises.
A narrow integer that overflows wraps. int8 holding 200 does not hold 200.
Mixed types in one column make it object, and any aggregation either fails or produces something meaningless.
A float used as a key in a merge or a group-by. Floating-point equality is unreliable, and two values that print identically may not match.
Each of these is caught by looking at dtypes immediately after loading, which is why that habit appears in three separate modules.
When to convert
At read time if you can — dtype= in read_csv. No extra copy, no window where the column is wrong.
Immediately after loading otherwise, before any analysis, so every later operation sees the right types.
Not repeatedly. Each astype allocates a full copy of the column. Converting inside a function that runs per group or per file is a common and invisible cost.
A summary
Text columns hold pointers; use deep=True to see their real size.
category for repeated values, and only for repeated values.
Downcast numerics when the range is genuinely bounded.
Int64, boolean and string hold missing values without changing type.
to_numeric(errors="coerce") converts what it can and shows you what it could not.
Set types as early as possible, and check dtypes right after loading — it is one line and it prevents a category of silent wrongness.
A closing note
Dtypes are the quiet determinant of whether a pandas script is fast, correct, and able to load its data at all.
The memory story is simple: text columns hold pointers to Python strings and dominate most real frames, category collapses them when values repeat, and narrow numeric types halve or quarter the rest. Two loops run once after loading routinely cut a frame's footprint in half.
The correctness story matters more. A numeric column stored as object compares as strings, so "100" < "20" is true and every sort and maximum is silently wrong. An identifier read as an integer loses its leading zeros permanently. An integer column with one gap becomes float, and ids start printing with a decimal point.
None of those raise. All of them are visible in one line of df.dtypes, run immediately after loading — which is why that habit appears in several modules and is worth more than any other single check in this track.
Check yourself
0 of 4
Answer without scrolling back up.
Why does `memory_usage()` undercount a text column?
Use deep=True, which follows the pointers. The gap between the two numbers is often a factor of ten.
When does converting a column to `category` cost more than it saves?
It pays when distinct values are well under half the row count, and enormously when they are a tiny fraction.
What is the difference between `int64` and `Int64`?
It fixes the oldest wart in pandas: a plain integer column with one gap becomes float64, so ids print as 1001.0
How do you convert a messy text column to numbers without stopping at the first bad value?
It turns failures into NaN so you can find them: s[converted.isna() & s.notna()] gives the rows that were present and unconvertible.
Cheat sheet
Dtypes and Memory
A frame that does not fit in memory cannot be analysed at all, and the usual reason a frame is larger than expected is not the number of rows. It is one or two columns stored in the widest possible type.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.