object, category and the nullable types - and why one text column can cost more than the rest of the frame.
Overview
Why this matters early
The dtype of each column decides three things: how much memory the column uses, what operations work on it, and how missing values behave. Getting the dtype right at load time is cheap. Fixing it after building logic on top of the wrong type is expensive and error-prone.
Memory is the most visible consequence. A column of country codes stored as object takes roughly ten times the memory of the same column stored as category. Multiply that by a few such columns and a large dataset, and the frame may not fit in memory at all. The fix is often a single astype call per column.
Speed follows memory. Operations on object columns fall back to per-element Python. Operations on native numeric or categorical columns use compiled code. The difference is not subtle. It can be a factor of ten on a group-by or a value_counts, simply from changing the dtype.
Missing-value handling also depends on the dtype. Classic NumPy int columns cannot hold NaN, so a single missing value promotes the entire column to float64. That changes the printed output and can break downstream comparisons. The nullable integer types solve this, but they require an explicit dtype at read time.
Checking dtypes is part of the inspection routine for a reason. A quick look at the output of df.dtypes after loading tells you whether a column is the type you expected. If it is object when you expected int64, something in the data prevented the inference, and that something is worth finding now rather than later.
This module covers the common dtype problems, how to diagnose them, and how to fix them with astype, pd.to_numeric, and the dtype argument at read time.
A frame that doesn't fit in memory can't be analysed at all. The usual reason a frame is larger than expected isn't the number of rows. It's one or two columns stored in the widest possible type.
Fixing dtypes is often the gap between a dataset that loads and one that doesn't. It takes one line per column.
Worth knowing
An object column holds pointers to Python objects, so memory_usage() undercounts it badly — use deep=True.
category stores each distinct value once plus a small code per row, and pays off only when values repeat.
int64 is the default and usually far wider than needed; pd.to_numeric(..., downcast=...) narrows it.
Nullable Int64 (capital I) holds integers and missing values together, using pd.NA rather than NaN.
A plain integer column with one missing value becomes float64, which is why ids print as 1001.0.
astype raises on bad input; pd.to_numeric(s, errors="coerce") turns the failures into NaN so you can find them.
Dtypes and Memory
object, group and the nullable types.
Every column has a dtype
And the ones inferred from messy data are often not the ones you want.
example_01.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"i": [1, 2, 3],
"f": [1.5, 2.5, 3.5],
"s": ["a", "b", "c"],
"b": [True, False, True],
"missing_int": [1, 2, np.nan],
})
print(df.dtypes)
print()
print("Two to notice:")
print(" s -> object: a column of POINTERS to Python strings")
print(" missing_int -> float64: NumPy ints cannot hold NaN")
Output
object columns are expensive
The default memory report does not tell you the truth about them.
example_02.pypandas
import pandas as pd
n = 30_000
df = pd.DataFrame({
"num": range(n),
"city": ["pune", "delhi", "mumbai", "goa"] * (n // 4),
})
shallow = df.memory_usage().sum()
deep = df.memory_usage(deep=True).sum()
print("memory_usage() : %8.1f KB" % (shallow / 1e3))
print("memory_usage(deep=True) : %8.1f KB" % (deep / 1e3))
print()
print("per column, deep:")
print((df.memory_usage(deep=True) / 1e3).round(1))
print()
print("The shallow number counts 8 bytes per pointer. The deep one")
print("follows the pointers and counts the strings themselves.")
Output
category, for repeated text
Store each distinct value once and keep small integer codes per row.
example_03.pypandas
import pandas as pd
n = 30_000
s = pd.Series(["pune", "delhi", "mumbai", "goa"] * (n // 4))
c = s.astype("category")
print("object : %8.1f KB" % (s.memory_usage(deep=True) / 1e3))
print("category : %8.1f KB" % (c.memory_usage(deep=True) / 1e3))
print("saving : %.0f%%" % (100 * (1 - c.memory_usage(deep=True) /
s.memory_usage(deep=True))))
print()
print("categories:", list(c.cat.categories))
print("codes :", list(c.cat.codes[:6]), "...")
print()
print("It only pays when values repeat. Unique values per row cost MORE")
print("as a category, because you store the codes as well as the values.")
Output
Downcasting numbers
int64 is the default and is usually far wider than the data needs.
example_04.pypandas
import pandas as pd
import numpy as np
s = pd.Series(range(50_000))
print("int64 :", s.memory_usage(deep=True) // 1000, "KB")
small = pd.to_numeric(s, downcast="unsigned")
print("downcast:", small.dtype, small.memory_usage(deep=True) // 1000, "KB")
print()
f = pd.Series(np.linspace(0, 1, 50_000))
print("float64:", f.memory_usage(deep=True) // 1000, "KB")
print("float32:", f.astype("float32").memory_usage(deep=True) // 1000, "KB")
print()
print("Downcast only when the range is known and bounded. An int8 column")
print("that later receives 200 wraps or raises, depending on how it got there.")
Output
int64 holds numbers up to about 9.2 quintillion. A column of ages doesn't need that.
pd.to_numeric(s, downcast="int") picks the narrowest signed type that fits. downcast="unsigned" does the same for non-minus data, and downcast="float" gives float32.
The saving is up to 8x for ints and 2x for floats.
The caution is the same as NumPy's: narrow int types wrap silently on overflow. Downcast when the range is truly bounded. Ages, day-of-month, a small enum. And leave counters and identifiers alone. An int8 column that later receives 200 doesn't give you 200.
Nullable dtypes hold integers AND missing
The capital-I Int64 is a different type from int64.
example_05.pypandas
import pandas as pd
import numpy as np
plain = pd.Series([1, 2, np.nan])
nullable = pd.Series([1, 2, None], dtype="Int64")
print("plain :", list(plain), plain.dtype, "<- became float")
print("nullable :", list(nullable), nullable.dtype, "<- stayed integer")
print()
print("the missing marker is pd.NA, not NaN:")
print(" nullable[2] is pd.NA:", nullable[2] is pd.NA)
print()
print("arithmetic propagates it:")
print(" ", list(nullable + 1))
print()
print("Use these when an id or a count must not turn into a float.")
Output
Converting safely
astype raises on bad input; to_numeric can be told what to do instead.
example_06.pypandas
import pandas as pd
s = pd.Series(["1", "2", "oops", "4"])
try:
s.astype(int)
except ValueError as e:
print("astype(int) ->", str(e)[:48])
print()
coerced = pd.to_numeric(s, errors="coerce")
print("to_numeric(errors='coerce'):", list(coerced))
print(" the bad value became NaN instead of stopping everything")
print()
print("find what failed:")
print(" rows that would not convert:", list(s[coerced.isna()]))
print()
print("dtype after coercion:", coerced.dtype)
Output
astype(int) raises on the first value it can't convert. The message doesn't tell you which row.
pd.to_numeric(s, errors="coerce") converts what it can and turns the rest into NaN. That is often the better tool for real data. It lets you find the offenders:
converted = pd.to_numeric(s, errors="coerce")
bad = s[converted.isna() & s.notna()]
Those are the rows that were present and unconvertible. The stray "N/A", the number with a thousands separator, the value with a trailing space.
errors="raise" is the default and is right when the data is supposed to be clean and you want to know at once if it isn't.
object is the expensive one
A column of strings has dtype object. That means an array of pointers, each leading to a split Python string one spot else in memory.
Two costs follow. The obvious one is memory. A pointer plus a Python string object plus the characters, against a few bytes for a number. The less obvious one is speed. Steps on object columns fall back to per-element Python. This Is the same reason apply is slow.
df.memory_usage() reports 8 bytes per pointer and stops there. For a text column that is a wild underestimate. df.memory_usage(deep=True) follows the pointers and reports the truth. The gap between the two numbers is often a factor of ten.
df.info(memory_usage="deep") gives the same accounting alongside the dtypes.
Always use deep=True when looking at a frame with text in it. The shallow number is close to meaningless there.
category
When a text column has few distinct values scaled to its length. A city, a status, a product code. group is the fix.
It stores the distinct values once in a lookup. One small int code per row. A column of 30,000 rows drawn from four cities goes from storing 30,000 strings to storing four strings and 30,000 one-byte codes.
The savings are large, and they compound: group-by and compare on a group are faster too. They operate on the codes.
The rule is repetition. A column where each value is distinct. An email address, a UUID. Costs more as a group. You store the codes plus all the first values. The rule of thumb is that it pays when the number of distinct values is well under half the row count. It pays enormously when it's a tiny fraction.
Two actions to know. groups carry an order if you give them one. This Is what makes sorting by a grade or a size work properly. And steps that give a value outside the group set give NaN rather than extending the set. This Is at times surprising and often the safer default.
Nullable dtypes
The oldest wart in pandas is that a NumPy int array can't represent a missing value. An int column with one gap turns float64 and identifiers start printing as 1001.0.
The nullable extension types fix this. Int64 with a capital I is a other dtype from int64. It holds ints and missing values together. The missing marker is pd.NA rather than np.nan.
Float64, boolean and string are the equivalents for the other kinds. The nullable boolean is truly handy. Plain bool columns also collapse to object when they gain a missing value.
They aren't the default, and there are corners where a library downstream expects NumPy types and doesn't handle them. But for identifiers and counts that must not become floats, they're the right answer.
pd.NA passes through math like NaN. Checks with it return pd.NA rather than False. A gap from NaN that matters if you're filtering on such a column.
A practical routine
After loading anything:
df.info(memory_usage="deep")
Then, for each object column, nunique() against len(df). Anything with heavy repetition turns a group. Anything numeric-looking gets to_numeric. Anything that is an identifier gets str at read time so its leading zeros live.
That is often a handful of lines. On a large frame it routinely cuts memory by more than half.
Seeing where the memory goes
df.memory_usage(deep=True).sort_values(ascending=False) ranks the columns. On most real frames one or two text columns dominate, and all else is noise.
That ranking tells you where to spend effort. Converting a column that holds 2% of the memory isn't worth the risk of changing a dtype.
df.info(memory_usage="deep") gives the same data alongside dtypes and null counts. This Is often the more convenient single call.
Categories in more detail
astype("group") builds the group set from the values present.
Two results follow from the set being fixed.
Assigning a value outside the set fails or makes NaN, based on the step. Adding a new city to a cat column needs cat.add_categories first.
Mixing two cats with other sets gives object unless the sets match. This bites when concatenating frames from other files, each of which saw a other subset of the groups. pd.api.types.union_categoricals handles it, or convert after concatenating rather than before.
cat.set_categories([...], sorted=True) gives an order. This Is what makes sorting by size or grade work right rather than alphabetically. cat.as_ordered() and compare ops then behave as you'd expect.
cat.remove_unused_categories() shrinks the set after filtering. This Else keeps each first group and can leave empty groups in a group-by.
Group-by on a cat includes each group by default, even absent ones, making rows of zeros. seen=True restricts it to groups really present. It's worth passing on purpose since the default has changed across versions.
What float32 costs
float32 halves memory and keeps about seven large digits.
For data that came from a sensor with three, that is ample. For building up a long sum it isn't. The fix is to build up in a wider type: s.sum(dtype="float64") reads a narrow column and adds in a wide accumulator.
float16 exists and is rarely a good idea outside deep learning. Three large digits isn't much. Most steps upcast it anyway.
Nullable types in practice
The extension types — Int64, Float64, boolean, string — fix the missing-value gaps in the NumPy-backed dtypes.
Three hands-on notes.
pd.NA passes through checks as pd.NA rather than False. Filtering a nullable column so needs care: df[df["x"] > 1] drops the missing rows either way. A mask holding pd.NA can't always be used right. .fillna(False) on the mask is the fix.
Some libraries downstream expect NumPy dtypes and don't handle extension types. Converting back with astype("float64") at the boundary is at times needed.
convert_dtypes() converts a whole frame to the best on hand nullable types in one call. This Is a quick way to see what it would look like.
A conversion routine
After loading, and before anything else:
for c in df.select_dtypes("object"):
if df[c].nunique() / len(df) < 0.5:
df[c] = df[c].astype("category")
for c in df.select_dtypes("integer"):
df[c] = pd.to_numeric(df[c], downcast="integer")
Two loops, run once, that routinely halve the memory of a real frame.
The threshold is a judgement rather than a rule. Well under half distinct is a clear win. Near half is marginal. Mostly distinct is a loss.
And the general principle from the numpy track applies here too. Set the dtype as early as possible, ideally at read time. Each conversion later allocates a second copy of the column.
A diagnosis routine
When a frame is unexpectedly large or slow, four lines find the cause:
The first two find which columns dominate. The third says whether they're picks for group. Low cardinality. Or identifiers that won't gain. The fourth shows mean string length. This Shows a column that is large despite having few rows.
Almost always the answer is one or two object columns. Almost always one of them should be a group.
dtype mistakes that cause wrong answers
Memory is the visible cost. These are the ones that change results.
An identifier read as an int loses leading zeros. Two other ids can collide once the zeros are gone.
**A numeric column stored as object** checks as strings: "100" < "20" is True. Sorting, compare and max all silently give the wrong answer, and nothing raises.
A narrow int that overflows wraps. int8 holding 200 doesn't hold 200.
Mixed types in one column make it object. Any aggregation either fails or makes a thing meaningless.
A float used as a key in a merge or a group-by. Floating-point equality is unreliable. Two values that print the same way may not match.
Each of these is caught by looking at dtypes at once after loading. This Is why that habit appears in three split modules.
When to convert
At read time if you can. dtype= in read_csv. No extra copy, no window where the column is wrong.
at once after loading else, before any analysis. Each later step sees the right types.
Not again and again. Each astype allocates a full copy of the column. Converting inside a function that runs per group or per file is a common and invisible cost.
A summary
Text columns hold pointers. Use deep=True to see their real size.
group for repeated values, and only for repeated values.
Downcast numerics when the range is truly bounded.
Int64, boolean and string hold missing values without changing type.
to_numeric(errors="coerce") converts what it can and shows you what it couldn't.
Set types as early as possible. Check dtypes right after loading. It's one line and it blocks a group of silent wrongness.
Questions people ask about Dtypes and Memory
Why is my numeric column stored as object? A single non-numeric value, such as a stray string or symbol, makes pandas read the whole column as text. Use pd.to_numeric(col, errors='coerce') to find and convert it.
How do I reduce a DataFrame's memory use? Convert repeated strings to category, downcast integers and floats, and drop columns you do not need. Check with df.memory_usage(deep=True).
When should I use the category dtype? Use it for columns with few distinct values repeated many times, such as country or status. It stores each value once and uses small integer codes.
What is the difference between int64 and Int64? int64 is the NumPy type and cannot hold missing values. Int64 is the nullable type, which stores missing values as pd.NA and keeps the column as integers.
Why does memory_usage show a smaller number than the real one? Without deep=True it counts only the pointers for object columns, not the strings they point to. Pass deep=True for a realistic figure.
Is float32 safe to use instead of float64? It halves memory but carries about seven significant digits, so it can lose precision in sums and money values. Use it where that loss is acceptable.
A closing note on Dtypes and Memory
Dtypes are the quiet determinant of whether a pandas script is fast, correct. Able to load its data at all.
The memory story is simple: text columns hold pointers to Python strings and dominate most real frames, group merges them when values repeat. Narrow numeric types halve or quarter the rest. Two loops run once after loading routinely cut a frame's footprint in half.
The correctness story matters more. A numeric column stored as object checks as strings. "100" < "20" is true and each sort and max is silently wrong. An identifier read as an int loses its leading zeros permanently. An int column with one gap turns float. Ids start printing with a decimal point.
None of those raise. All of them are visible in one line of df.dtypes, run at once after loading. Which is why that habit appears in a few modules and is worth more than any other single check in this track.
Check yourself
0 of 4
Answer without scrolling back up.
Why does `memory_usage()` undercount a text column?
Use deep=True, which follows the pointers. The gap between the two numbers is often a factor of ten.
When does converting a column to `category` cost more than it saves?
It pays when distinct values are well under half the row count, and enormously when they are a tiny fraction.
What is the difference between `int64` and `Int64`?
It fixes the oldest wart in pandas: a plain integer column with one gap becomes float64, so ids print as 1001.0
How do you convert a messy text column to numbers without stopping at the first bad value?
It turns failures into NaN so you can find them: s[converted.isna() & s.notna()] gives the rows that were present and unconvertible.
Cheat sheet
Dtypes and Memory
Inspect and convert column dtypes in a pandas DataFrame. Switch object columns to category or nullable types and measure the memory saved.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.