Where the time actually goes in a pandas script, in the order worth fixing it.
Overview
The order to work in
Performance tuning in pandas follows a predictable order. The biggest gains come from the simplest changes, and the biggest mistake is starting with the hardest optimisation while ignoring the obvious one.
The most common performance problem is per-row Python. An apply call with axis=1, an iterrows loop, or a list comprehension that processes one row at a time. Each of these drops out of compiled code and into the Python interpreter, which is roughly a hundred times slower. Replacing the loop with a vectorised column operation gives you that factor back.
The second item on the list is dtype sizing. A column of country codes stored as object takes ten times the memory of the same column stored as category. A column of small integers stored as int64 takes eight times the memory of int8. Halving the memory often doubles the speed, because the data fits in cache more often.
The third item is filtering early. If your analysis uses only rows from 2024, filtering at the top of the pipeline means every subsequent step processes fewer rows. That is free speed. The later the filter, the more work pandas does on rows you throw away.
The fourth item is reducing unnecessary copies. Each operation that returns a new DataFrame allocates memory and copies data. In a long pipeline, that adds up. Using inplace where it exists, or assigning back to the same variable, can reduce the copying.
The fifth and last item is reaching past pandas entirely. For very large data, Polars and DuckDB are faster. For numeric computation on uniform arrays, dropping to NumPy avoids the overhead of index alignment. These tools solve problems that pandas was not designed for.
This module works through the list in order, with timing measurements that show the payoff of each step. The point is not just what to do, but what to do first.
Pandas speed work is unusually predictable. In descending order of payoff:
Drop per-row Python. Worth 10–100x, and it's nearly always the whole problem.
Right-size the dtypes. Halves or better on memory. Speeds up all that touches the column.
Filter early. Free, and often the largest structural win.
Reduce copies. Worth a modest amount, and only in hot paths.
Reach past pandas. When the data or the method truly doesn't suit it.
Working down that list in order means the large wins come first. Working up it means spending an afternoon on step 4 while a apply(axis=1) sits untouched.
Worth knowing
Per-row Python is nearly always the whole story — apply(axis=1) is orders of magnitude slower than column arithmetic.
iterrows is the one to never use: it is far slower than itertuples and converts each row to an object Series, losing dtypes.
category for repeated text and to_numeric(downcast=...) for narrow numbers are the cheapest memory wins.
Filter before you compute — every operation costs in proportion to the rows it touches.
Each chained assign copies. Usually worth it for readability, but the cost is real.
Measure first. The bottleneck is rarely where it feels like it is, and optimising a fast operation gains nothing.
Performance
Where the time really goes, in the order worth fixing it.
Per-row Python is the whole story
Four spellings of the same operation, two orders of magnitude apart.
example_01.pypandas
import pandas as pd
import numpy as np
import time
n = 100_000
df = pd.DataFrame({"a": np.arange(n), "b": np.arange(n)})
def timed(fn):
t = time.perf_counter(); fn(); return time.perf_counter() - t
t_iter = timed(lambda: [r.a * r.b for r in df.itertuples()])
t_apply = timed(lambda: df.apply(lambda r: r["a"] * r["b"], axis=1))
t_zip = timed(lambda: [a * b for a, b in zip(df["a"], df["b"])])
t_vec = timed(lambda: df["a"] * df["b"])
print("apply(axis=1) : %.4f s" % t_apply)
print("itertuples : %.4f s" % t_iter)
print("zip of columns : %.4f s" % t_zip)
print("vectorised : %.4f s" % t_vec)
print()
print("vectorised is %.0fx faster than apply(axis=1)" % (t_apply / max(t_vec, 1e-9)))
Output
iterrows is the one never to use
It converts every row to an object Series, losing dtypes on the way.
example_02.pypandas
import pandas as pd
import numpy as np
import time
df = pd.DataFrame({"a": np.arange(20_000), "b": np.arange(20_000)})
t = time.perf_counter()
for _, row in df.iterrows():
pass
rows = time.perf_counter() - t
t = time.perf_counter()
for _ in df.itertuples():
pass
tuples = time.perf_counter() - t
print("iterrows : %.4f s" % rows)
print("itertuples : %.4f s" % tuples)
print("ratio : %.0fx" % (rows / max(tuples, 1e-9)))
print()
mixed = pd.DataFrame({"i": [1], "s": ["x"]})
first = next(mixed.iterrows())[1]
print("and iterrows loses the dtypes:", first.dtype,
"<- the int became object")
Output
dtype is the cheapest memory win
category and downcasting, measured on the same frame.
Every operation costs in proportion to the rows it touches.
example_04.pypandas
import pandas as pd
import numpy as np
import time
n = 200_000
rng = np.random.default_rng(0)
df = pd.DataFrame({"keep": rng.random(n) < 0.05, "v": rng.random(n)})
t = time.perf_counter()
a = df.assign(x=df["v"] * 2)[df["keep"]]
late = time.perf_counter() - t
t = time.perf_counter()
sub = df[df["keep"]]
b = sub.assign(x=sub["v"] * 2)
early = time.perf_counter() - t
print("compute then filter : %.4f s" % late)
print("filter then compute : %.4f s" % early)
print("rows kept : %d of %d" % (len(b), n))
print()
print("Obvious, and routinely done the wrong way round because the")
print("filter reads more naturally at the end of a chain.")
Output
Chained operations each copy
Convenience has a price you can measure.
example_05.pypandas
import pandas as pd
import numpy as np
import time
n = 200_000
df = pd.DataFrame({"a": np.arange(n, dtype="float64")})
t = time.perf_counter()
out = (df.assign(b=lambda d: d["a"] * 2)
.assign(c=lambda d: d["b"] + 1)
.assign(d=lambda d: d["c"] / 3))
chained = time.perf_counter() - t
t = time.perf_counter()
df2 = df.copy()
df2["b"] = df2["a"] * 2
df2["c"] = df2["b"] + 1
df2["d"] = df2["c"] / 3
inplace = time.perf_counter() - t
print("three chained assigns : %.4f s" % chained)
print("three direct assigns : %.4f s" % inplace)
print("same result :", np.allclose(out["d"], df2["d"]))
print()
print("Chaining is usually worth its cost for readability. Worth")
print("knowing the cost exists before optimising something else.")
Output
Measure before changing anything
The bottleneck is rarely where it feels like it is.
example_06.pypandas
import pandas as pd
import numpy as np
import time
n = 100_000
rng = np.random.default_rng(0)
df = pd.DataFrame({
"k": rng.integers(0, 500, n),
"t": ["value-%d" % i for i in range(n)],
"v": rng.random(n),
})
def timed(label, fn):
t = time.perf_counter(); fn(); d = time.perf_counter() - t
print("%-22s %.4f s" % (label, d)); return d
a = timed("groupby sum", lambda: df.groupby("k")["v"].sum())
b = timed("sort_values", lambda: df.sort_values("v"))
c = timed("str.upper", lambda: df["t"].str.upper())
d = timed("column arithmetic", lambda: df["v"] * 2)
print()
print("Slowest here: %s" % max([("groupby", a), ("sort", b),
("str", c), ("arith", d)],
key=lambda p: p[1])[0])
print("Optimising the arithmetic would have gained nothing.")
Output
Per-row Python
df.apply(func, axis=1), iterrows, and a for loop over rows all do the same thing. Call back into the interpreter once per row.
The first editor measures four spellings of the same product. The vectorised form is orders of magnitude faster than apply(axis=1). notably a plain zip over two columns beats apply comfortably. Because apply also builds a Series object per row.
The hands-on rule: if a lambda indexes into a row, the step is column math in disguise.
iterrows
Worth singling out because it's common and there's no situation where it's the right choice.
It's much slower than itertuples, and it converts each row to a Series. A row holding an int and a string turns an object Series. The int arrives as an object. Code that then does math on it's slower again, and code that checks types breaks.
itertuples is faster and preserves dtypes. If you must iterate, use that. Better still, don't iterate.
dtypes
Covered in its own module, and it belongs here too because memory and speed are the same problem. Less data to move is less time spent moving it.
group for repeated text is the biggest single win on most real frames. It speeds up group-by and compare and saving memory. to_numeric(downcast=...) narrows numeric columns.
The third editor shows both applied to one frame.
Filter early
each step costs in proportion to the rows it touches. Filtering to 5% of the data before finding means each later step does 5% of the work.
This is obvious stated plainly and routinely done backwards. A filter reads more on its own at the end of a chain than in the middle of one. df.assign(...).query(...) finds for each row and then throws most of them away. df.query(...).assign(...) doesn't.
The same applies to columns: usecols at read time. Dropping columns you won't use, both reduce all downstream.
Copies
Most pandas steps return a new object. A chain of three assign calls allocates three frames.
That is often the right trade. The chain is readable, doesn't mutate anything. Works well in a notebook. The cost only matters inside a loop or on a frame large enough that allocation dominates.
Two things to know rather than to apply all over. Direct set (df["b"] = ...) modifies in place and skips the copy. inplace=True on other methods often does not avoid a copy despite its name, while breaking chaining and returning None. It isn't the tuning it appears to be.
Measure
The last editor times four steps on the same frame. The ranking isn't what most people would guess. It changes with the data.
%timeit in a notebook, or time.perf_counter around a block, is enough for most choices. Take the best of a few runs rather than the mean. The slow runs are measuring the machine.
Two rules that save the most wasted effort:
Profile the real workload. A small sample has other traits. Other memory pressure, other cache action, at times a other code path.
Fix the top item and measure again. The bottleneck moves. The second item on the first list is often no longer second.
And keep the ceiling in view: an step taking 5% of the runtime can't give back more than 5%. Cleverly it's rewritten.
When to leave pandas
pandas assumes the data fits in memory on one machine. It's optimised for convenience rather than raw speed.
Polars is much faster on large joins and aggregations, with a similar model and a lazy engine.
DuckDB runs SQL over frames and files. Is great for analytical queries larger than memory.
Dask splits frames across cores or machines with a pandas-like API.
NumPy right, when the data is homogeneous and you don't need labels.
The signal that it's time is often memory rather than speed. When a frame no longer fits, no amount of tuning inside pandas fixes it.
Profiling a pandas script
%timeit in a notebook and time.perf_counter around a block cover most choices.
For a whole script, cProfile finds the hot function and line_profiler finds the line inside it. Array code often has one line taking most of the time. This Line-level profiling shows at once and function-level profiling hides.
df.info(memory_usage="deep") and memory_profiler cover the memory side.
Two rules save the most wasted effort. Profile the real workload. A small sample has other cache action and at times takes a other code path. And re-profile after each fix, because the bottleneck moves.
Reducing before combining
The largest structural wins often come from doing less work rather than doing the same work faster.
Read fewer columns — usecols at read time.
Read fewer rows — filter during the chunk loop rather than after.
Aggregate before joining — joining two summaries is far cheaper than joining raw tables and summarising later.
Select columns before merging — carrying twenty unused columns through a join costs memory and time.
Each of these changes the size of what all downstream touches. This Compounds through a pipeline in a way that micro-tuning doesn't.
Categoricals as an optimisation
Converting a repeated string column to group helps in four places at once:
Memory, often by an order of magnitude.
Group-by, which then operates on int codes.
Merge, when both sides share the same groups.
.str steps, which apply to the groups rather than to each row.
That last one is worth restating: df["city"].str.upper() on a million rows with four distinct cities does four steps when the column is cat. A million when it's object.
The cost is the conversion itself and the care needed when mixing frames with other group sets.
Avoiding repeated work
Two patterns that quietly dominate slow scripts:
Recomputing inside a loop. A group statistic, a lookup table, a parsed date. Found once outside the loop rather than once per loop.
Repeated boolean masks. Building the same mask a few times to select other columns. Build it once, name it, reuse it.
Both are plain programming discipline rather than anything pandas-exact. Both are easy to miss because each single line looks cheap.
The limits
pandas is single-threaded for most steps, holds all in memory, and is optimised for convenience.
A rough guide to when to look elsewhere:
Under a million rows — pandas is comfortable. Tuning is rarely needed beyond dropping loops.
One to ten million — dtypes and access patterns start to matter. The techniques in this module earn their keep.
Over ten million, or wider than memory. The constraint is often memory rather than speed. Polars, DuckDB or Dask are the answer rather than a cleverer pandas formula.
truly sequential methods — Numba or a rewrite, at any size.
Recognising which regime you're in blocks both premature tuning and the opposite mistake of spending days making pandas do a thing it structurally can't.
A short checklist
Is there a loop over rows? Drop it.
Is anything growing in a loop? Collect and combine once.
Are the dtypes right? groups for repeated text, narrow numerics where bounded.
Is the filter as early as it can be?
Have you measured, or are you guessing?
The first two account for most real slowdowns. Neither needs knowing anything about pandas internals.
A worked speed-up
A slow script often has one dominant problem. The sequence for finding it's always the same.
Time the whole thing. Time each stage. Find the stage taking most of it. Look at what that stage does per row.
The fixes, in the order they often apply:
A row-wise apply turns column math or np.select.
A loop building a frame turns a list and one concat.
A merge inside a loop turns one merge outside it.
A repeated group statistic turns one transform.
An object column that should be a group turns one.
A filter at the end moves to the beginning.
Each of those is a small edit with a large effect. Together they account for most of the gap between a script that takes minutes and one that takes seconds.
Memory, when the frame will not fit
The techniques differ from the speed ones, because the constraint is other.
usecols at read time — never load what you won't use.
dtype at read time — groups and narrow numerics.
chunksize — process and reduce a piece at a time.
del and gc.collect() — release intermediates by name in a long-running process.
Avoid keeping both a frame and a transformed copy alive. Reassign rather than naming a new var, when the old one isn't needed.
And check whether the step needs the whole frame at all. Many aggregations can be found chunk-wise and mixed.
What not to optimise
Some things are already fast and are often rewritten for no gain:
Column math — already compiled.
Boolean masking — already compiled. The cost is the copy, not the compare.
**groupby with a string aggregation** — already compiled.
Reading a small file — measured in milliseconds.
Anything running once on a small frame.
Rewriting these makes less readable code and no measurable gain. This Is a bad trade. The reading step exists to prevent just that.
A summary
Drop per-row Python first. It's often the whole problem.
Never grow a frame in a loop.
Right-size dtypes — groups and narrow numerics.
Filter early, so all downstream does less.
itertuples if you must iterate. Never iterrows.
inplace=True isn't an tuning.
Measure the real workload, fix the top item, measure again.
And know when the answer is a other tool. When the data doesn't fit, no pandas technique fixes it.
Questions people ask about Performance
Why is my pandas code slow? The usual cause is a Python loop over rows: iterrows, apply with axis=1, or growing a frame in a loop. Replace it with whole-column operations.
Is iterrows ever acceptable? For small frames or one-off scripts, yes. On more than a few thousand rows, vectorise instead, because it is typically far faster.
How do I reduce memory use quickly? Read only the columns you need, use category for repeated strings, and downcast numbers. Check the result with memory_usage(deep=True).
How do I find out which line is slow? Time suspects with %timeit in a notebook or time.perf_counter, and use a profiler such as cProfile for a whole script. Measure first, because the slow step is often not where you expect.
Should I filter before or after merging? Before. Reducing the rows and columns first makes the merge smaller and faster.
When should I switch away from pandas? When data no longer fits in memory or operations remain slow after vectorising. Polars, DuckDB or Dask handle larger data and parallel work.
A closing note on Performance
Pandas speed is unusually predictable. This Makes it unusually easy to get right.
Almost each slow script has the same cause: Python running once per row, spelled as apply(axis=1), iterrows, or a loop. Dropping it's worth one or two orders of magnitude. Nothing else on the list comes close.
After that the wins are structural rather than clever. Don't grow frames in loops, filter before finding rather than after. Give columns types that fit the data. None of these need knowing anything about pandas internals.
What does need discipline is measuring. The bottleneck is regularly not where it feels like it's. The steps people most often rewrite. Column math, boolean masks, group-by with a string aggregation. Are already compiled and already fast.
And there's a ceiling. When the data no longer fits in memory, no pandas technique fixes it. The answer is Polars, DuckDB, Dask or a DB.
Check yourself
0 of 4
Answer without scrolling back up.
What is nearly always the largest performance problem in a pandas script?
If a lambda indexes into a row, the operation is column arithmetic in disguise. This is worth 10-100x, ahead of everything else.
Why is `iterrows` never the right choice?
A row with an int and a string becomes object, so the integer arrives as an object. If you must iterate, use itertuples.
Why does `df.query(...).assign(...)` usually beat `df.assign(...).query(...)`?
Obvious stated plainly and routinely done backwards, because a filter reads more naturally at the end of a chain.
Does `inplace=True` avoid a copy?
It is not the optimisation it appears to be. Direct assignment df['b'] = ... does modify in place and skips the copy.
Cheat sheet
Performance
Profile a slow pandas workflow and fix the top bottlenecks. Replace row-level loops, shrink memory with better dtypes and go vectorised.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.