Group statistics broadcast back to every row, and whole groups kept or dropped.
Overview
The shape decides the method
Three group-by operations differ only in the shape of what they return, and choosing between them is nearly always obvious once you ask what shape you want.
agg returns one row per group. A summary table.
transform returns one row per original row. A new column.
filter returns a subset of the original rows. Fewer rows, same columns.
Most confusion here is someone reaching for agg when they wanted transform, then writing a merge to get back to the original shape.
Worth knowing
agg reduces to one row per group; transform returns one row per original row, so it can be assigned straight back as a column.
transform replaces the aggregate-then-merge pattern in one line, with no chance of the join changing the row count or order.
Centring, ranking and computing a share within a group are all transform, and all would otherwise need a loop or a merge.
Filling missing values with a group mean uses information a global mean throws away.
filter keeps or drops whole groups by a predicate, and returns rows rather than groups.
Choose by output shape: a summary is agg, a new column is transform, fewer rows is filter.
transform and filter
Group statistics broadcast back to every row.
agg reduces, transform broadcasts
Same statistic, different shape - and transform is the one you can assign back.
example_01.pypandas
Output
Why transform beats a merge
The alternative is aggregate then join back, which is three steps and a chance to lose rows.
example_02.pypandas
Output
Centring and ranking within groups
The common uses: a value relative to its own group.
example_03.pypandas
Output
Filling missing values per group
A group mean is usually a better guess than a global one.
example_04.pypandas
Output
filter keeps or drops whole groups
The predicate is asked once per group and applies to all its rows.
example_05.pypandas
Output
Choosing between the three
The shape of what you want decides which one you need.
example_06.pypandas
Output
transform
df.groupby("city")["sales"].transform("sum") gives every row its city's total.
Because the result is aligned with the original frame, it assigns straight back:
Two lines for "what fraction of its city's sales is this row", which is a question that otherwise needs an aggregation, a rename, a merge and a check that the merge did not change anything.
That last point is the real argument. The aggregate-then-merge route is three steps, each of which can go wrong: the aggregate can drop NaN keys, the merge can multiply rows if the key is not unique, and the result can come back in a different order. transform cannot do any of those, because it never leaves the original index.
What transform accepts
A string naming a built-in ("sum", "mean", "max", "count") is fastest, because it dispatches to compiled code.
A function that maps a Series to a Series of the same length also works — transform(lambda s: s - s.mean()) centres within group.
A function returning a scalar is broadcast to the whole group, which is why transform("sum") works at all.
What it cannot do is change the shape. A function returning a different length raises, which is the difference between transform and apply and the reason transform is predictable.
Some methods exist directly on the groupby object and do not need transform: rank, cumsum, cumcount, shift, diff and pct_change all return per-row results already. g.rank() is both clearer and faster than g.transform("rank").
Common uses
A share of the group total — the example above.
Centring within group — df["v"] - g.transform("mean"). Standard preprocessing when groups have different baselines.
Rank within group — g.rank(ascending=False). "Where does this row come in its own team."
Group-wise fill — df["temp"].fillna(g.transform("mean")). Filling a missing temperature with the mean for that *city* rather than the mean of everywhere, which uses information a global fill throws away.
Flagging outliers relative to the group — comparing each value against its group's standard deviation.
filter
df.groupby("city").filter(lambda g: len(g) >= 2) keeps every row belonging to a group with at least two rows.
The predicate receives the whole sub-frame and returns a single True or False. It is asked once per group, and the answer applies to all that group's rows.
The result is rows, not groups — the same columns as the input, with whole groups removed.
Typical uses: dropping groups with too little data to be meaningful, keeping only customers with more than one order, or removing categories below a volume threshold.
Note this is a Python callback per group, so it is slower than an aggregation. For simple size conditions, computing sizes with transform("size") and filtering with a boolean mask is faster and does the same thing:
That form is worth knowing because it composes with other conditions, whereas filter does not.
A note on apply
groupby(...).apply(func) is the general escape hatch, and it can return any shape.
That flexibility costs speed — it is a Python call per group — and predictability, since the shape of the result depends on what the function returned. In pandas 2.2 it also warns about whether the grouping columns are passed to the function, which is a sign of how awkward its semantics have become.
Reach for agg, transform or filter first. They cover the large majority of cases, run in compiled code, and return a shape you can predict from the method name alone.
Ranking and cumulative operations within groups
Several methods on a groupby already return one row per input row, so they need no transform:
cumsum, cumcount, cummax — running totals and counters within each group.
rank — position within the group.
shift, diff, pct_change — comparison with the previous row of the same group.
That last set matters more than it looks. df["v"].diff() on a frame containing several entities computes a difference across the boundary between one entity and the next, which is meaningless. df.groupby("id")["v"].diff() does not.
Any time you use shift, diff or rolling on panel data — several entities stacked in one frame — the group-by version is almost certainly the one you want, and using the plain version is a quiet, plausible-looking error.
cumcount() numbers the rows within each group from zero, which is how you take "the first three per group" or label repeat visits.
Percentages and shares
The single most common transform is a share of the group:
A share of the whole frame needs no group-by: df["sales"] / df["sales"].sum().
A share within a nested group takes a list of keys: transform("sum") on groupby(["region", "city"]).
Because the denominator comes back aligned, these compose — you can compute a share of city and a share of region in the same frame and compare them.
Standardising within groups
Centring and scaling relative to a group is standard preprocessing when groups have different baselines — different stores, different sensors, different years:
g = df.groupby("store")["sales"]
df["z"] = (df["sales"] - g.transform("mean")) / g.transform("std")
Two cautions. A group with one row has a standard deviation of NaN, so the result is NaN for that row rather than zero. And a group with zero variance divides by zero, giving inf. Both are worth handling explicitly rather than discovering downstream.
Filtering by a group property
filter takes a callback per group, which is flexible and slow.
For the common cases, a transform and a boolean mask is faster and composes better:
The mask form can be combined with other conditions using &, which filter cannot. It is also easier to reason about, because it is the same masking you use everywhere else.
Reach for filter when the predicate genuinely needs the whole sub-frame — a condition involving several columns at once, or a statistical test per group.
Why apply is the last resort
groupby(...).apply(func) can return anything, which is its appeal and its problem.
It is a Python call per group, so it is slow.
The shape of the result depends on what the function returned, which makes it unpredictable to read.
And in pandas 2.2 it warns about whether the grouping columns are passed to the function, because the historical behaviour was ambiguous enough to need changing.
Before using it, check whether the operation is:
a summary per group — agg;
a value per row — transform, or a groupby method like rank or cumsum;
a subset of rows — a mask built from transform, or filter;
several columns from one computation — agg with named aggregation.
Those four cover the large majority of cases, run in compiled code, and return a shape you can predict from the method name.
A worked example
A frequent request: for each customer, how does this order compare with their own average, and where does it rank among their orders?
Five columns, no loop, no merge, and every one aligned with the original rows.
Written with an aggregate-and-merge it would be four separate summaries and four joins, each of which could change the row count.
When transform is not enough
transform requires the function to return either a scalar or something the same length as the group. Two cases fall outside that.
A different number of rows per group — taking the top two per group, for instance. That is a filter: rank with transform or a groupby method, then mask.
Several columns from one computation — transform operates column by column, so a calculation needing two columns together does not fit. groupby(...).apply handles it, or restructure so each output column is computed separately.
filter versus a mask
They do the same job with different trade-offs.
filter(lambda g: ...) — the callback sees the whole sub-frame, so any condition is expressible. It is a Python call per group.
df[df.groupby(k)[c].transform("size") >= n] — compiled, faster, and composes with other conditions using &.
Use the mask for simple conditions on a single statistic, which is most of them. Use filter when the predicate genuinely needs several columns or a computation with no vectorised form.
transform assigns straight back, because it keeps the original index.
It replaces aggregate-then-merge, without the risk of changing row count or order.
Group-wise rank, cumsum, cumcount, shift and diff are methods in their own right — no transform needed.
On panel data, always use the group-wise shift/diff, or the comparison crosses between entities.
Group fills use information a global fill throws away.
And prefer a transform-built mask to filter for simple size or total conditions.
A closing note
transform is the operation that removes the most unnecessary joins from real pandas code.
The pattern it replaces — aggregate to a summary, rename the column, merge it back — is three steps, each of which can change the row count or the order, and all of which exist only to get a group statistic aligned with the rows it came from. transform does that by construction, because it never leaves the original index.
Once you have it, a family of questions becomes one line each: this row's share of its group, its rank within its group, its distance from its group's mean, its group's size as a filter.
filter is the less-used sibling and is usually better expressed as a mask built from transform, which composes with other conditions and runs in compiled code.
Between agg, transform and a mask, the general-purpose apply is rarely needed — which is the point, because it is the slow and unpredictable one.
One more thing
transform accepts a list of functions in recent pandas, returning one column per function with a hierarchical column index. That is occasionally convenient and usually clearer written as separate assignments, since each output then has a name you chose.
More useful is that transform works on a whole frame, not just one column: df.groupby("k").transform("mean") returns a frame of group means for every numeric column at once, aligned with the original rows. Subtracting it centres the entire frame within groups in a single expression.
In summary
Three operations, distinguished only by the shape they return: agg gives one row per group, transform one row per input row, filter a subset of rows.
transform is the one that removes work, because it replaces the aggregate-rename-merge pattern with a single expression that cannot change the row count or the order.
Prefer the group-by methods that already return per-row results — rank, cumsum, cumcount, shift, diff — over transform where they apply, and use the group-wise forms on panel data so comparisons do not cross between entities.
Check yourself
0 of 4
Answer without scrolling back up.
What is the difference between `agg` and `transform`?
Because transform stays aligned with the original index, it assigns straight back as a column.
Why is `transform` safer than aggregate-then-merge?
The merge route can drop NaN keys, multiply rows on a non-unique key, or reorder the result. transform can do none of those.
What does `groupby(...).filter(pred)` return?
The predicate is asked once per group and applies to all its rows. The result has the same columns as the input.
Why prefer `g.rank()` over `g.transform('rank')`?
cumsum, cumcount, shift, diff and pct_change are the same - they return per-row results without needing transform.
Cheat sheet
transform and filter
Three group-by operations differ only in the shape of what they return, and choosing between them is nearly always obvious once you ask what shape you want.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.