Group statistics broadcast back to every row, and whole groups kept or dropped.
Overview
The shape decides the method
The previous module covered group-by with agg, which returns one row per group. That is the right tool when you want a summary table. But many real tasks need the group statistic broadcast back to every row. Normalising sales by each store's total, flagging rows above the group average, ranking within each category -- all of these need a value per original row, not per group.
That is what transform does. It computes a statistic per group and returns a Series with the same index as the input. The result aligns to the original frame and can be assigned as a new column directly. No merge is needed, and the alignment is handled by the index.
The common alternative is to compute the agg result and then merge it back. That works, but it is slower, harder to read, and introduces a join key that must match exactly. Transform does the same thing in one step because it knows the group boundaries from the split.
filter is the third method. It returns the original rows, but only for groups that pass a test. Keep only cities where total sales exceed a threshold. Keep only categories with more than ten records. The test applies to the group, and the result is a subset of the original rows.
Knowing which method to use is almost always obvious once you ask what shape you want. One row per group means agg. One row per original row means transform. A subset of the original rows means filter. That decision tree covers the vast majority of group-by tasks.
This module covers transform with built-in functions and with lambdas, filter with group-level tests, and the cases where agg and a merge are still the better option. The examples show the same problem solved with each approach, so the differences are concrete.
Three group-by steps differ only in the shape of what they return. Choosing between them is nearly always obvious once you ask what shape you want.
agg returns one row per group. A summary table.
transform returns one row per first row. A new column.
filter returns a subset of the first rows. Fewer rows, same columns.
Most confusion here is someone reaching for agg when they wanted transform, then writing a merge to get back to the first shape.
Worth knowing
agg reduces to one row per group; transform returns one row per original row, so it can be assigned straight back as a column.
transform replaces the aggregate-then-merge pattern in one line, with no chance of the join changing the row count or order.
Centring, ranking and computing a share within a group are all transform, and all would otherwise need a loop or a merge.
Filling missing values with a group mean uses information a global mean throws away.
filter keeps or drops whole groups by a predicate, and returns rows rather than groups.
Choose by output shape: a summary is agg, a new column is transform, fewer rows is filter.
transform and filter
Group statistics broadcast back to each row.
agg reduces, transform broadcasts
Same statistic, different shape - and transform is the one you can assign back.
example_01.pypandas
import pandas as pd
df = pd.DataFrame({
"city": ["pune", "delhi", "pune", "delhi"],
"sales": [10, 20, 30, 40],
})
print("agg gives one row per group:")
print(df.groupby("city")["sales"].sum())
print()
print("transform gives one row per ORIGINAL row:")
print(df.groupby("city")["sales"].transform("sum"))
print()
df["city_total"] = df.groupby("city")["sales"].transform("sum")
df["share"] = (df["sales"] / df["city_total"] * 100).round(1)
print(df)
Output
Why transform beats a merge
The alternative is aggregate then join back, which is three steps and a chance to lose rows.
example_02.pypandas
import pandas as pd
df = pd.DataFrame({
"city": ["pune", "delhi", "pune"],
"sales": [10, 20, 30],
})
# the long way
totals = df.groupby("city", as_index=False)["sales"].sum()
totals = totals.rename(columns={"sales": "total"})
merged = df.merge(totals, on="city")
print("aggregate then merge:")
print(merged)
# the short way
df["total"] = df.groupby("city")["sales"].transform("sum")
print()
print("transform:")
print(df)
print()
print("Same answer, one line, and no chance of the merge changing the")
print("row count or the order.")
Output
Centring and ranking within groups
The common uses: a value relative to its own group.
example_03.pypandas
import pandas as pd
df = pd.DataFrame({
"team": ["a", "a", "a", "b", "b"],
"score": [10, 20, 30, 100, 200],
})
g = df.groupby("team")["score"]
df["team_mean"] = g.transform("mean")
df["centred"] = df["score"] - df["team_mean"]
df["rank_in_team"] = g.rank(ascending=False).astype(int)
df["pct_of_team"] = (df["score"] / g.transform("sum") * 100).round(1)
print(df)
print()
print("Every one of these would need a loop or a merge otherwise.")
Output
Filling missing values per group
A group mean is usually a better guess than a global one.
example_04.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"city": ["pune", "pune", "delhi", "delhi"],
"temp": [30.0, np.nan, 10.0, np.nan],
})
print("global mean fill:")
print(list(df["temp"].fillna(df["temp"].mean()).round(1)))
print(" both gaps got 20.0, which is right for neither city")
print()
df["filled"] = df["temp"].fillna(df.groupby("city")["temp"].transform("mean"))
print("group mean fill:")
print(df)
print(" pune got 30, delhi got 10")
Output
filter keeps or drops whole groups
The predicate is asked once per group and applies to all its rows.
example_05.pypandas
import pandas as pd
df = pd.DataFrame({
"city": ["pune", "pune", "delhi", "goa", "goa", "goa"],
"sales": [10, 20, 5, 1, 2, 3],
})
print("group sizes:"); print(df.groupby("city").size())
print()
big = df.groupby("city").filter(lambda g: len(g) >= 2)
print("groups with at least 2 rows:")
print(big)
print()
print("by an aggregate instead of a count:")
print(df.groupby("city").filter(lambda g: g["sales"].sum() > 6))
print()
print("Note this returns ROWS, not groups - the shape is like the input.")
Output
Choosing between the three
The shape of what you want decides which one you need.
example_06.pypandas
import pandas as pd
df = pd.DataFrame({"k": ["a", "a", "b"], "v": [1, 2, 3]})
print("rows in :", len(df))
print()
print("agg ->", len(df.groupby("k")["v"].sum()), "rows (one per group)")
print("transform ->", len(df.groupby("k")["v"].transform("sum")), "rows (one per input row)")
print("filter ->", len(df.groupby("k").filter(lambda g: len(g) > 1)), "rows (a subset)")
print()
print("Ask: do I want a summary, a new column, or fewer rows?")
print(" a summary -> agg")
print(" a column -> transform")
print(" fewer rows -> filter")
Output
transform
df.groupby("city")["sales"].transform("sum") gives each row its city's total.
Because the result is aligned with the first frame, it assigns straight back:
Two lines for "what fraction of its city's sales is this row". This Is a question that else needs an aggregation, a rename, a merge and a check that the merge didn't change anything.
That last point is the real arg. The aggregate-then-merge route is three steps, each of which can go wrong. The aggregate can drop NaN keys, the merge can times rows if the key isn't unique. The result can come back in a other order. transform can't do any of those. It never leaves the first index.
What transform accepts
A string naming a built-in ("sum", "mean", "max", "count") is fastest. It dispatches to compiled code.
A function that maps a Series to a Series of the same length also works. transform(lambda s: s - s.mean()) centres within group.
A function returning a scalar is broadcast to the whole group. This Is why transform("sum") works at all.
What it can't do is change the shape. A function returning a other length raises. This Is the gap between transform and apply and the reason transform is predictable.
Some methods exist right on the groupby object and don't need transform: rank, cumsum, cumcount, shift, diff and pct_change all return per-row results already. g.rank() is both clearer and faster than g.transform("rank").
Common uses
A share of the group total — the example above.
Centring within group — df["v"] - g.transform("mean"). Standard prep when groups have other baselines.
Rank within group — g.rank(ascending=False). "Where does this row come in its own team."
Group-wise fill — df["temp"].fillna(g.transform("mean")). Filling a missing temp with the mean for that *city* rather than the mean of all over. This Uses data a global fill throws away.
Flagging outliers scaled to the group. Checking each value against its group's standard deviation.
filter
df.groupby("city").filter(lambda g: len(g) >= 2) keeps each row belonging to a group with at least two rows.
The predicate receives the whole sub-frame and returns a single True or False. It's asked once per group. The answer applies to all that group's rows.
The result is rows, not groups. The same columns as the input, with whole groups removed.
Common uses: dropping groups with too little data to be real, keeping only customers with more than one order, or dropping groups below a volume threshold.
Note this is a Python callback per group. It's slower than an aggregation. For simple size rules, finding sizes with transform("size") and filtering with a boolean mask is faster and does the same thing:
That form is worth knowing because it composes with other rules, whereas filter doesn't.
A note on apply
groupby(...).apply(func) is the general escape hatch, and it can return any shape.
That flexibility costs speed — it's a Python call per group — and predictability. The shape of the result depends on what the function returned. In pandas 2.2 it also warns about whether the grouping columns are passed to the function. This Is a sign of how awkward its semantics have become.
Reach for agg, transform or filter first. They cover the large majority of cases, run in compiled code. Return a shape you can predict from the method name alone.
Ranking and cumulative operations within groups
a few methods on a groupby already return one row per input row. They need no transform:
cumsum, cumcount, cummax — running totals and counters within each group.
rank — spot within the group.
shift, diff, pct_change — compare with the previous row of the same group.
That last set matters more than it looks. df["v"].diff() on a frame holding a few entities finds a gap across the boundary between one entity and the next. This Is meaningless. df.groupby("id")["v"].diff() doesn't.
Any time you use shift, diff or rolling on panel data. A few entities stacked in one frame. The group-by version is almost certainly the one you want. Using the plain version is a quiet, plausible-looking error.
cumcount() numbers the rows within each group from zero. This Is how you take "the first three per group" or label repeat visits.
Percentages and shares
The single most common transform is a share of the group:
A share of the whole frame needs no group-by: df["sales"] / df["sales"].sum().
A share within a nested group takes a list of keys: transform("sum") on groupby(["region", "city"]).
Because the denominator comes back aligned, these compose. You can find a share of city and a share of region in the same frame and compare them.
Standardising within groups
Centring and scaling scaled to a group is standard prep when groups have other baselines. Other stores, other sensors, other years:
g = df.groupby("store")["sales"]
df["z"] = (df["sales"] - g.transform("mean")) / g.transform("std")
Two cautions. A group with one row has a standard deviation of NaN. The result is NaN for that row rather than zero. And a group with zero variance cuts by zero, giving inf. Both are worth handling by name rather than discovering downstream.
Filtering by a group property
filter takes a callback per group. This Is flexible and slow.
For the common cases, a transform and a boolean mask is faster and composes better:
Five columns, no loop, no merge, and each one aligned with the first rows.
Written with an aggregate-and-merge it would be four split summaries and four joins, each of which could change the row count.
When transform is not enough
transform needs the function to return either a scalar or a thing the same length as the group. Two cases fall outside that.
A other number of rows per group. Taking the top two per group, say. That is a filter. Rank with transform or a groupby method, then mask.
a few columns from one math. transform operates column by column. A math needing two columns together doesn't fit. groupby(...).apply handles it, or restructure so each output column is found apart.
filter versus a mask
They do the same job with other trade-offs.
filter(lambda g: ...) — the callback sees the whole sub-frame. Any rule is expressible. It's a Python call per group.
df[df.groupby(k)[c].transform("size") >= n] — compiled, faster, and composes with other rules using &.
Use the mask for simple rules on a single statistic. This Is most of them. Use filter when the predicate truly needs a few columns or a math with no vectorised form.
transform assigns straight back, because it keeps the first index.
It replaces aggregate-then-merge, without the risk of changing row count or order.
Group-wise rank, cumsum, cumcount, shift and diff are methods in their own right — no transform needed.
On panel data, always use the group-wise shift/diff, or the compare crosses between entities.
Group fills use data a global fill throws away.
And prefer a transform-built mask to filter for simple size or total rules.
A closing note on transform and filter
transform is the step that drops the most not needed joins from real pandas code.
The pattern it replaces — aggregate to a summary, rename the column, merge it back — is three steps, each of which can change the row count or the order. All of which exist only to get a group statistic aligned with the rows it came from. transform does that by build. It never leaves the first index.
Once you've it, a family of questions turns one line each. This row's share of its group, its rank within its group, its distance from its group's mean, its group's size as a filter.
filter is the less-used sibling and is often better expressed as a mask built from transform. This Composes with other rules and runs in compiled code.
Between agg, transform and a mask, the all-round apply is rarely needed. Which is the point. It's the slow and unpredictable one.
One more thing
transform accepts a list of functions in recent pandas, returning one column per function with a nested column index. That is at times convenient and often clearer written as split sets. Each output then has a name you chose.
More handy is that transform works on a whole frame, not just one column: df.groupby("k").transform("mean") returns a frame of group means for each numeric column at once, aligned with the first rows. Subtracting it centres the entire frame within groups in a single formula.
Questions people ask about transform and filter
What is the difference between agg and transform? agg returns one row per group, while transform returns a result with the same length as the input. That makes transform suited to adding a group statistic as a new column.
How do I add each group's mean as a new column? Use df['g_mean'] = df.groupby('g')['x'].transform('mean').
How do I calculate each row's share of its group total? Divide the column by its group sum: df['x'] / df.groupby('g')['x'].transform('sum').
What does groupby().filter() do? It keeps or drops whole groups based on a condition on the group, such as filter(lambda g: len(g) > 5). The rows in the surviving groups come back unchanged.
Why is apply slower than transform? transform with a built-in name uses optimised code for all groups. apply runs a Python function on each group, which is slower.
How do I rank rows within each group? Use df.groupby('g')['x'].rank(ascending=False), which numbers the values separately inside every group.
transform and filter, in summary
Three steps, distinguished only by the shape they return: agg gives one row per group, transform one row per input row, filter a subset of rows.
transform is the one that drops work. It replaces the aggregate-rename-merge pattern with a single formula that can't change the row count or the order.
Prefer the group-by methods that already return per-row results. rank, cumsum, cumcount, shift, diff. Over transform where they apply. Use the group-wise forms on panel data so checks don't cross between entities.
Check yourself
0 of 4
Answer without scrolling back up.
What is the difference between `agg` and `transform`?
Because transform stays aligned with the original index, it assigns straight back as a column.
Why is `transform` safer than aggregate-then-merge?
The merge route can drop NaN keys, multiply rows on a non-unique key, or reorder the result. transform can do none of those.
What does `groupby(...).filter(pred)` return?
The predicate is asked once per group and applies to all its rows. The result has the same columns as the input.
Why prefer `g.rank()` over `g.transform('rank')`?
cumsum, cumcount, shift, diff and pct_change are the same - they return per-row results without needing transform.
Cheat sheet
transform and filter
Use transform to broadcast group statistics back to every row and filter to keep only groups that pass a test. Compare both to plain groupby.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.