Split, apply, combine - the operation that is the reason to use pandas at all.
Overview
The three steps
Group-by is the operation that justifies learning pandas. Without it, computing a subtotal per category requires a loop, a dict for accumulation, a decision about what to do with missing keys, and another loop to format the output. With group-by, it is one line.
The concept is simple: split the rows into groups based on a key, apply a function to each group, and combine the results. The split is implicit. You never see the individual groups unless you ask for them. The apply step is where the actual computation happens. The combine step assembles the results into a Series or a DataFrame.
Understanding these three steps separately makes the output predictable. The split step decides how many groups you get. The apply step decides the shape of each group's result. The combine step decides the shape of the final output. A sum gives one number per group, so the result is a Series with one row per group. A describe gives multiple numbers per group, so the result is a DataFrame.
The key column becomes the index of the result. That is why the output of a group-by often looks different from the input. If you want a flat frame with the key as a regular column, pass as_index=False. That parameter exists precisely because the default behaviour surprises people who expected a DataFrame and got a Series with a named index.
You can group by multiple columns by passing a list. The result then has a MultiIndex with one level per key. You can also group by a computed value, a cut result, or a boolean mask. Anything that maps each row to a group label works as a key.
This module covers the basic pattern, multiple keys, selecting columns before aggregating, and the methods that apply to a GroupBy object. Once you see the split-apply-combine pattern, every aggregation question becomes a question about which function to apply and which key to split on.
df.groupby("city")["sales"].sum() does three things:
Split the rows into groups by the key.
Apply a function to each group.
Combine the results into one object, one row per group.
Almost each question about aggregated data has this shape. Recognising it's most of what makes pandas worth using. The option in plain Python. A dict, a loop, a choice about missing keys, another loop to find the summary. Is a dozen lines that this replaces with one.
Worth knowing
groupby splits, applies and combines — the group keys become the index unless you pass as_index=False.
The groupby object is lazy; nothing is computed until you aggregate. ngroups, size() and get_group() inspect it.
Several keys give a hierarchical index, one level per key; unstack() turns the last level into columns.
Rows whose key is missing are dropped by default, so group totals can silently fail to match the frame total. Pass dropna=False.
Named aggregation — agg(total=("sales", "sum")) — is the clearest form and controls the output names.
size counts rows; count counts non-missing values per column. They differ exactly where data is missing.
groupby
Split, apply, combine - the step that is the reason to use pandas.
Split, apply, combine
One key, one aggregation, one row per group.
example_01.pypandas
import pandas as pd
df = pd.DataFrame({
"city": ["pune", "delhi", "pune", "goa", "delhi"],
"sales": [10, 20, 30, 40, 50],
})
print(df.groupby("city")["sales"].sum())
print()
print("the group keys became the INDEX:")
g = df.groupby("city")["sales"].sum()
print(" index:", list(g.index))
print(" type :", type(g).__name__)
print()
print("as_index=False keeps them as a column instead:")
print(df.groupby("city", as_index=False)["sales"].sum())
Output
The object itself is lazy
Nothing is computed until you ask for something.
example_02.pypandas
import pandas as pd
df = pd.DataFrame({
"city": ["pune", "delhi", "pune"],
"sales": [10, 20, 30],
})
g = df.groupby("city")
print("the object:", type(g).__name__)
print("number of groups:", g.ngroups)
print("sizes:"); print(g.size())
print()
print("look inside one:")
print(g.get_group("pune"))
print()
print("iterate if you must - but an aggregation is nearly always better:")
for name, part in g:
print(" %-6s %d rows" % (name, len(part)))
Output
Several keys give a hierarchical index
One level per key, in the order you named them.
example_03.pypandas
import pandas as pd
df = pd.DataFrame({
"city": ["pune", "pune", "delhi", "delhi"],
"year": [2023, 2024, 2023, 2024],
"sales": [10, 20, 30, 40],
})
out = df.groupby(["city", "year"])["sales"].sum()
print(out)
print()
print("index levels:", out.index.names)
print()
print("unstack turns the last level into columns:")
print(out.unstack())
print()
print("reset_index gives a flat frame back:")
print(out.reset_index())
Output
Missing keys are dropped by default
Rows whose group key is NaN vanish, and the totals stop adding up.
example_04.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"city": ["pune", None, "delhi", np.nan],
"sales": [10, 20, 30, 40],
})
print("total sales in the frame:", int(df["sales"].sum()))
print()
print("default groupby:")
print(df.groupby("city")["sales"].sum())
print(" sums to", int(df.groupby("city")["sales"].sum().sum()), "- 60 went missing")
print()
print("dropna=False keeps them:")
print(df.groupby("city", dropna=False)["sales"].sum())
Output
Aggregating several columns at once
agg with a dict, or named aggregation for clean output names.
example_05.pypandas
import pandas as pd
df = pd.DataFrame({
"city": ["pune", "delhi", "pune", "delhi"],
"sales": [10, 20, 30, 40],
"units": [1, 2, 3, 4],
})
print("same function on everything:")
print(df.groupby("city").sum(numeric_only=True))
print()
print("a dict, one entry per column:")
print(df.groupby("city").agg({"sales": "sum", "units": "mean"}))
print()
print("named aggregation - the clearest form:")
print(df.groupby("city").agg(
total=("sales", "sum"),
biggest=("sales", "max"),
orders=("sales", "size"),
))
Output
size versus count
One counts rows, the other counts non-missing values.
example_06.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"city": ["pune", "pune", "delhi"],
"sales": [10, np.nan, 30],
})
print("size - rows per group:")
print(df.groupby("city").size())
print()
print("count - non-missing per column:")
print(df.groupby("city").count())
print()
print("pune has 2 rows but only 1 sales value.")
print()
print("nunique counts DISTINCT values:")
print(df.groupby("city")["sales"].nunique())
Output
A gap that matters just where the data is imperfect.
size() counts rows in each group, including rows with missing values.
count() counts non-missing values, per column. A group of five rows with two gaps in a column reports three for that column.
nunique() counts distinct values.
Using count where you meant size undercounts wherever data is missing. This Is just where you're least likely to notice.
The keys become the index
By default the group keys become the index of the result. That is convenient for lookup and for plotting. It's a surprise if you expected a plain column.
as_index=False keeps them as plain columns, giving a result that looks more like a table. reset_index() later does the same thing.
Which you want depends on what happens next. If the result feeds a merge or gets written to a file, the flat form is often easier. If you're going to select groups by name, the index form is better.
Results come back sorted by key. sort=False skips that sort. This Is faster on many groups and gives you the order of first appearance instead.
The object is lazy
df.groupby("city") finds nothing. It returns an object that knows how the rows would be split.
That is why you can inspect it cheaply: ngroups counts the groups, size() gives rows per group, get_group(name) pulls one out. Iterating yields (name, subframe) pairs.
Iterating is legitimate for inspection and almost always the wrong way to find a thing. The loop runs Python once per group. This Is the same speed trap as apply. If you find yourself building up results in a list inside a groupby loop, there's nearly always an agg that does it in one call.
Several keys
df.groupby(["city", "year"]) gives one level of index per key, in the order named.
The result is a Series or frame with a MultiIndex. This Has its own module later. Two steps cover most direct needs:
unstack() moves the last index level into columns, turning a long result into a wide table. Cities down the side, years across the top.
reset_index() flattens all back into plain columns.
Missing keys disappear
This is the action most likely to give a wrong number quietly.
Rows whose group key is NaN are excluded by default. They aren't put in a split group. They're dropped.
The result is that group totals don't add up to the frame total. Nothing says so. On a frame where 5% of the key column is missing, each aggregate is quietly found on 95% of the data.
dropna=False keeps them as a group with a NaN key.
The habit worth forming: after any group-by that matters, check that the total of the result matches the total of the input. It's one line and it catches this at once.
Aggregating
df.groupby("city").sum(numeric_only=True) applies one function to each numeric column.
agg({"sales": "sum", "units": "mean"}) applies a other function per column.
Named aggregation is the clearest form and the one to prefer:
Each output column is named by name, a few statistics can come from the same input column. The result has flat column names rather than the MultiIndex that a dict-of-lists makes. That last point saves a cleanup step almost each time.
Functions can be strings ("sum", "mean", "nunique"), NumPy functions, or your own callables. The strings are fastest. They dispatch to compiled setups rather than calling back into Python.
Grouping by something that is not a column
groupby accepts more than a column name.
A Series of the same length groups by its values. This Is how you group by a derived value without adding a column:
df.groupby(df["date"].dt.year)["sales"].sum()
A list of Series gives multiple keys the same way.
A function is applied to each index label and groups by the result.
**pd.Grouper** handles time: groupby(pd.Grouper(key="date", freq="ME")) groups by month without needing a DatetimeIndex. This Is what makes it handy alongside other keys.
**level=** groups by an index level rather than a column.
That flexibility means most "I need to add a column just to group by it" situations don't need the column.
as_index, sort and dropna
Three args change the shape or content of the result. All three are worth passing on purpose.
as_index=False keeps the keys as columns. The result looks like a table rather than an indexed Series. It's often what you want when the output feeds a merge or a file.
sort=False skips sorting the keys. On many groups that is a real saving. It gives order of first appearance instead of sorted order.
dropna=False keeps rows whose key is missing. The default drops them. This Is the action most likely to give a total that doesn't merge.
seen=True matters for cat keys: without it, group-by makes a row for each group including absent ones. This Fills the result with zeros.
Aggregating strings and other non-numerics
sum on a string column concatenates. This Is at times handy and often an accident.
The aggregations that really make sense on text:
"first" / "last" — a representative value, skipping missing.
"nunique" — how many distinct.
"count" — how many present.
lambda s: ", ".join(s) — collect them into one string.
list — collect them into a list, giving a column of lists. Convenient. It makes the column object. It's a display or export step rather than a thing to find on.
numeric_only=True on sum and mean skips non-numeric columns rather than doing a thing surprising with them. It's worth passing by name. The default has changed across versions.
Group-by is a split, and splits can be expensive
The cost of a group-by is roughly: hashing the keys, sorting or grouping the rows, then applying the aggregation once per group.
Three things make it faster:
A cat key. Grouping on int codes beats grouping on strings, often much.
**sort=False** when you don't need sorted output.
A string aggregation name rather than a lambda. "sum" dispatches to compiled code. lambda s: s.sum() calls Python once per group.
That last one is the most common avoidable slowdown. agg("sum") and agg(lambda s. S.sum()) give same results and differ by an order of magnitude on many groups.
Checking a group-by
Two checks catch most errors:
Do the totals merge?result["sales"].sum() against df["sales"].sum(). If they differ, a key was missing and the rows were dropped.
Is the group count what you expect?df.groupby(k).ngroups against df[k].nunique(). If the group count is larger, the key has variants you didn't know about. Whitespace, case, or a type mismatch.
Both are one line, and both catch problems that else surface as a number that is slightly wrong.
Reading a group-by result
The output shape depends on what you selected and how you aggregated. Knowing which of four shapes you've blocks most downstream confusion.
df.groupby("k")["v"].sum() — a Series, indexed by key.
df.groupby("k")[["v", "w"]].sum() — a DataFrame, indexed by key.
df.groupby("k", as_index=False)["v"].sum() — a DataFrame with the key as a column.
df.groupby(["k", "j"])["v"].sum() — a Series with a MultiIndex.
The double-bracket detail is the one people miss. Selecting one column with ["v"] gives a Series. With [["v"]] gives a one-column frame. That gap passes to all after it.
Aggregations worth knowing
Beyond sum, mean and count:
nunique — distinct values per group. This Answers "how many other products did each customer buy".
first / last — skip missing values. So They combine partial records.
idxmax / idxmin — the label of the extreme row. This Is how you get "the best row per group" rather than just its value: df.loc[df.groupby("k")["v"].idxmax()].
agg(lambda s: ...) — anything else, at the cost of a Python call per group.
show() — a full summary per group, handy for exploration and too wide for a report.
That idxmax pattern is worth remembering. It comes up constantly and the obvious options are all worse.
Empty groups and missing keys
Two other things that both cause totals not to merge.
Missing keys — rows whose group key is NaN are dropped unless dropna=False.
Empty groups — a cat key makes a row for each group, including ones with no rows, unless seen=True.
The first loses data silently. The second adds rows of zeros that were never in the data. Both defaults have changed across versions. This Is a reason to pass them by name rather than lean on the current action.
A summary
Split, apply, combine — and the group keys become the index unless you say else.
The object is lazy. Nothing runs until you aggregate.
Missing keys are dropped by default.
Named aggregation gives flat column names and lets one column feed a few statistics.
String aggregation names are much faster than lambdas.
idxmax plus .loc gets the whole extreme row per group.
And check that the group total matches the frame total. One line that catches the most common silent error here.
A closing note on groupby
Group-by is what pandas is for. Most of its surprises are about what silently doesn't appear in the result.
Rows whose key is missing are dropped. cat keys give rows for groups that never occurred. count and size differ wherever data is missing. Each default is defensible on its own. Together they mean a group-by result can fail to merge with the frame it came from in a few ways at once.
The check that catches all of them is one line. Compare the total of the result against the total of the input. If they differ, a thing was excluded. The reason will be one of the above.
Beyond that, the hands-on advice is to prefer named aggregation. This Makes flat column names and lets one input column feed a few statistics. To use string aggregation names rather than lambdas. This Is the gap between compiled code and a Python call per group.
One more thing
A groupby object supports nth(0). This Takes the nth row of each group and isn't the same as first(): first() skips missing values and nth(0) doesn't. When a group's first row has a gap, the two give other answers. Which you want depends on whether "first" means the first row or the first on hand value.
ngroup() numbers the groups. This Is a compact way to turn a set of keys into a single int label for indexing or plotting.
Questions people ask about groupby
What does groupby actually do? It splits the rows into groups by key, applies a function to each group, and combines the results. Nothing is computed until you call an aggregation.
Why is my grouping column now the index? By default the keys become the index of the result. Pass as_index=False or call reset_index() to get them back as a column.
What is the difference between size and count? size counts rows in each group, including missing values. count counts non-missing values in each column.
Why do rows with a missing key disappear? groupby drops NaN keys by default. Pass dropna=False to keep them as their own group.
How do I apply different aggregations to different columns? Use named aggregation: df.groupby('g').agg(total=('x', 'sum'), avg=('y', 'mean')).
Can I group by more than one column? Yes, pass a list of column names. The result then has a MultiIndex unless you set as_index=False.
groupby, in summary
Split, apply, combine — and the details that decide whether the answer reconciles.
Missing keys are dropped unless you say else. cat keys give rows for groups that never appeared. size counts rows and count counts values. They differ wherever data is missing.
Named aggregation is the clearest form and gives flat column names. String aggregation names are much faster than lambdas. And checking the result's total against the input's total is one line that catches the most common silent failure.
Check yourself
0 of 4
Answer without scrolling back up.
What happens to rows whose group key is NaN?
Group totals then quietly fail to match the frame total. Pass dropna=False, and check the totals match after any group-by that matters.
Why prefer named aggregation over a dict?
It also lets several statistics come from the same input column, and saves a column-flattening step almost every time.
A group has 5 rows, 2 with a missing sales value. What do `size()` and `count()` report for sales?
size counts rows, count counts non-missing values. Using count where you meant size undercounts exactly where data is imperfect.
Why is iterating over a groupby usually wrong for computing?
Iterating is fine for inspection. If you are accumulating results in a list inside the loop, there is nearly always an agg for it.
Cheat sheet
groupby
Split a DataFrame by one or more columns, apply aggregations and combine the results. See the split-apply-combine pattern that defines groupby.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.