resample, rolling and shift - the three operations that need the rows to be in time order.
Overview
The index does the work
Everything in this module depends on the frame having a DatetimeIndex. resample raises without one, and time-based windows have nothing to measure against.
df.set_index("date") after pd.to_datetime is the usual setup, and it is the step people skip before wondering why resample will not run.
Once the index is a timeline, pandas can do things that would otherwise need a lot of arithmetic: select a month by name, aggregate by period, window by duration rather than by row count.
Worth knowing
resample and time-based rolling require a DatetimeIndex and raise without one.
resample("ME").sum() is group-by for time — downsampling aggregates, and takes agg like any group-by.
Upsampling creates empty rows: ffill asserts the value held, interpolate asserts it moved smoothly. Different claims.
rolling(n) gives NaN until the window fills; min_periods=1 starts immediately, and rolling("3D") windows by time.
shift, diff and pct_change compare each row with its past — the basis of lagged features.
All of these assume the rows are in time order, and pandas does not check. sort_index() first.
Time Series
resample, rolling and shift.
A DatetimeIndex is the prerequisite
resample and rolling-by-time both need one.
example_01.pypandas
Output
resample changes the frequency
Downsampling aggregates; it is groupby for time.
example_02.pypandas
Output
Upsampling creates gaps you must fill
Going finer invents rows, and they start empty.
example_03.pypandas
Output
rolling windows
A moving statistic over the last n rows, or the last n days.
example_04.pypandas
Output
shift compares a row with its past
The basis of differences, growth rates and lagged features.
example_05.pypandas
Output
Order matters, and pandas will not check it
Every operation here assumes the rows are sorted in time.
example_06.pypandas
Output
resample
resample is group-by where the groups are time periods.
Downsampling — daily to monthly — aggregates, and takes the same methods as groupby: sum(), mean(), agg(["sum", "mean"]).
The frequency string is the argument that matters. D daily, W weekly, ME month end, MS month start, QE quarter end, YE year end, h hourly, min minutely.
Note that pandas 2.2 renamed several of these. M became ME, H became h, T became min. Older code and older tutorials use the old spellings, which now warn or fail depending on version. If a frequency string does not work, that rename is the likely reason.
The label of each output row is the end of the period for ME, and the start for MS. label= and closed= control which boundary is used and which side is inclusive, and they matter when periods must line up with something external.
Upsampling invents rows
Going to a finer frequency creates rows that did not exist, and they start as NaN.
asfreq() leaves them empty, which is honest and rarely useful on its own.
ffill() carries the last known value forward. That asserts the value held constant until the next observation — correct for a price, a status, a setting.
interpolate() fits between the known points. That asserts the value moved smoothly — correct for a temperature, a position, anything continuous.
These are different claims about the world, and choosing between them is a modelling decision rather than a formatting one. Both fabricate data; the question is which fabrication is less wrong for what the number means.
rolling
s.rolling(3).mean() gives a moving average over the last three rows.
The first two results are NaN, because the window is not yet full. That is deliberate — a three-row average computed from one row is not a three-row average. min_periods=1 overrides it and starts computing immediately, which is convenient and slightly dishonest at the edges.
s.rolling("3D") windows by time rather than row count. This is the one to use when observations are irregularly spaced, because three rows might span three days or three months, and only the time-based window means what you said.
center=True puts the window around each point rather than behind it. It is right for smoothing a curve for display and wrong for anything predictive, because it uses future values.
expanding() is a window that grows from the start — a running total or a cumulative mean.
ewm() gives exponentially weighted statistics, where recent observations count for more.
shift, diff, pct_change
shift(1) moves every value down one row, so each row can see the previous one. shift(-1) looks forward.
diff() is s - s.shift(1). pct_change() is the relative version.
These are how you build lagged features, growth rates and change detection. The first row is always NaN, because it has no predecessor.
Within groups, use the group-by versions — df.groupby("id")["v"].diff() — or the shift crosses from one entity into another and produces a difference between unrelated rows. That is a common and quiet error in panel data.
Order is assumed, not checked
Every operation here — shift, diff, rolling, resample — assumes the rows are in time order.
pandas does not verify it. diff() on unsorted data returns numbers, and they are meaningless.
sort_index() before any of these on data you did not sort yourself. It is one line, and the failure it prevents produces plausible output rather than an error.
The same applies to duplicated timestamps: two rows for the same instant make diff and rolling ambiguous, and are usually a sign of a data problem worth looking at before aggregating over it.
Frequency strings
The alias is the argument that decides what resample and date_range do, and pandas 2.2 renamed several of them.
D day, B business day, W week (ending Sunday by default, W-MON to change it), MS month start, ME month end, QS/QE quarter, YS/YE year, h hour, min minute, s second.
The renames: M to ME, H to h, T to min, S to s. Older code and most tutorials use the old spellings, which now warn or fail. When a frequency string does not work, this is the first thing to check.
Multiples work: "15min", "2W", "3ME".
Anchored offsets pin the boundary: "W-FRI" for weeks ending Friday, "QE-JAN" for quarters ending in January, which matters for fiscal years that do not start in January.
Which label a period gets
resample("ME") labels each group with the end of the period; "MS" with the start.
label="left" or "right" overrides which boundary names the group, and closed= decides which end is inclusive.
These matter whenever the output must line up with something produced elsewhere. A monthly total labelled 2024-01-31 and one labelled 2024-01-01 are the same number with different index labels, and joining them gives nothing.
Settling on a convention early — usually period start — saves reconciliation later.
Grouping by time and something else
resample groups by time only. To group by time and a category, pd.Grouper combines with ordinary keys:
key= names the date column, so this works without setting an index, which is what makes Grouper more useful than resample in practice.
The result is a MultiIndex, and unstack("city") gives the wide table.
Rolling in more depth
min_periods decides how many observations a window needs before producing a value. The default for a fixed-size window is the window size; for a time-based window it is 1.
closed= controls whether the window includes its endpoints, which matters for time-based windows where an observation may sit exactly on the boundary.
win_type gives weighted windows — Gaussian, triangular — for smoothing.
rolling(...).apply(func, raw=True) runs a custom function per window. raw=True passes a NumPy array rather than a Series and is substantially faster.
Within groups, df.groupby("id")["v"].rolling(3).mean() windows inside each group and returns a MultiIndexed result, which usually needs droplevel(0) before it can be assigned back.
Gaps, and what they hide
A rolling window over rows assumes the rows are evenly spaced. Real time series have gaps — weekends, outages, missing readings — and a three-row window may span three days or three weeks.
Two fixes:
Window by time: rolling("3D") means three days whatever the row spacing.
Regularise first: resample("D").mean() produces a row per day, with NaN where there was no data, after which row-based windows mean what they say.
Which is right depends on whether an absent observation should count as missing or simply not exist. That is a modelling question, and resample forces you to answer it, which is an argument for doing it early.
A checklist for time-series work
Parse the dates and check the dtype.
Set a DatetimeIndex, or plan to use pd.Grouper(key=...).
sort_index().
Check for duplicated timestamps.
Decide on a time zone, or commit to naive throughout.
Decide whether gaps are missing or absent, and regularise if they are missing.
Then resample, rolling and shift mean what they appear to mean.
Aligning two series with different frequencies
A common task: daily data and monthly targets, joined for comparison.
The wrong approach is a merge on dates, which matches almost nothing.
The right approach is to bring both to the same frequency first:
Or, going the other way, upsample the monthly figure and forward-fill it so every day carries its month's target.
Which direction is right depends on the question. Aggregating up loses detail and is usually correct for reporting; spreading down invents precision and is usually correct for per-row comparison. Either way, the alignment step is explicit rather than implied by a join.
MS versus ME matters here: two monthly series labelled at opposite ends of the month will not join at all.
Lags and leads for modelling
shift(1) gives the previous value — a lag, safe to use as a predictor.
shift(-1) gives the next value — a lead, which is the target in a forecasting problem and must never be a feature.
Using a lead as a feature is data leakage, and it produces models that score beautifully and fail in production. Because both are one method with a sign, the mistake is easy to make and invisible in the code.
Rolling features have the same hazard: rolling(3, center=True) uses future values. For anything predictive, windows must look backwards only, which is the default without center.
Within groups, all of these need the group-wise form, or the lag crosses from one entity to the next.
Missing periods
A gap in a time series is either "no observation" or "no event", and the two need different handling.
resample("D").sum() gives 0 for days with no rows — correct if the series counts events.
resample("D").mean() gives NaN — correct if the series measures something that existed but was not recorded.
Choosing the wrong one produces a series that looks complete and is wrong at exactly the interesting points.
asfreq() makes the gaps explicit without aggregating, which is the honest first step when you are not yet sure.
A summary
A DatetimeIndex is the prerequisite for resample and time-based windows.
Frequency aliases were renamed in pandas 2.2 — ME, h, min.
Downsampling aggregates; upsampling invents rows that need filling deliberately.
ffill asserts a value held; interpolate asserts it moved smoothly.
rolling("3D") for irregular spacing; min_periods for the edges.
shift(-1) is a lead and belongs only in a target, never a feature.
Use the group-wise forms on panel data.
sort_index() first, always, because none of these check.
A closing note
Time series work rewards setting things up properly and punishes assumptions.
The setup is short: parse the dates with an explicit format, check the dtype, decide on a time zone, set a DatetimeIndex if you need period selection, and sort. Every operation in this module assumes that order and none of them check it, so diff on unsorted rows returns numbers that mean nothing.
The recurring decision is what a gap means. A day with no rows is either zero events or an unrecorded measurement, and resample("D").sum() and .mean() encode those two different answers. Choosing without noticing gives a series that looks complete and is wrong exactly where the interesting things happen.
And on panel data — several entities stacked in one frame — shift, diff and rolling must be done within groups, or the comparison runs across the boundary from one entity into the next and produces a plausible number from unrelated rows.
One more thing
resample accepts origin= and offset=, which control where the period boundaries fall. That matters for data that should be bucketed on something other than midnight or the first of the month — a business day starting at 6am, or weeks aligned to a fiscal calendar.
Check yourself
0 of 4
Answer without scrolling back up.
What does `resample` require?
df.set_index('date') after pd.to_datetime is the usual setup, and the step people skip before wondering why resample will not run.
What is the difference between `ffill` and `interpolate` when upsampling?
Different claims about the world. Both fabricate data - the question is which fabrication is less wrong for what the number means.
Why does `rolling(3).mean()` start with NaN?
min_periods=1 overrides it, which is convenient and slightly dishonest at the edges.
What does pandas do if you call `diff()` on rows that are not in time order?
Order is assumed, not checked. sort_index() before any shift, diff, rolling or resample on data you did not sort yourself.
Cheat sheet
Time Series
Everything in this module depends on the frame having a DatetimeIndex. resample raises without one, and time-based windows have nothing to measure against.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.