Neural networks learn from (input, target) pairs. A sliding window manufactures those pairs out of one long sequence.
Overview
The Problem It Solves
Supervised learning needs pairs: an input X and the correct answer y. But a time series or a sentence arrives as one long, unlabeled stream. The sliding window converts that stream into training data by declaring: "the last w values are the input, and the very next value is the target."
Window Controls
Window size3
The Math
sequence length N = 14
window size w = 3
training samples N - w = 11
Daily Temperature Sequence
Input X Target y
Generated Training Pairs
0 SAMPLES
#
Input X
Target y
Click "Slide One Step" to create the first training pair.
The Sliding Window: Turning One Sequence into Many Examples
How raw sequential data becomes the (X, y) pairs that supervised learning requires.
How It Works
Given a sequence of length N and a window of size w, slide the window one position at a time:
Sample 1: X = [v₁, ..., vₓ], y = vₓ₊₁
Sample 2: X = [v₂, ..., vₓ₊₁], y = vₓ₊₂
... and so on until the window hits the end, producing N − w samples.
The same trick works for text: given the words "the cat sat on", predict "mat". That formulation — predict the next token from a fixed context — is the seed of the idea that grew into modern language models.
Choosing the Window Size
Too small: the model can't see far enough back — it misses weekly rhythms, long dependencies, sentence-level context.
Too large: fewer training samples (N − w shrinks), more parameters, and irrelevant ancient history dilutes the signal.
Fixed forever: whatever w you pick, the network's view of the past is frozen at that width — a limitation that recurrent networks were invented to remove.
Turning a stream into training examples
A model needs rows: a fixed number of inputs and a target. A sequence is one long stream. The sliding window is the conversion.
Pick a window length, slide it along, and each position becomes one example: the window is the input, whatever follows is the target.
With the token sequence [the, cat, sat, on, the, mat] and a window of 3:
Input
Target
the, cat, sat
on
cat, sat, on
the
sat, on, the
mat
Six tokens have become three training examples. From here any model that takes fixed-size input will work — a feed-forward network, gradient boosting, or a transformer with a fixed context.
Two consequences fall out immediately. The first window tokens produce no example, because there is no complete history for them. And consecutive examples overlap heavily, so they are not independent — which is exactly why the usual random train/test split is invalid.
The three settings
Window size (lookback) — how much history each example sees. Too short and the pattern is invisible: a window of 3 cannot represent a weekly cycle in daily data. Too long and you have many features, fewer examples, and ancient history diluting recent signal.
Start from the domain. Daily data with weekly seasonality wants at least 7. Hourly data with a daily cycle wants at least 24. Then confirm with an autocorrelation plot, which shows directly how far back the useful dependence reaches.
Horizon — how far ahead the target sits. One step is easiest. For several steps there are three strategies: recursive (predict one, feed it back, repeat — errors compound), direct (a separate model per horizon — more models, no compounding), and multi-output (one model predicting the whole vector at once).
Stride — how far the window moves each time. A stride of 1 extracts every possible window and maximises training examples; larger strides reduce overlap and training time at the cost of fewer examples.
def windows(seq, size, horizon=1, stride=1):
X, y = [], []
for i in range(0, len(seq) - size - horizon + 1, stride):
X.append(seq[i:i + size])
y.append(seq[i + size:i + size + horizon])
return X, y
Splitting without cheating
This is where most sliding-window projects go wrong, and the failure produces excellent metrics and a useless model.
Window 1 contains positions 1–3; window 2 contains 2–4. Put one in training and the other in test, and the test example shares two thirds of its input with a training example. Worse, a shuffled split lets the model train on later data and be evaluated on earlier data — a capability it will never have in production.
The correct approach is chronological: train on the earliest portion, validate on the middle, test on the most recent. For cross-validation, use an expanding window:
Fold
Train on
Test on
1
Months 1–6
Month 7
2
Months 1–7
Month 8
3
Months 1–8
Month 9
Scikit-learn's TimeSeriesSplit implements exactly this. Add a gap of at least window steps between train and test so no test window overlaps a training window.
The same rule applies to scaling: fit the scaler on the training period only. Computing a mean over the whole series leaks the future into the past.
Turning a stream into rows, and the leak that follows
A sliding window is how a sequence becomes a supervised dataset. The windows overlap, which creates near-duplicate rows -- and every mistake in this article follows from that one fact.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
series = np.round(100 + np.cumsum(rng.normal(0, 1.0, 60)), 2)
print("a series of %d values:" % len(series))
print(" %s ..." % series[:10])
print()
def windows(s, lookback, horizon=1, stride=1):
X, y = [], []
for i in range(0, len(s) - lookback - horizon + 1, stride):
X.append(s[i:i + lookback])
y.append(s[i + lookback + horizon - 1])
return np.array(X), np.array(y)
L, HZ = 5, 1
X, y = windows(series, L, HZ)
print("lookback=%d, horizon=%d:" % (L, HZ))
print("%8s %42s %10s" % ("row", "features", "target"))
for i in range(4):
print("%8d %42s %10.2f" % (i, X[i], y[i]))
print(" ...")
print(" %d values became %d rows." % (len(series), len(X)))
print()
print("look at rows 0 and 1. they share %d of their %d values:"
% (L - 1, L))
print(" row 0 %s" % X[0])
print(" row 1 %s" % X[1])
print(" and row 0's TARGET is row 1's last feature. the rows are not")
print(" independent, and nothing downstream knows that.")
print()
print("HOW MANY ROWS YOU ACTUALLY GET:")
print("%10s %10s %10s %14s %s" % ("lookback", "horizon", "stride", "rows", "overlap"))
for lb, hz, st in ((5, 1, 1), (20, 1, 1), (20, 1, 5), (20, 10, 1), (50, 1, 1)):
n = max(0, (len(series) - lb - hz) // st + 1)
print("%10d %10d %10d %14d %13.0f%%"
% (lb, hz, st, n, 100 * max(0, lb - st) / lb))
print(" a longer lookback costs rows at both ends. a stride above 1 buys")
print(" independence back by throwing rows away.")
print()
print("THE LEAK. split those rows at random and the test set is surrounded")
print("by its own neighbours:")
split = int(0.8 * len(X))
perm = rng.permutation(len(X))
Xs, ys = X[perm], y[perm]
print(" chronological: train rows 0-%d, test rows %d-%d"
% (split - 1, split, len(X) - 1))
print(" shuffled : test row 0 was originally row %d, and rows %d and %d"
% (perm[split], perm[split] - 1, perm[split] + 1))
print(" are almost certainly in TRAINING.")
overlap = 0
train_idx = set(perm[:split].tolist())
for t in perm[split:]:
if (t - 1) in train_idx or (t + 1) in train_idx:
overlap += 1
print(" %d of %d shuffled test rows have an immediate neighbour in the"
% (overlap, len(X) - split))
print(" training set -- %.0f%% of them." % (100 * overlap / (len(X) - split)))
print(" those neighbours share %d of %d feature values and a nearly"
% (L - 1, L))
print(" identical target. the model is being tested on what it memorised.")
print()
def fit_eval(Xtr, ytr, Xte, yte):
A = np.column_stack([Xtr, np.ones(len(Xtr))])
w = np.linalg.lstsq(A, ytr, rcond=None)[0]
P = np.column_stack([Xte, np.ones(len(Xte))])
return np.abs(P @ w - yte).mean()
print("measured, with a k-nearest-neighbour style model that can memorise:")
def knn_mae(Xtr, ytr, Xte, yte, k=1):
out = []
for xq, yq in zip(Xte, yte):
d = np.abs(Xtr - xq).sum(1)
out.append(abs(ytr[np.argsort(d)[:k]].mean() - yq))
return np.mean(out)
print(" 1-NN, chronological split : MAE %.4f"
% knn_mae(X[:split], y[:split], X[split:], y[split:]))
print(" 1-NN, shuffled split : MAE %.4f"
% knn_mae(Xs[:split], ys[:split], Xs[split:], ys[split:]))
print(" the shuffled number is better and it is not real. the nearest")
print(" neighbour of a test window is the window one step away from it,")
print(" which the shuffle put in training.")
print()
print("THE OTHER DECISIONS, briefly:")
print(" HORIZON -- predicting 1 step ahead and 20 steps ahead are")
print(" different problems, and a model trained for one is not a model")
print(" for the other:")
for hz in (1, 5, 20):
Xh, yh = windows(series, 10, hz)
sp = int(0.8 * len(Xh))
naive = np.abs(Xh[sp:, -1] - yh[sp:]).mean()
print(" horizon %2d: %d rows, naive 'repeat the last value' MAE %.4f"
% (hz, len(Xh), naive))
print()
print(" SCALING -- fit the scaler on the training rows only. a scaler fitted")
print(" on the whole series has already seen the future's mean and sd.")
print()
print(" PADDING -- if you pad short sequences, mask the padding out of the")
print(" loss. a model that is scored on predicting zeros will learn to")
print(" predict zeros.")
print()
print("the one rule underneath all of it: rows made from overlapping windows")
print("are not independent samples, and every technique that assumes they")
print("are -- shuffling, random cross-validation, bootstrap -- is wrong here.")
Output
Experiments to try
Click "Slide One Step" a few times. Watch the green window and amber target move together along the series, and each step append one row to the training table.
Press "Auto Play" and let the window consume the whole sequence. The sample counter stops at exactly N − w.
Drag the window size to 6 and replay. Bigger context per sample — but count how many fewer samples you end up with.
In one line
The sliding window is the simplest bridge between sequential data and supervised learning: it manufactures labeled examples from an unlabeled stream. Its fixed width is both its strength (simplicity) and its weakness — the model can never look further back than w steps, which is exactly the limitation that motivates recurrent architectures.
For text specifically
Language model training is exactly this pattern, at scale.
A corpus is tokenised into one long stream, chunked into windows of the context length, and each window's target is the same window shifted by one position. Predicting token t from tokens 1…t−1 is the sliding window with a horizon of 1 and the causal mask doing the work.
Two practical details:
Chunking versus true sliding. Extracting every possible window with stride 1 gives the most examples and enormous redundancy. Language model training typically uses non-overlapping chunks (stride = window), which is far cheaper and loses little, since each token still appears in a training example.
Document boundaries. Concatenating documents into one stream means some windows straddle two unrelated documents. Most implementations accept this; careful ones insert a separator token or mask attention across the boundary.
For classification rather than generation, windowing is used differently: a long document is split into overlapping chunks, each is classified or embedded, and the results are pooled or the highest-scoring chunk is used.
Features beyond the raw window
The lagged values are a starting point, not the finished feature set. For numeric series, three additions usually earn their place:
Rolling statistics — mean, standard deviation, min and max over the window. A rolling mean is often a stronger feature than the individual lags.
Calendar features — day of week, month, hour, holiday flags. Most human-generated series have strong calendar structure. Encode cyclical values as sine and cosine pairs so hour 23 sits next to hour 0.
Differences — the change from the previous value, or the ratio to the same time last week. This matters especially for tree models, which cannot extrapolate: a forest trained on values between 100 and 200 will never predict 250, but it can predict "a 5% increase" from a differenced target.
import pandas as pd
df = pd.DataFrame({"y": series})
for lag in range(1, 8):
df[f"lag_{lag}"] = df["y"].shift(lag)
df["roll_mean_7"] = df["y"].shift(1).rolling(7).mean()
The .shift(1) before .rolling() is essential. Without it the rolling mean at time t includes the value at time t — the thing being predicted. That is a leak, it is easy to write by accident, and it produces suspiciously excellent results.
Questions people ask
How do I choose the window size? From known seasonality, confirmed by an autocorrelation plot, then tuned with time-series cross-validation.
Should windows overlap? For small datasets, yes — stride 1 maximises examples. For large corpora, non-overlapping chunks are cheaper and adequate.
Why can I not shuffle? Overlapping windows leak across the split, and a shuffled split lets the model see the future.
What about missing timestamps? Resample to a regular frequency first and decide explicitly how to fill gaps. Windows that silently span a gap are meaningless.
How much data do I need? Enough to cover several full seasonal cycles — two years of daily data to learn annual seasonality.
Can one model serve many series? Often better than one model each: train across all series with an identifier feature, so series with little history borrow strength from the others.
Recap in one screen
A sliding window turns a stream into fixed-size (input, target) rows that any model can consume.
Window size sets how far back, horizon sets how far ahead, stride sets the overlap.
Consecutive windows overlap, so a random split leaks — split chronologically with a gap.
Fit scalers on the training period only, and shift before rolling or you leak the target.
Language model training is this pattern with a horizon of one and non-overlapping chunks.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
The sliding window is the simplest bridge between sequential data and supervised learning: it manufactures labeled examples from an unlabeled stream. Its fixed width is both its strength (simplicity) and its weakness — the model can never look further back than w steps, which is exactly the limitation that motivates recurrent architectures.
What does this module say about “The Problem It Solves”?
Supervised learning needs pairs: an input X and the correct answer y . But a time series or a sentence arrives as one long, unlabeled stream. The sliding window converts that stream into training data by declaring: "the last w values are the input, and the very next value is the target."
What does this module say about “How It Works”?
Given a sequence of length N and a window of size w, slide the window one position at a time:
Cheat sheet
Sequential Data Preparation with Sliding Window
Supervised learning needs pairs: an input X and the correct answer y. But a time series or a sentence arrives as one long, unlabeled stream. The sliding window converts that stream into training data by declaring: "the last w values are the input, and the very next value is the target."
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.