A sequence is data where order carries meaning. Shuffle a sentence and watch its meaning fall apart.
Input Sentence
Tip: Try "dog bites man" — a 3-word sentence where a single swap completely flips the meaning.
The Sequence (order preserved)
SEQUENCE INTACT
The Same Words as a Bag (order ignored)
A "bag of words" only knows which words appear and how often — never where. Notice this view never changes when you shuffle: to a bag, every ordering is identical.
Sequences Beyond Text
Stock prices
102 → 105 → 103 → 110
Yesterday's price helps predict today's.
DNA
A → T → G → C → A
Rearranging bases changes the protein.
Music
C → E → G → C
The same notes in a different order is a different melody.
Overview
The Core Idea
A sequence is an ordered collection of items where the spot of each item carries meaning. "Dog bites man" and "man bites dog" hold just the same three words, yet they describe opposite events. That gap lives fully in the order. And order is just what ordinary tabular data does not have. Text, audio, time series, DNA, and video are all sequences. Their elements have a spot, and moving an element changes the meaning. A model that ignores order (like a bag-of-words classifier) discards data that is often more key than the words themselves. Sequential models. RNNs, LSTMs, GRUs, and transformers — are designs designed to preserve and exploit that ordering. An RNN processes tokens one at a time, carrying a hidden state forward; a transformer processes all tokens in parallel but injects spot through positional encodings. Both give an output that depends on the full ordered context, not just the single elements. The choice of model depends on the sequence length and the task: RNNs are simple but struggle with long-range dependencies. While transformers handle long sequences well but cost more memory. This page shows how shuffling a sequence changes its meaning, compares bag-of-words against order-aware forms, and covers the three main designs for sequential data.
The foundational idea behind all sequential models — from sliding windows to RNNs and Transformers.
The Core Idea
A sequence is an ordered collection of items where the spot of each item carries meaning. "Dog bites man" and "man bites dog" hold just the same three words, yet they describe opposite events. That gap lives fully in the order. And order is just what ordinary tabular data does not have. Text, audio, time series, DNA, and video are all sequences. Their elements have a spot, and moving an element changes the meaning. A model that ignores order (like a bag-of-words classifier) discards data that is often more key than the words themselves. Sequential models. RNNs, LSTMs, GRUs, and transformers — are designs designed to preserve and exploit that ordering. An RNN processes tokens one at a time, carrying a hidden state forward; a transformer processes all tokens in parallel but injects spot through positional encodings. Both give an output that depends on the full ordered context, not just the single elements. The choice of model depends on the sequence length and the task: RNNs are simple but struggle with long-range dependencies. While transformers handle long sequences well but cost more memory. This page shows how shuffling a sequence changes its meaning, compares bag-of-words against order-aware forms, and covers the three main designs for sequential data.
Sequential vs. Tabular Data
In a classic spreadsheet-style dataset, each row is free. Shuffling the rows of a housing-price table changes nothing about what a model can learn. Sequential data breaks this assumption in two ways:
Order matters: the items form a meaningful progression — words in a sentence, prices over days, frames in a video.
Context matters: the meaning of an element depends on its neighbours. The word "bank" means something different after "river" than after "savings".
Because of this, sequence models need a way to recall or look at what came before — the reason behind sliding windows, recurrent networks, and attention.
Order is part of the data
A sequence is data where the order carries meaning. Shuffle it and you have destroyed data, not merely rearranged it.
"The dog bit the man" and "The man bit the dog" hold the same words and describe different events. A stock price series read backwards describes a different market. A DNA sequence read out of order codes for nothing.
Compare that with a table of customers: shuffle the rows and nothing is lost, because each row is free of the others. That independence is what most standard machine learning assumes, and sequences violate it.
Data
Sequential?
Why
Text
Yes
Word order is grammar
Audio waveform
Yes
Time ordering is the signal
Video
Yes
Frame order is motion
Sensor readings
Yes
Time ordering carries trend
Customer records
No
Rows are independent
Images
Spatially, not temporally
2-D structure, no time axis
What makes sequences hard
Four properties, each needing a set architectural response.
Variable length. One sentence is 5 words, another 500. Fixed-size inputs cannot accommodate both, so models must either pad and mask, or process step by step.
Order matters. Any model treating the input as an unordered set is unusable. This is why attention needs positional encodings bolted on.
Long-range dependencies. A pronoun may refer to a noun fifty words earlier; a stock's behaviour may depend on last quarter. Designs that decay data over distance cannot capture this.
Elements are not free. Standard train/test splitting assumes independence. Shuffling a time series before splitting lets the model see the future. This gives excellent scores and worthless models.
That last point is the one that catches people in practice. Sequence data needs chronological splits, and windows that overlap the boundary must be excluded.
The designs, in order of history
Model
How it handles order
Limitation
Bag of words
It does not
Order lost entirely
n-grams
Local windows of n
Cannot reach beyond n
RNN
A carried hidden state
Gradients decay; no parallelism
LSTM / GRU
Gated state with an additive path
Long ranges still hard
1-D CNN
Local filters over time
Fixed receptive field
Transformer
Attention plus positional encoding
Quadratic in length
State-space (Mamba)
Recurrence with linear scaling
Newer, less tooling
Each row fixes the last row's main problem. The transformer's dominance comes from two structural wins. Constant path length between any two spots, and full parallelism across the sequence. At the cost of quadratic attention.
What makes data sequential, and how to check
Order matters or it does not, and that single question decides your whole design. Here is a test that measures it rather than assuming, run on data where the answer is known both ways.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
print("a sequence is data where the ORDER carries information. the test is")
print("simple: shuffle it and see whether the answer changes.")
print()
N, T = 400, 12
X_seq = rng.normal(size=(N, T))
# the label depends on the ORDER: is the second half larger than the first?
y_seq = (X_seq[:, T // 2:].mean(1) > X_seq[:, :T // 2].mean(1)).astype(int)
# and one where it does not: is the total positive?
y_bag = (X_seq.sum(1) > 0).astype(int)
def shuffled_agreement(X, y_fn):
perm = rng.permutation(T)
return (y_fn(X) == y_fn(X[:, perm])).mean()
print("two labels over the SAME %d x %d data:" % (N, T))
print(" 'second half larger than the first' -- depends on order")
print(" 'the total is positive' -- does not")
print()
print("shuffle the columns and see how often the label survives:")
for name, fn in (("order-dependent",
lambda X: (X[:, T // 2:].mean(1) > X[:, :T // 2].mean(1)).astype(int)),
("order-free",
lambda X: (X.sum(1) > 0).astype(int))):
agree = np.mean([shuffled_agreement(X_seq, fn) for _ in range(20)])
print(" %-20s label unchanged after shuffling: %.4f" % (name, agree))
print(" the second is exactly 1.0 -- a sum does not care about order.")
print(" the first is near chance, which is what 'sequential' means.")
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
print("THAT TEST IS WORTH RUNNING ON REAL DATA, because the answer is not")
print("always obvious:")
cases = [
("words in a sentence", "yes", "'dog bites man' vs 'man bites dog'"),
("pixels in an image", "sort of", "2-D neighbourhood, not a 1-D order"),
("rows in a customer table", "no", "each row is independent"),
("daily sales", "yes", "trend, seasonality, autocorrelation"),
("items in a shopping basket", "no", "a set, not a sequence"),
("clicks in a session", "yes", "what you looked at before matters"),
("DNA bases", "yes", "codons are read in triples, in order"),
("bag of words", "no", "by construction -- the name says so"),
]
print("%30s %10s %s" % ("data", "sequential", "why"))
for a, b, c in cases:
print("%30s %10s %s" % (a, b, c))
print()
Output
The below block covers the left steps.
example_03.pyNumPy
import numpy as np
print("THE PROPERTIES THAT COME WITH IT, and what each one breaks:")
print()
print(" 1. VARIABLE LENGTH. sentences are not all the same length.")
lengths = [len(s.split()) for s in
("hi", "the cat sat", "the quick brown fox jumps over the lazy dog")]
print(" %s -> a fixed-input model needs one shape and these are %s"
% (lengths, "different"))
print(" that is why you pad, mask, or use an architecture that loops.")
print()
print(" 2. LONG-RANGE DEPENDENCE. the useful signal can be far away:")
sent = "the keys that i left on the kitchen table are missing"
print(" %r" % sent)
w = sent.split()
print(" 'are' agrees with 'keys' -- %d words earlier, across a clause"
% (w.index("are") - w.index("keys")))
print(" that has its own singular noun ('table') sitting closer.")
print()
print(" 3. AUTOCORRELATION. neighbouring values are not independent, so")
print(" the usual statistics quietly stop applying:")
walk = np.cumsum(rng.normal(size=500))
iid = rng.normal(size=500)
for name, s in (("random walk", walk), ("independent draws", iid)):
r1 = np.corrcoef(s[:-1], s[1:])[0, 1]
Output
The next block goes on with the left demonstrations.
example_04.pyNumPy
print(" %-20s correlation with itself one step back: %+.4f"
% (name, r1))
print(" a shuffled train/test split on the first row leaks the answer:")
print(" neighbouring points are nearly identical, so a test point sits")
print(" between two training points that already tell you its value.")
print()
print("HOW EACH ARCHITECTURE HANDLES ORDER:")
rows = [("bag of words", "discards it", "fixed", "no"),
("n-gram", "local window only", "fixed", "no"),
("RNN / LSTM", "built in, sequentially", "any", "yes"),
("1-D convolution", "local window, learned", "any", "no"),
("transformer", "added, via positions", "up to a limit", "yes")]
print("%18s %26s %16s %14s"
% ("model", "how it sees order", "input length", "long range?"))
for a, b, c, d in rows:
print("%18s %26s %16s %14s" % (a, b, c, d))
print()
print("the transformer row is the odd one: attention is order-blind by")
print("construction, so position has to be ADDED to the input. every other")
print("row gets order from its structure.")
print()
print("so the first question about any dataset is the shuffle test above.")
print("if the label survives a shuffle, you do not have sequential data and")
print("a sequence model will only cost you parameters.")
Output
Tests to try
Shuffle the default sentence. Watch the top panel. The words get amber borders and the read-out sentence becomes gibberish. While the "bag" panel below is fully unchanged. The bag never noticed.
Type "dog bites man" and click "Reverse the Order". Three words, one reversal, opposite meaning — the smallest doable case that spot is data.
Restore the first. The status badge flips back to green the moment each word returns to its home index.
The short of it
A sequence is not just a set of values — it is values plus their order. Any model that throws the order away (like a bag of words) throws data away with it. All of it else in this NLP track — sliding windows, encodings, embeddings, recurrent cells — exists to let neural networks use that ordering data instead of losing it.
Preparing sequence data
Any the model, the same practical decisions arise.
Fix the length. Pad short sequences and truncate long ones, and supply an attention mask so the model ignores padding. Which end you truncate matters: for classification the beginning is often most informative; for time series the most recent values are.
Window it. For forecasting, slide a window along the series so each spot becomes one training case. The last k values as features, the next value as the target. This converts a sequence into an ordinary supervised table.
Split chronologically. Train on the past, validate on the middle, test on the most recent. Never shuffle. Leave a gap when windows would if not straddle the boundary.
Scale using training-period stats only. Computing a mean over the whole series leaks the future into the past.
# windowing a series into (features, target) pairs
X, y = [], []
for i in range(len(series) - window):
X.append(series[i:i + window])
y.append(series[i + window])
Sequence-to-what?
The shape of the task decides the design's output arrangement:
Shape
Example
Sequence → one label
Sentiment, intent classification
Sequence → label per element
Named entities, part-of-speech tagging
Sequence → different-length sequence
Translation, summarisation
Sequence → next element
Forecasting, language modelling
One input → sequence
Image captioning, text-to-speech
For the first, pool over all spots rather than taking only the last. The final state is biased towards the end of the input.
For the third, an encoder-decoder or a decoder-only model with the input as a prefix. This is where the attention method was at first invented, to remove the bottleneck of compressing a whole source sentence into one vector.
Questions people ask
Are images sequences? They have spatial structure but no time axis. So convolutions suit them better. Vision transformers do treat patches as a sequence, with 2-D positional encodings.
Can I use a standard neural network on sequences? Only by flattening a fixed-length window. This discards the notion of order and cannot generalise across spots. It is a baseline, not a solution.
How long can a sequence be? LSTMs handle hundreds of steps reliably; transformers reach hundreds of thousands of tokens with engineering effort and quadratic cost.
Why can I not shuffle time series before splitting? Because the model would train on the future and be tested on the past. This is not a power it will have in production.
What about irregular timestamps? Resample to a regular count, or use a model that takes the time delta as an explicit input.
Is a transformer always better than an LSTM? Not always. For streaming with bounded memory, small datasets and on-device inference, recurrent models remain the practical choice.
Recap in one screen
A sequence is data where order carries meaning, so elements are not free.
Variable length, order response, long-range dependence and non-independence are the four troubles.
Padding with masks, or step-by-step handling, handles variable length.
Split chronologically and scale on training-period stats — shuffling leaks the future.
Transformers win on path length and parallelism; recurrent models still win on streaming and small data.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Core Idea”?
What does this module say about “Sequential vs. Tabular Data”?
What does this module say about “Order is part of the data”?
Cheat sheet
What is a Sequence?
See why 'dog bites man' and 'man bites dog' differ: order carries meaning, and models that ignore it lose the most important signal.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.