Feed a sentence to a plain feed-forward network and watch three problems appear: fixed size, order blindness, and zero memory.
Problem 1 — Fixed Input Size
This network was built with exactly 6 input slots. Type sentences of different lengths and watch it truncate or pad.
Problem 2 — Order Blindness
A bag-of-words input layer sees only word counts. Compare two sentences that share every word:
The Three Limitations
01Fixed input size — real sentences vary in length; the network's input layer cannot.
02No sense of order — a bag-of-words vector is identical for any word arrangement.
03No memory — every input is processed in isolation; nothing carries over between steps.
Network Input Layer (6 slots, forever)
OK
Truncated — the network never sees these words:
Bag-of-Words: Two Sentences, One Vector
CLICK COMPARE
The comparison will appear here.
Why Plain Neural Networks Fail on Sequences
Three structural limitations of feed-forward networks — and why they motivated recurrent architectures.
The Setup
A classic feed-forward network (ANN / MLP) is a brilliant function approximator — for fixed-size, unordered inputs. Sequential data violates both assumptions at once: sentences have different lengths, and their meaning lives in the ordering. The result is three distinct failure modes.
Limitation 1: The Input Layer is a Fixed-Width Door
The first layer of an ANN has a hard-coded number of neurons. A 6-slot network offers exactly two bad options for real text: truncate longer inputs (information destroyed before learning even starts) or pad shorter ones with dummy values (the network wastes capacity learning to ignore filler). There is no third option — the architecture physically cannot stretch.
Limitation 2: Order Blindness
The standard fixed-size representation for text — bag of words — counts word occurrences and discards positions. "Dog bites man" and "man bites dog" produce bit-for-bit identical vectors, so the network is mathematically incapable of distinguishing them, no matter how long you train. Whatever information lives in the ordering is gone before the first neuron fires.
Limitation 3: No Memory Between Inputs
Each forward pass is an island. When an ANN processes word 5, it retains nothing about words 1–4 unless they were crammed into the same fixed input. There is no persistent state that flows from one step to the next — which is precisely the thing a recurrent cell adds, carrying a hidden state hₜ forward through time.
What a plain network cannot do with a sequence
A standard feed-forward network takes a fixed-size vector and produces an output. Text is not a fixed-size vector, and forcing it to be one costs three specific things.
It cannot handle variable length. A network with 100 input units takes exactly 100 numbers. A 5-word sentence and a 500-word document cannot both be fed to it without padding to a fixed maximum — wasting most of the input on short texts and truncating long ones.
It has no notion of order. Flatten "the dog bit the man" into 5 embedding vectors concatenated, and the network sees position 1, position 2 and so on as unrelated input slots. It can learn that slot 2 tends to hold a noun, and it learns nothing transferable about nouns at other positions.
It shares nothing across positions. A pattern learned at position 3 has to be learned again, separately, at position 40. Every position needs its own examples, so the data requirement multiplies by the sequence length.
That third point is the deepest. It is the same argument as for convolution in images: without parameter sharing along the relevant axis, the model wastes capacity relearning the same thing.
The concrete failure
Take sentiment classification on a fixed 100-word window with 300-dimensional embeddings. Flattened, that is 30,000 input values.
A first hidden layer of 512 units needs 30,000 × 512 = 15.4 million weights, for one layer, on a task a recurrent model handles with a few hundred thousand.
"Not good" appearing at words 3–4 and at words 60–61 are, to the network, two unrelated patterns in unrelated input slots.
A 12-word review pads to 100, so 88% of the input is filler, and the network must learn to ignore it.
A 200-word review is truncated, and the second half is simply unavailable.
Every one of those is a structural consequence of treating an ordered variable-length sequence as a fixed-size vector.
What each successor fixes
Architecture
Fixes
By
1-D CNN
Position sharing, some order
Sliding filters along the sequence
RNN
Variable length, order, sharing
One state carried step by step
LSTM / GRU
Long-range decay
Gated additive memory
Transformer
Range and parallelism
Attention plus positional encoding
A 1-D convolution is the smallest change that helps: a filter slides along the sequence, so a pattern learned anywhere applies everywhere, and the parameter count drops enormously. Its limit is the receptive field — a filter of width 5 sees five words, and reaching further needs depth.
An RNN processes one element at a time, carrying a state, so the same weights apply at every position and any length is accepted. Its limit is gradient decay over distance, and no parallelism.
A transformer makes every position one step from every other and computes them all simultaneously. Its limit is quadratic cost in length.
Four things a plain network cannot do with a sequence
A feed-forward network expects a fixed-size input and treats every position as a separate feature. Each of those two facts causes a specific failure, demonstrated here on data where the right answer is known.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
print("1. FIXED INPUT SIZE. a dense layer's weight matrix has one row per")
print(" input feature, chosen when you build it:")
for T in (5, 20, 100):
print(" %3d-token input -> first layer is %3d x hidden. a %d-token"
% (T, T, T + 1))
print(" %-16s sentence does not fit." % "")
print(" padding is the usual patch, and it wastes both compute and")
print(" capacity on positions that carry nothing:")
lengths = [4, 7, 31, 5, 9]
print(" real lengths %s -> pad everything to %d" % (lengths, max(lengths)))
print(" %.0f%% of the padded input is filler."
% (100 * (1 - sum(lengths) / (len(lengths) * max(lengths)))))
print()
print("2. NO WEIGHT SHARING ACROSS POSITIONS. position 3 and position 40")
print(" have entirely separate weights, so nothing learned at one")
print(" transfers to the other.")
T, H = 20, 16
dense = T * H
rnn = 1 * H + H * H
print(" dense layer over %d positions : %d weights" % (T, dense))
print(" one RNN cell reused %d times : %d weights" % (T, rnn))
print(" and here is what that costs. train a dense model to detect a")
print(" pattern that only ever appears at position 2, then test it when")
print(" the pattern moves:")
N = 600
def make(pos):
X = rng.normal(0, 0.4, (N, T))
y = rng.integers(0, 2, N)
X[y == 1, pos] += 3.0 # the signal, at one position
return X, y
Xtr, ytr = make(2)
w = np.linalg.lstsq(np.column_stack([Xtr, np.ones(N)]), ytr * 2.0 - 1, rcond=None)[0]
def acc(X, y):
return (((np.column_stack([X, np.ones(len(X))]) @ w) > 0) == (y == 1)).mean()
print(" trained with the signal at position 2:")
for test_pos in (2, 3, 8, 15):
Xte, yte = make(test_pos)
print(" signal at position %2d -> accuracy %.4f"
% (test_pos, acc(Xte, yte)))
print(" it learned 'look at column 2', not 'look for this pattern'. an RNN")
print(" or a convolution applies the same weights everywhere and does not")
print(" have to relearn the pattern per position.")
print()
print("3. NO NOTION OF ORDER BEYOND POSITION IDENTITY. the network sees")
print(" %d independent features, not a sequence:" % T)
a = np.zeros(T); a[3] = 1.0
b = np.zeros(T); b[4] = 1.0
c = np.zeros(T); c[19] = 1.0
print(" nothing in the architecture says position 3 is adjacent to 4")
print(" and far from 19. it has to learn that from data, per pair,")
print(" and there are %d pairs." % (T * (T - 1) // 2))
print()
print("4. NO MEMORY BETWEEN EXAMPLES. each forward pass is independent, so")
print(" streaming data has to be re-fed as overlapping windows:")
stream = np.arange(10)
W = 4
windows = [stream[i:i + W] for i in range(len(stream) - W + 1)]
print(" a stream of %d values, window %d:" % (len(stream), W))
for w_ in windows[:4]:
print(" %s" % w_)
print(" ...")
total = sum(len(x) for x in windows)
print(" %d values become %d window-values -- each one is recomputed"
% (len(stream), total))
print(" %.1f times on average." % (total / len(stream)))
print()
print("WHAT EACH ARCHITECTURE FIXES:")
rows = [("variable length", "no", "yes", "yes", "up to a limit"),
("shared weights", "no", "yes", "yes", "yes"),
("order built in", "no", "yes", "local only", "added explicitly"),
("parallel over time", "yes", "no", "yes", "yes"),
("long-range path", "direct", "O(n) steps", "O(n/k) layers", "direct")]
print("%22s %10s %12s %14s %18s"
% ("", "dense", "RNN", "1-D conv", "transformer"))
for r in rows:
print("%22s %10s %12s %14s %18s" % r)
print()
print("the dense row is not all bad -- it is parallel and has a direct path")
print("between any two positions, which is exactly what a transformer also")
print("has. what a transformer adds is weight sharing and variable length,")
print("which is why it replaced both the dense model and the RNN rather")
print("than sitting between them.")
Output
Experiments to try
Load the long sentence. The 9-word input overflows the 6 slots — the red strip shows words the network will simply never see.
Load the short sentence. Now the network pads with <PAD> tokens — filler the model has to learn to ignore.
Run the comparison. Both sentences produce the exact same bag-of-words vector — the "SAME VECTOR" verdict is the whole argument for sequence-aware models in one screenshot.
The short of it
ANNs don't fail on sequences because they are weak — they fail because their architecture makes three promises (fixed size, unordered input, stateless processing) that sequential data breaks. Recurrent networks are the structural fix: variable-length input consumed one step at a time, with a hidden state that remembers.
Where a plain network is still fine
The limitations are real and they are not universal. A feed-forward network on a fixed window is perfectly reasonable when:
The sequence has a genuinely fixed length — a sensor reading at 24 fixed hourly points, a form with a known number of fields.
The features are engineered summaries rather than raw sequence. Mean, standard deviation, trend, min and max over a window turn a sequence into a small fixed vector, and a plain network or a gradient-boosted model on those features is often excellent.
Order does not carry the signal — some classification tasks are genuinely bag-of-words problems.
You want a baseline. TF-IDF plus logistic regression is a feed-forward model on a fixed vector, and it is a strong baseline for text classification.
That second row is worth emphasising because it is the pragmatic answer in time-series work: windowing plus engineered features plus gradient boosting frequently beats a sequence model on business data with limited history, and it is far cheaper to build.
The parameter-sharing principle
Stepping back, the same idea appears wherever a structure repeats:
Domain
Repeats along
Shared by
Images
Space
Convolution filters
Sequences
Time
Recurrent weights
Sequences
Position
Transformer layers
Graphs
Nodes
Message functions
In each case the model encodes an assumption: that a pattern is equally meaningful wherever it occurs. That assumption is what reduces the parameter count and the data requirement, and it is exactly what a plain feed-forward network lacks along the sequence axis.
When the assumption is false — when absolute position genuinely matters — sharing hurts, which is why transformers need explicit positional encodings and why some vision tasks use locally-connected layers without sharing.
Questions people ask
Can I just pad everything to a fixed length? You can, and you inherit wasted capacity on short inputs, truncation on long ones, and no sharing across positions.
Is a 1-D CNN good enough for text? Often, for classification. It is fast and captures local phrases well. It cannot reach across a document the way attention can.
Why do transformers need positional encodings if they handle sequences? Attention itself is order-blind — it computes a weighted mixture over a set. Position has to be injected separately.
Do I always need a sequence model for time series? No. Windowing plus engineered features plus gradient boosting is a strong and often superior baseline.
What about very long documents? Chunk them, embed each chunk, and pool or retrieve — or use a long-context model. Feeding 50,000 tokens to a standard transformer is expensive.
Is order always important? No. For topic classification, bag-of-words often suffices. For anything involving negation, syntax or reasoning, it is essential.
Recap in one screen
A plain network needs fixed-size input, so sequences must be padded or truncated.
It treats each position as an unrelated input slot, so patterns are not shared across positions.
The parameter count explodes: a flattened 100-word window into 512 units is 15 million weights.
Convolutions share along position; RNNs carry a state; transformers attend across all positions.
Fixed-length or well-summarised sequences are still legitimate feed-forward problems.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Setup”?
A classic feed-forward network (ANN / MLP) is a brilliant function approximator — for fixed-size, unordered inputs. Sequential data violates both assumptions at once: sentences have different lengths, and their meaning lives in the ordering. The result is three distinct failure modes.
What does this module say about “Limitation 1: The Input Layer is a Fixed-Width Door”?
The first layer of an ANN has a hard-coded number of neurons. A 6-slot network offers exactly two bad options for real text: truncate longer inputs (information destroyed before learning even starts) or pad shorter ones with dummy values (the network wastes capacity learning to ignore filler). There is no third option — the architecture physically cannot stretch.
What does this module say about “Limitation 2: Order Blindness”?
The standard fixed-size representation for text — bag of words — counts word occurrences and discards positions. "Dog bites man" and "man bites dog" produce bit-for-bit identical vectors , so the network is mathematically incapable of distinguishing them, no matter how long you train. Whatever information lives in the ordering is gone before the first neuron fires.
Cheat sheet
Limitations of ANN with Sequential Data
A classic feed-forward network (ANN / MLP) is a brilliant function approximator — for fixed-size, unordered inputs. Sequential data violates both assumptions at once: sentences have different lengths, and their meaning lives in the ordering. The result is three distinct failure modes.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.