Feed a sentence to a plain feed-forward network and watch three problems appear: fixed size, order blindness, and zero memory.
Problem 1 — Fixed Input Size
This network was built with exactly 6 input slots. Type sentences of different lengths and watch it truncate or pad.
Problem 2 — Order Blindness
A bag-of-words input layer sees only word counts. Compare two sentences that share every word:
The Three Limitations
01Fixed input size — real sentences vary in length; the network's input layer cannot.
02No sense of order — a bag-of-words vector is identical for any word arrangement.
03No memory — every input is processed in isolation; nothing carries over between steps.
Network Input Layer (6 slots, forever)
OK
Truncated — the network never sees these words:
Bag-of-Words: Two Sentences, One Vector
CLICK COMPARE
The comparison will appear here.
Overview
The Setup
A classic feed-forward network (ANN / MLP) is a brilliant function approximator — for fixed-size, unordered inputs. Sequential data violates both assumptions at once: sentences have else lengths. And their meaning lives in the sorting. The result is three distinct failure modes.
The first is the fixed-size input: a feed-forward network has a hard-coded count of input neurons. So variable-length sequences must be either truncated (losing data) or padded (wasting capacity on filler). The second is order blindness: a bag-of-words form discards word spots fully. This makes "dog bites man" and "man bites dog" the same inputs. The third is the absence of memory between time steps: each forward pass is free. So the network retains nothing about last inputs unless they were concatenated into the same fixed vector.
These three limitations are not minor inconveniences — they are structural. No amount of training data or deeper layers can teach a feed-forward network to distinguish word order when the input form has now destroyed it. And no fixed-width input can gracefully handle both a 3-word query and a 3,000-word document.
Recurrent neural networks (RNNs) address all three by handling one element at a time and carrying a hidden state forward. The hidden state is a fixed-size vector that builds up data from last time steps. This gives the network a form of memory. LSTMs and GRUs refine this with gating methods that control what to recall and what to forget. Transformers take a else approach fully: they process the whole sequence in parallel with self-attention. This computes pairwise links between each spot without the sequential bottleneck of recurrence.
Grasp why feed-forward networks fail on sequences clarifies what recurrent and attention-based designs were designed to fix — and why the jump from MLPs to RNNs to transformers was driven by the data's structure, not just by scale.
Feed a sentence to a plain feed-forward network and watch three problems appear: fixed size, order blindness, and zero memory.
Three structural limitations of feed-forward networks — and why they motivated recurrent architectures.
The Setup
A classic feed-forward network (ANN / MLP) is a brilliant function approximator — for fixed-size, unordered inputs. Sequential data violates both assumptions at once: sentences have else lengths. And their meaning lives in the sorting. The result is three distinct failure modes.
The first is the fixed-size input: a feed-forward network has a hard-coded count of input neurons. So variable-length sequences must be either truncated (losing data) or padded (wasting capacity on filler). The second is order blindness: a bag-of-words form discards word spots fully. This makes "dog bites man" and "man bites dog" the same inputs. The third is the absence of memory between time steps: each forward pass is free. So the network retains nothing about last inputs unless they were concatenated into the same fixed vector.
These three limitations are not minor inconveniences — they are structural. No amount of training data or deeper layers can teach a feed-forward network to distinguish word order when the input form has now destroyed it. And no fixed-width input can gracefully handle both a 3-word query and a 3,000-word document.
Recurrent neural networks (RNNs) address all three by handling one element at a time and carrying a hidden state forward. The hidden state is a fixed-size vector that builds up data from last time steps. This gives the network a form of memory. LSTMs and GRUs refine this with gating methods that control what to recall and what to forget. Transformers take a else approach fully: they process the whole sequence in parallel with self-attention. This computes pairwise links between each spot without the sequential bottleneck of recurrence.
Grasp why feed-forward networks fail on sequences clarifies what recurrent and attention-based designs were designed to fix — and why the jump from MLPs to RNNs to transformers was driven by the data's structure, not just by scale.
Limitation 1: The Input Layer is a Fixed-Width Door
The first layer of an ANN has a hard-coded count of neurons. A 6-slot network offers just two bad options for real text: truncate longer inputs (data destroyed before learning still starts) or pad shorter ones with dummy vals (the network wastes capacity learning to ignore filler). There is no third option — the design physically cannot stretch.
Limitation 2: Order Blindness
The standard fixed-size form for text — bag of words — counts word instances and discards spots. "Dog bites man" and "man bites dog" give bit-for-bit the same vectors. So the network is in math incapable of distinguishing them, no matter how long you train. Any data lives in the sorting is gone before the first neuron fires.
Limitation 3: No Memory Between Inputs
Each forward pass is an island. When an ANN processes word 5, it retains nothing about words 1–4 unless they were crammed into the same fixed input. There is no persistent state that flows from one step to the next —. This is just the thing a recurrent cell adds, carrying a hidden state hₜ forward through time.
What a plain network cannot do with a sequence
A standard feed-forward network takes a fixed-size vector and gives an output. Text is not a fixed-size vector, and forcing it to be one costs three set things.
It cannot handle variable length. A network with 100 input units takes just 100 counts. A 5-word sentence and a 500-word document cannot both be fed to it without padding to a fixed max. Wasting most of the input on short texts and truncating long ones.
It has no notion of order. Flatten "the dog bit the man" into 5 embedding vectors concatenated. And the network sees spot 1, spot 2 and so on as unrelated input slots. It can learn that slot 2 tends to hold a noun. And it learns nothing transferable about nouns at else spots.
It shares nothing across spots. A pattern learned at spot 3 has to be learned again, apart, at spot 40. Each spot needs its own cases, so the data requirement multiplies by the sequence length.
That third point is the deepest. It is the same input as for convolution in images. Without parameter sharing with the apt axis, the model wastes capacity relearning the same thing.
The concrete failure
Take sentiment classification on a fixed 100-word window with 300-dimensional embeddings. Flattened, that is 30,000 input vals.
A first hidden layer of 512 units needs 30,000 × 512 = 15.4 million weights, for one layer, on a task a recurrent model handles with a few hundred thousand.
"Not good" appearing at words 3–4 and at words 60–61 are, to the network, two unrelated patterns in unrelated input slots.
A 12-word review pads to 100, so 88% of the input is filler. And the network must learn to ignore it.
A 200-word review is truncated, and the second half is simply unavailable.
Each one of those is a structural result of treating an ordered variable-length sequence as a fixed-size vector.
What each successor fixes
Architecture
Fixes
By
1-D CNN
Position sharing, some order
Sliding filters along the sequence
RNN
Variable length, order, sharing
One state carried step by step
LSTM / GRU
Long-range decay
Gated additive memory
Transformer
Range and parallelism
Attention plus positional encoding
A 1-D convolution is the smallest change that helps: a filter slides with the sequence. So a pattern learned anywhere applies all over. And the parameter count drops enormously. Its limit is the receptive field — a filter of width 5 sees five words, and reaching more needs depth.
An RNN processes one element at a time, carrying a state. So the same weights apply at each spot and any length is accepted. Its limit is gradient decay over distance, and no parallelism.
A transformer makes each spot one step from each else and computes them all at once. Its limit is quadratic cost in length.
Four things a plain network cannot do with a sequence
A feed-forward network expects a fixed-size input and treats each spot as a separate feature. Each of those two facts causes a set failure, showed here on data where the right answer is known.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
print("1. FIXED INPUT SIZE. a dense layer's weight matrix has one row per")
print(" input feature, chosen when you build it:")
for T in (5, 20, 100):
print(" %3d-token input -> first layer is %3d x hidden. a %d-token"
% (T, T, T + 1))
print(" %-16s sentence does not fit." % "")
print(" padding is the usual patch, and it wastes both compute and")
print(" capacity on positions that carry nothing:")
lengths = [4, 7, 31, 5, 9]
print(" real lengths %s -> pad everything to %d" % (lengths, max(lengths)))
print(" %.0f%% of the padded input is filler."
% (100 * (1 - sum(lengths) / (len(lengths) * max(lengths)))))
print()
print("2. NO WEIGHT SHARING ACROSS POSITIONS. position 3 and position 40")
print(" have entirely separate weights, so nothing learned at one")
print(" transfers to the other.")
T, H = 20, 16
dense = T * H
rnn = 1 * H + H * H
print(" dense layer over %d positions : %d weights" % (T, dense))
print(" one RNN cell reused %d times : %d weights" % (T, rnn))
print(" and here is what that costs. train a dense model to detect a")
print(" pattern that only ever appears at position 2, then test it when")
print(" the pattern moves:")
N = 600
def make(pos):
X = rng.normal(0, 0.4, (N, T))
y = rng.integers(0, 2, N)
X[y == 1, pos] += 3.0 # the signal, at one position
return X, y
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
Xtr, ytr = make(2)
w = np.linalg.lstsq(np.column_stack([Xtr, np.ones(N)]), ytr * 2.0 - 1, rcond=None)[0]
def acc(X, y):
return (((np.column_stack([X, np.ones(len(X))]) @ w) > 0) == (y == 1)).mean()
print(" trained with the signal at position 2:")
for test_pos in (2, 3, 8, 15):
Xte, yte = make(test_pos)
print(" signal at position %2d -> accuracy %.4f"
% (test_pos, acc(Xte, yte)))
print(" it learned 'look at column 2', not 'look for this pattern'. an RNN")
print(" or a convolution applies the same weights everywhere and does not")
print(" have to relearn the pattern per position.")
print()
print("3. NO NOTION OF ORDER BEYOND POSITION IDENTITY. the network sees")
print(" %d independent features, not a sequence:" % T)
a = np.zeros(T); a[3] = 1.0
b = np.zeros(T); b[4] = 1.0
c = np.zeros(T); c[19] = 1.0
print(" nothing in the architecture says position 3 is adjacent to 4")
print(" and far from 19. it has to learn that from data, per pair,")
print(" and there are %d pairs." % (T * (T - 1) // 2))
print()
Output
The below block covers the left steps.
example_03.pyNumPy
import numpy as np
print("4. NO MEMORY BETWEEN EXAMPLES. each forward pass is independent, so")
print(" streaming data has to be re-fed as overlapping windows:")
stream = np.arange(10)
W = 4
windows = [stream[i:i + W] for i in range(len(stream) - W + 1)]
print(" a stream of %d values, window %d:" % (len(stream), W))
for w_ in windows[:4]:
print(" %s" % w_)
print(" ...")
total = sum(len(x) for x in windows)
print(" %d values become %d window-values -- each one is recomputed"
% (len(stream), total))
print(" %.1f times on average." % (total / len(stream)))
print()
print("WHAT EACH ARCHITECTURE FIXES:")
rows = [("variable length", "no", "yes", "yes", "up to a limit"),
("shared weights", "no", "yes", "yes", "yes"),
("order built in", "no", "yes", "local only", "added explicitly"),
("parallel over time", "yes", "no", "yes", "yes"),
("long-range path", "direct", "O(n) steps", "O(n/k) layers", "direct")]
print("%22s %10s %12s %14s %18s"
% ("", "dense", "RNN", "1-D conv", "transformer"))
for r in rows:
print("%22s %10s %12s %14s %18s" % r)
print()
print("the dense row is not all bad -- it is parallel and has a direct path")
print("between any two positions, which is exactly what a transformer also")
print("has. what a transformer adds is weight sharing and variable length,")
print("which is why it replaced both the dense model and the RNN rather")
print("than sitting between them.")
Output
Tests to try
Load the long sentence. The 9-word input overflows the 6 slots — the red strip shows words the network will simply never see.
Load the short sentence. Now the network pads with <PAD> tokens — filler the model has to learn to ignore.
Run the check. Both sentences give the exact same bag-of-words vector — the "SAME VECTOR" verdict is the whole input for sequence-aware models in one screenshot.
The short of it
ANNs don't fail on sequences since they are weak — they fail. This is since their design makes three promises (fixed size, unordered input, stateless handling) that sequential data breaks. Recurrent networks are the structural fix: variable-length input consumed one step at a time, with a hidden state that recalls.
Where a plain network is still fine
The limitations are real and they are not universal. A feed-forward network on a fixed window is perfectly fair when:
The sequence has a truly fixed length — a sensor reading at 24 fixed hourly points, a form with a known count of fields.
The features are engineered summaries rather than raw sequence. Mean, standard deviation, trend, min and max over a window turn a sequence into a small fixed vector. And a plain network or a gradient-boosted model on those features is much excellent.
Order does not carry the sign — some classification tasks are truly bag-of-words problems.
You want a baseline. TF-IDF plus logistic regression is a feed-forward model on a fixed vector. And it is a strong baseline for text classification.
That second row is worth emphasising since it is the pragmatic answer in time-series work. Windowing plus engineered features plus gradient boosting much beats a sequence model on business data with limited history. And it is far cheaper to build.
The parameter-sharing principle
Stepping back, the same idea appears wherever a structure repeats:
Domain
Repeats along
Shared by
Images
Space
Convolution filters
Sequences
Time
Recurrent weights
Sequences
Position
Transformer layers
Graphs
Nodes
Message functions
In each case the model encodes an assumption: that a pattern is equally meaningful wherever it occurs. That assumption is what reduces the parameter count and the data requirement. And it is just what a plain feed-forward network lacks with the sequence axis.
When the assumption is false — when absolute spot truly matters — sharing hurts. This is why transformers need explicit positional encodings and why some vision tasks use locally-connected layers without sharing.
Questions people ask
Can I just pad all of it to a fixed length? You can. And you inherit wasted capacity on short inputs, truncation on long ones. And no sharing across spots.
Is a 1-D CNN good enough for text? Much, for classification. It is fast and captures local phrases well. It cannot reach across a document the way attention can.
Why do transformers need positional encodings if they handle sequences? Attention it is order-blind. It computes a weighted mixture over a set. Spot has to be injected apart.
Do I always need a sequence model for time series? No. Windowing plus engineered features plus gradient boosting is a strong and much superior baseline.
What about quite long documents? Chunk them, embed each chunk, and pool or retrieve. Or use a long-context model. Feeding 50,000 tokens to a standard transformer is costly.
Is order always key? No. For topic classification, bag-of-words much suffices. For a thing involving negation, syntax or reasoning, it is key.
Recap in one screen
A plain network needs fixed-size input, so sequences must be padded or truncated.
It treats each spot as an unrelated input slot, so patterns are not shared across spots.
The parameter count explodes: a flattened 100-word window into 512 units is 15 million weights.
Convolutions share with spot; RNNs carry a state; transformers attend across all spots.
Fixed-length or well-summarised sequences are still legitimate feed-forward problems.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Setup”?
What does this module say about “Limitation 1: The Input Layer is a Fixed-Width Door”?
What does this module say about “Limitation 2: Order Blindness”?
Cheat sheet
Limitations of ANN with Sequential Data
See the three ways a feed-forward network fails on sequences: fixed input size, order blindness, and no memory between steps.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.