Visualize how Long Short-Term Memory networks pass both a Cell State (long-term memory) and a Hidden State (short-term memory) forward to process sequences.
Standard Recurrent Neural Networks (RNNs) pass a single "hidden state" forward through time. While great in theory, standard RNNs suffer from the vanishing gradient problem. During training, as the network looks back across many time steps (long sentences), the signals necessary to update the weights become microscopically small.
The result? A standard RNN essentially develops "amnesia." By the time it reads the 20th word in a sentence, it has mostly forgotten the 1st word. This makes it terrible at understanding long-range dependencies in text.
Discover how Long Short-Term Memory networks use a dual-state architecture to overcome the limitations of standard RNNs.
LSTMs solve this amnesia by introducing a more complex internal structure that passes two distinct states forward at every time step (as seen in the visualization):
To control what gets added to or removed from the Cell State conveyor belt, the LSTM uses neural network layers called gates. Inside every single LSTM cell block shown in the visual, there are actually four separate fully-connected layers operating simultaneously:
Because there are 4 independent layers doing calculations on the inputs ($X_t$) and previous hidden state ($H_{t-1}$), an LSTM has roughly 4 times as many parameters as a standard RNN of the same size. Click the Parameters button to see the math.
Take "the film was not very good" and watch what an LSTM does with it, one token at a time.
At each step the cell receives the current word's embedding and its own two states from the previous step, computes four things from them, and produces new states.
| Step | Word | What the gates plausibly do |
|---|---|---|
| 1 | the | Little to store; state stays near its initial value |
| 2 | film | Input gate opens on subject dimensions; "film" is the topic |
| 3 | was | Tense recorded; subject retained (forget gate near 1) |
| 4 | not | A negation dimension is set; this is the important step |
| 5 | very | Intensifier stored; negation retained |
| 6 | good | Candidate is strongly positive — and the negation dimension is still set |
The interesting step is 4. "Not" carries no sentiment on its own; its job is to invert what follows. A trained cell stores that in some dimension and keeps it (forget gate near 1) so that at step 6 the positive candidate for "good" is combined with an active negation signal.
That is what a bag-of-words model cannot do: to it, this sentence contains "good" and is therefore positive.
fᵗ = σ(Wᶠ[hᵗ₋₁, xᵗ]) iᵗ = σ(Wᵢ[hᵗ₋₁, xᵗ])
gᵗ = tanh(Wᵕ[hᵗ₋₁, xᵗ]) oᵗ = σ(Wₒ[hᵗ₋₁, xᵗ])
cᵗ = fᵗ ⊙ cᵗ₋₁ + iᵗ ⊙ gᵗ hᵗ = oᵗ ⊙ tanh(cᵗ)
All four transformations take the same input and are computed as a single fused matrix multiplication in practice, then split.
The two states do different jobs. The cell state is long-term memory, updated additively so information can persist for many steps. The hidden state is the working output — passed to the next layer, to the prediction head, and to the next step's gate computations.
For a 20-word sentence, a single-layer LSTM with 256 hidden units produces a 20×256 tensor: one hidden state per word.
Which part you use depends on the task:
Classification. The final hidden state summarises the whole sentence — or better, pool over all states with a mean or max, since the final state is biased towards the end of the input.
Tagging (named entities, part of speech). Use every hidden state, one prediction per word.
Generation. Feed each hidden state to an output layer over the vocabulary, and the predicted token becomes the next step's input.
As an encoder. The final state (or all states, with attention) is passed to a decoder.
lstm = nn.LSTM(300, 256, batch_first=True)
out, (h, c) = lstm(embedded) # out: (batch, seq, 256)
pooled = out.mean(dim=1) # better than h[-1] for classificationPack variable-length sequences. Sentences in a batch have different lengths, so short ones are padded. Without pack_padded_sequence, the LSTM processes the padding, and the final hidden state describes the padding rather than the last real word.
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
packed = pack_padded_sequence(embedded, lengths, batch_first=True,
enforce_sorted=False)
out, (h, c) = lstm(packed)
out, _ = pad_packed_sequence(out, batch_first=True)Consider bidirectionality. For classification and tagging, where the whole sentence is available, running one LSTM forward and another backward and concatenating gives each position both left and right context. It typically helps materially. Never use it for generation or streaming, where the future does not exist.
Clip gradients. clip_grad_norm_ with a threshold of 1–5. Recurrent models are the classic case for it, and omitting it eventually produces a NaN run.
Keep it shallow. One or two layers. Deep recurrent stacks are hard to train and rarely repay the cost.
An LSTM wins when memory must be bounded — streaming inference carries one fixed-size state regardless of how much text has passed, while a transformer's KV cache grows with every token. It also wins on small datasets, where its inductive bias helps and a transformer trained from scratch overfits, and on-device, where it is small and needs no attention kernels.
A transformer wins on long-range dependencies (any two positions are one step apart), on parallelism (all positions computed at once, so training scales), and consequently on anything requiring scale.
| Task | Reasonable choice |
|---|---|
| Sentiment on a few thousand labelled reviews | LSTM, or a fine-tuned small transformer |
| Streaming transcription with bounded memory | LSTM |
| Document-level question answering | Transformer |
| On-device text classification | LSTM or a distilled transformer |
| Anything generative at quality | Transformer |
An LSTM reading a real sentence, with all four gates printed at every token and the cell state tracked alongside. The interesting part is which tokens make the gates move.
Does an LSTM read left to right? Yes, unless bidirectional, in which case a second pass reads right to left and the outputs are concatenated.
Should I use the last hidden state or pool? Pool, for classification — the last state over-weights the end of the sequence.
How long a text can it handle? Hundreds of tokens reliably. Beyond that, information from the start is largely gone.
Do I need embeddings before the LSTM? Yes — token ids must become vectors first, either from a learned embedding layer or pretrained vectors.
Why is my final hidden state meaningless? Almost certainly unpacked padding: the state describes the pad tokens.
One layer or two? Two at most for most tasks; the second sometimes helps and often does not.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Vanishing Gradient Problem”?
Standard Recurrent Neural Networks (RNNs) pass a single "hidden state" forward through time. While great in theory, standard RNNs suffer from the vanishing gradient problem . During training, as the network looks back across many time steps (long sentences), the signals necessary to update the weights become microscopically small.
What does this module say about “The Dual State Solution: Cell and Hidden States”?
LSTMs solve this amnesia by introducing a more complex internal structure that passes two distinct states forward at every time step (as seen in the visualization):
What does this module say about “The Four Gates (Why so many parameters?)”?
To control what gets added to or removed from the Cell State conveyor belt, the LSTM uses neural network layers called gates . Inside every single LSTM cell block shown in the visual, there are actually four separate fully-connected layers operating simultaneously:
Visualize how Long Short-Term Memory networks pass both a Cell State (long-term memory) and a Hidden State (short-term memory) forward to process sequences.