Take a microscopic look inside a single Recurrent Neural Network block. Visualize how it simply merges past memory with new input data to generate a new context state.
An RNN is usually drawn as a cell with an arrow looping back to itself. That is compact and slightly misleading. To understand training, unroll it: draw one copy of the cell per timestep, with the hidden state flowing left to right between copies.
Unrolled, an RNN over a 50-token sentence is a 50-layer deep network. Every layer is the same layer — identical weights — but the computational graph really is that deep, and the gradient really does have to travel all the way back through it.
Unrolling a recurrent cell across time gives a network as deep as the sequence is long - sharing one set of weights the whole way. That is what makes RNNs efficient, and what makes them hard to train.
A recurrent cell with input size d and hidden size h has:
Wx: d × h Wh: h × h b: h
total = dh + h² + h
With d = 100 and h = 128 that is 12,800 + 16,384 + 128 = 29,312 parameters — and that number does not change whether the sequence is 5 tokens or 5,000. The h² term usually dominates, which is why hidden size is the expensive dimension and doubling it roughly quadruples the recurrent weights.
Training unrolls the network, runs a normal forward pass, then backpropagates from the loss all the way back to timestep 1. Because the same weights appear at every step, the gradient for Wh is the sum of its gradient contributions from all timesteps.
This is called backpropagation through time, and it has two practical consequences. Memory grows linearly with sequence length, since every intermediate hidden state must be kept for the backward pass. And the gradient reaching early timesteps is a long product, so it vanishes or explodes exactly as depth causes it to in a feedforward network.
Truncated BPTT is the standard mitigation: backpropagate only k steps back, typically 20 to 50, and treat anything earlier as constant. It bounds memory and gradient depth at the cost of never learning dependencies longer than k.The cell is the same in every case. What changes is where inputs enter and where outputs are taken, and that determines what the network can do.
| Shape | Input → output | Example |
|---|---|---|
| One-to-many | One vector → sequence | Image captioning |
| Many-to-one | Sequence → one value | Sentiment classification |
| Many-to-many, aligned | Sequence → same-length sequence | Part-of-speech tagging |
| Many-to-many, unaligned | Sequence → different-length sequence | Translation |
Many-to-one takes an output only at the end. Use pooled hidden states rather than only the final one, since the final state over-weights the end of the input.
Many-to-many aligned takes an output at every step, and a bidirectional layer usually helps because right context disambiguates.
Many-to-many unaligned needs two networks: an encoder that reads the input and a decoder that produces the output. This is where attention was invented.
For translation, an encoder reads the source sentence and its final hidden state is passed to a decoder, which generates the target one token at a time.
That final state is a fixed-size vector — 512 numbers, say — and it must contain everything about the source sentence. For five words that is comfortable. For fifty it is a severe compression, and translation quality was observed to fall sharply with sentence length.
Attention removed the bottleneck: instead of passing only the final state, keep all the encoder's hidden states, and let the decoder compute a weighted combination at each output step, choosing what to look at.
That change — introduced for exactly this problem — improved translation immediately, and within a few years the attention mechanism had displaced the recurrence it was invented to assist. "Attention is all you need" is a literal description of what happened next.
Stacking recurrent layers means feeding one layer's output sequence into the next:
lstm = nn.LSTM(300, 256, num_layers=2, batch_first=True, dropout=0.2)The dropout argument applies between layers, not within the recurrence — applying dropout to the recurrent connection with a different mask each step damages the memory, which is why variational dropout (the same mask at every step) exists.
Two layers is the practical limit for most tasks. Deep recurrent stacks are hard to train: the gradient must travel back through both depth and time, and the two decays compound. Residual connections between layers help and are less standard here than in transformers.
Two other structural choices:
Bidirectional layers run a second pass backwards and concatenate, doubling the output width. Excellent for tagging and classification, unusable for generation or streaming.
Where the output head goes. On the final layer, always — but note that in a stacked bidirectional model, extracting the right final states requires care, since each direction's final state corresponds to a different end of the sequence.
An RNN is a loop with shared weights. Unrolling it shows why that sharing is the whole idea and also the whole problem.
An RNN is one cell unrolled across time, sharing weights at every step, so its parameter count is fixed at dh + h² + h regardless of sequence length. Training backpropagates through the whole unrolled graph, which makes the effective depth equal to the sequence length — hence vanishing gradients, mandatory gradient clipping, and truncated BPTT. The sequential dependency is also what stops it parallelising, which is what eventually made attention the better architecture.
class TextClassifier(nn.Module):
def __init__(self, vocab, emb=300, hidden=256, classes=2):
super().__init__()
self.embed = nn.Embedding(vocab, emb, padding_idx=0)
self.lstm = nn.LSTM(emb, hidden, num_layers=2, batch_first=True,
bidirectional=True, dropout=0.3)
self.drop = nn.Dropout(0.3)
self.head = nn.Linear(hidden * 2, classes) # *2 for bidirectional
def forward(self, ids, lengths):
x = self.embed(ids)
packed = pack_padded_sequence(x, lengths, batch_first=True,
enforce_sorted=False)
out, _ = self.lstm(packed)
out, _ = pad_packed_sequence(out, batch_first=True)
mask = (ids != 0).unsqueeze(-1)
pooled = (out * mask).sum(1) / mask.sum(1).clamp(min=1)
return self.head(self.drop(pooled))Four details in that code are the ones that matter in practice: padding_idx=0 keeps the pad embedding at zero, packing stops the LSTM processing padding, hidden * 2 accounts for bidirectionality, and the masked mean pooling ignores padding when summarising.
Getting any of the four wrong produces a model that trains and underperforms without an error message.
Clip gradients. clip_grad_norm_(params, 5.0). Recurrent models are the classic case, and omitting it eventually produces NaN.
Sort or bucket by length when batching, so batches contain similar lengths and less computation is wasted on padding.
Watch memory against sequence length. Activation memory grows linearly with the number of steps, so a model that fits at 200 tokens may fail at 500.
Use pretrained embeddings where available, and consider freezing them with small datasets — there is far less to overfit.
Truncate backpropagation for very long sequences, detaching the state every k steps.
How many layers should I use? One or two. Deeper recurrent stacks rarely repay the training difficulty.
Should I use the last hidden state or pool? Pool, for classification.
Where does dropout go? Between layers via the dropout argument, and after pooling before the head. Not on the recurrent connection with a per-step mask.
Bidirectional or not? Bidirectional for classification and tagging; unidirectional for generation and streaming.
Is an encoder-decoder RNN still used? Rarely for translation. The structure survives in transformers, with attention doing the work the recurrence used to.
Why did attention replace this? It removed the fixed-size bottleneck between encoder and decoder, and then proved sufficient without the recurrence at all.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
An RNN is one cell unrolled across time, sharing weights at every step, so its parameter count is fixed at dh + h² + h regardless of sequence length. Training backpropagates through the whole unrolled graph, which makes the effective depth equal to the sequence length — hence vanishing gradients, mandatory gradient clipping, and truncated BPTT.
What does this module say about “Unrolling”?
An RNN is usually drawn as a cell with an arrow looping back to itself. That is compact and slightly misleading. To understand training, unroll it: draw one copy of the cell per timestep, with the hidden state flowing left to right between copies.
What does this module say about “Internal Flow”?
Unrolling a recurrent cell across time gives a network as deep as the sequence is long - sharing one set of weights the whole way. That is what makes RNNs efficient, and what makes them hard to train.
Take a microscopic look inside a single Recurrent Neural Network block. Visualize how it simply merges past memory with new input data to generate a new context state.