What is RNN?

Unrolling

An RNN is often drawn as a cell with an arrow looping back to itself. That is compact and slightly misleading. To understand training, unroll it: draw one copy of the cell per timestep, with the hidden state flowing left to right between copies.

Unrolled, an RNN over a 50-token sentence is a 50-layer deep network. Each layer is the same layer — the same weights — but the compute graph really is that deep. And the gradient really does have to travel all the way back through it. Weight sharing is the structural edge: a single set of parameters handles any sequence length. So a model trained on 20-token sentences can process 200-token ones at inference with no extra parameters. But the gradient that must travel back through 50 (or 200) the same matrix products either explodes or vanishes. This is why plain RNNs struggle with dependencies longer than about 10-20 steps. LSTMs and GRUs add gating methods that create an additive path for the gradient, mitigating the vanishing problem at the cost of more parameters per cell. The hidden state has a fixed size (the hidden dimension), no matter how long the sequence is. All data about the past must be compressed into that one vector. This is the basic bottleneck. Attention removes it by letting the decoder look right at each encoder state. This page unrolls an RNN across a sequence, shows the hidden-state flow, and covers how gradient size changes with depth.

Counting the parameters

A recurrent cell with input size d and hidden size h has:

W_x: d × h    W_h: h × h    b: h

total = dh + h² + h

With d = 100 and h = 128 that is 12,800 + 16,384 + 128 = 29,312 parameters. And that number does not change whether the sequence is 5 tokens or 5,000. The h² term often rules, which is why hidden size is the costly dimension and doubling it roughly quadruples the recurrent weights.

Backpropagation through time

Training unrolls the network, runs a normal forward pass. Then backpropagates from the loss all the way back to timestep 1. Because the same weights appear at each step, the gradient for W_h is the sum of its gradient shares from all timesteps.

This is called backpropagation through time. And it has two real results. Memory grows linearly with sequence length, since each middle hidden state must be kept for the backward pass. And the gradient reaching early timesteps is a long product. So it vanishes or explodes just as depth causes it to in a feedforward network.

The shapes a recurrent network can take

The cell is the same in each case. What changes is where inputs enter and where outputs are taken. And that decides what the network can do.

Many-to-one takes an output only at the end. Use pooled hidden states rather than only the final one. Since the final state over-weights the end of the input.

Many-to-many aligned takes an output at each step, and a bidirectional layer often helps because right context disambiguates.

Many-to-many unaligned needs two networks: an encoder that reads the input and a decoder that gives the output. This is where attention was invented.

The encoder-decoder, and the bottleneck that motivated attention

For translation, an encoder reads the source sentence and its final hidden state is passed to a decoder. This makes the target one token at a time.

That final state is a fixed-size vector. 512 numbers, say. And it must hold all of it about the source sentence. For five words that is comfortable. For fifty it is a severe compression, and translation grade was observed to fall sharply with sentence length.

Attention removed the bottleneck: instead of passing only the final state, keep all the encoder's hidden states, and let the decoder compute a weighted combination at each output step, choosing what to look at.

That change — added for just this problem — improved translation at once, and within a few years the attention method had displaced the recurrence it was invented to assist. "Attention is all you need" is a literal description of what happened next.

Depth, and how to add it

Stacking recurrent layers means feeding one layer's output sequence into the next:

lstm = nn.LSTM(300, 256, num_layers=2, batch_first=True, dropout=0.2)

The dropout input applies between layers, not within the recurrence. Applying dropout to the recurrent connection with a different mask each step damages the memory. This is why variational dropout (the same mask at each step) exists.

Two layers is the real limit for most tasks. Deep recurrent stacks are hard to train: the gradient must travel back through both depth and time. And the two decays compound. Residual connections between layers help and are less standard here than in transformers.

Two other structural choices:

Bidirectional layers run a second pass backwards and concatenate. This doubles the output width. Excellent for tagging and classification, unusable for generation or streaming.

Where the output head goes. On the final layer, always. But note that in a stacked bidirectional model, extracting the right final states needs care. Since each way's final state corresponds to a different end of the sequence.

One weight matrix, applied over and over

An RNN is a loop with shared weights. Unrolling it shows why that sharing is the whole idea and also the whole problem.

import numpy as np

rng = np.random.default_rng(0)
H, X, T = 4, 3, 6
Wx = rng.normal(0, 0.5, (X, H))
Wh = rng.normal(0, 0.5, (H, H))
b = np.zeros(H)
seq = rng.normal(0, 1.0, (T, X))

print("the entire architecture is one line, run in a loop:")
print("    h = tanh(x @ Wx + h @ Wh + b)")
print()
print("parameters: Wx %s + Wh %s + b %s = %d numbers, and they do NOT"
      % ((X, H), (H, H), (H,), Wx.size + Wh.size + b.size))
print("grow with sequence length. a 5-token sequence and a 5000-token one")
print("use exactly the same weights.")
print()

h = np.zeros(H)
print("%6s %28s %28s" % ("step", "input x", "hidden h"))
for t, x in enumerate(seq):
    h = np.tanh(x @ Wx + h @ Wh + b)
    print("%6d %28s %28s" % (t, np.round(x, 3), np.round(h, 4)))
print()
print("h is the only thing carried forward. everything the network knows")
print("about the first %d tokens has to fit in those %d numbers." % (T - 1, H))
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

print("THE SHARING IS THE POINT. compare against a feed-forward network")
print("that takes the whole sequence at once:")
for t in (10, 100, 1000):
    rnn = X * H + H * H + H
    ff = (t * X) * H + H
    print("   %5d tokens: RNN %6d parameters, feed-forward %8d"
          % (t, rnn, ff))
print("   and the feed-forward version can only ever accept exactly the")
print("   length it was built for. the RNN accepts any length, and what it")
print("   learned about position 3 applies at position 300.")
print()

print("UNROLLED, it is a very deep network with tied weights:")
print("   x0 -> [tanh] -> h0")
print("   x1 -> [tanh] -> h1     each box is the SAME Wh")
print("   x2 -> [tanh] -> h2")
print("   ...")
print("   a %d-token sequence is a %d-layer network. that is where the" % (T, T))
print("   trouble starts.")
print()

The below block covers the left steps.

import numpy as np

print("THE PROBLEM. the gradient from a late step back to step 0 passes")
print("through Wh once per step, so it is governed by that matrix's")
print("eigenvalues:")
print("   %-14s %18s %22s" % ("weight sd", "spectral radius", "after 50 steps"))
for scale in (0.15, 0.3, 0.5, 0.8):
    W = rng.normal(0, scale, (H, H))
    rho = np.abs(np.linalg.eigvals(W)).max()
    val = rho ** 50
    print("   %-14.2f %18.4f %22s"
          % (scale, rho, "%.3e" % val if val < 1e100 else "overflow"))
print("   below 1 it vanishes, above 1 it explodes, and only exactly 1")
print("   survives. tanh's derivative is at most 1 and usually less, which")
print("   pushes the product further down.")
print()
LONG = 40
long_seq = rng.normal(0, 1.0, (LONG, X))
h = np.zeros(H)
J = np.eye(H)
grads = []
for t, x in enumerate(long_seq):
    h = np.tanh(x @ Wx + h @ Wh + b)
    J = J @ (Wh * (1 - h ** 2))          # accumulate the Jacobian
    grads.append(np.abs(J).max())

The next block goes on with the left demonstrations.

print("   measured on a %d-step sequence with these actual weights --" % LONG)
print("   how much gradient still reaches step 0 from step t:")
for t in (0, 2, 5, 10, 20, 30, 39):
    g = grads[t]
    print("      from step %2d: %.3e  %s"
          % (t, g, "#" * max(0, int(30 * min(g, 1.0)))))
print("   by step %d it is %.1e of what it started as. no amount of"
      % (LONG - 1, grads[-1] / grads[0]))
print("   training fixes that -- it is a property of repeated multiplication.")
print()

print("WHAT AN RNN IS STILL GOOD FOR. it processes one token at a time in")
print("O(1) memory per step, where attention needs the whole sequence:")
for t in (128, 1024, 8192):
    print("   %5d tokens: RNN state %4d numbers, attention matrix %12s"
          % (t, H, "{:,}".format(t * t)))
print("   that is why RNN-shaped models keep being revisited for very long")
print("   sequences and for streaming, where you cannot hold the whole")
print("   input at once. state-space models are the current form of that")
print("   argument.")
print()
print("but for anything that fits in a context window, attention won on two")
print("counts: it has a direct path between any two positions rather than a")
print("chain of %d multiplications, and it is parallel across the sequence"
      % T)
print("where this loop is inherently sequential.")

Things to try

1. See where the parameters live. Press Parameters and read the breakdown. The recurrent matrix is the largest block, because it is h × h while the input matrix is only d × h.
2. Grow the hidden state. Raise Hidden State from 10 to 40 and check the parameter count again. It grows roughly with the square, not linearly. The h² term taking over.
3. Grow the input instead. Raise Input Vector by the same amount. The parameter count rises far less, because that dimension only appears in the d × h term.
4. Watch one step execute. Press Simulate Cell Step and follow the two matrix products combining and passing through tanh. That single step is what gets repeated once per token, with the same weights each time.

The sequence shapes an RNN can be wired into

• Many-to-one — read the whole sequence, use only the final hidden state. Sentiment classification.
• Many-to-many, aligned — one output per input. Part-of-speech tagging and named entity recognition.
• Many-to-many, unaligned — an encoder reads the input into a state and a decoder makes a different-length output. Translation, and the design attention was invented to improve.
• One-to-many — a single input makes a sequence. Image captioning.

Where this goes wrong

• Not clipping gradients. The repeated product makes explosion common. Clipping the global norm at around 1.0 is close to mandatory for recurrent models.
• Padding without masking. Batching needs equal lengths, so short sequences are padded. And unless those spots are masked, the model learns from the padding and the loss is computed over meaningless tokens.
• Running out of memory on long sequences. Each hidden state is retained for the backward pass. Truncate the BPTT window rather than reducing the batch to one.
• Expecting parallelism. Timestep t needs the state from t−1, so an RNN is inherently sequential and cannot use a GPU the way a transformer can. That, more than precision, is why transformers displaced them.

In one line

An RNN is one cell unrolled across time, sharing weights at each step. So its parameter count is fixed at dh + h² + h no matter sequence length. Training backpropagates through the whole unrolled graph, which makes the effective depth equal to the sequence length. Hence vanishing gradients, mandatory gradient clipping, and truncated BPTT. The sequential dependency is also what stops it parallelising, which is what in time made attention the better design.

Building one end to end

class TextClassifier(nn.Module):
    def __init__(self, vocab, emb=300, hidden=256, classes=2):
        super().__init__()
        self.embed = nn.Embedding(vocab, emb, padding_idx=0)
        self.lstm = nn.LSTM(emb, hidden, num_layers=2, batch_first=True,
                            bidirectional=True, dropout=0.3)
        self.drop = nn.Dropout(0.3)
        self.head = nn.Linear(hidden * 2, classes)     # *2 for bidirectional

    def forward(self, ids, lengths):
        x = self.embed(ids)
        packed = pack_padded_sequence(x, lengths, batch_first=True,
                                      enforce_sorted=False)
        out, _ = self.lstm(packed)
        out, _ = pad_packed_sequence(out, batch_first=True)

        mask = (ids != 0).unsqueeze(-1)
        pooled = (out * mask).sum(1) / mask.sum(1).clamp(min=1)
        return self.head(self.drop(pooled))

Four details in that code are the ones that matter in practice: padding_idx=0 keeps the pad embedding at zero, packing stops the LSTM handling padding, hidden * 2 accounts for bidirectionality. And the masked mean pooling ignores padding when summarising.

Getting any of the four wrong gives a model that trains and lags without an error message.

Training notes

Clip gradients. clip_grad_norm_(params, 5.0). Recurrent models are the classic case, and omitting it in time gives NaN.

Sort or bucket by length when batching, so batches hold close lengths and less work is wasted on padding.

Watch memory against sequence length. Trigger memory grows linearly with the number of steps. So a model that fits at 200 tokens may fail at 500.

Use pretrained embeddings where available, and think of freezing them with small datasets. There is far less to overfit.

Truncate backpropagation for very long sequences, detaching the state each k steps.

Questions people ask

How many layers should I use? One or two. Deeper recurrent stacks rarely repay the training trouble.

Should I use the last hidden state or pool? Pool, for classification.

Where does dropout go? Between layers via the dropout input, and after pooling before the head. Not on the recurrent connection with a per-step mask.

Bidirectional or not? Bidirectional for classification and tagging; unidirectional for generation and streaming.

Is an encoder-decoder RNN still used? Rarely for translation. The structure survives in transformers, with attention doing the work the recurrence used to.

Why did attention replace this? It removed the fixed-size bottleneck between encoder and decoder. Then proved enough without the recurrence at all.

Recap in one screen

• One cell, four arrangements: one-to-many, many-to-one, aligned and unaligned many-to-many.
• Unaligned tasks need an encoder and a decoder. And the fixed-size state between them was the bottleneck attention was invented to remove.
• Stack at most two layers; put dropout between them, not inside the recurrence.
• Bidirectional doubles the output width and needs the complete input.
• Pack padded batches, clip gradients, and pool over states rather than taking the last one.