How RNN Processes Text

Why language needs memory

Standard neural networks have a major limitation: they have no memory of the past. They process each input on its own. This is a problem for language, where the order of words is important. For example, "dog bites man" and "man bites dog" use the same words but have fully different meanings.

The Unrolled RNN: Step-by-Step Handling

The visualization above "unrolls" the RNN loop, showing it as a sequence of the same cells, one for each word (or "time step"). Here’s how the data flows:

1. Time Step 1 (First Word): The network takes the first word's embedding (a numeric vector, $X_1$) and an initial hidden state ($H_0$, usually all zeros). It processes them to produce an output ($Y_1$) and, most notably, a new hidden state ($H_1$).
2. Time Step 2 (Second Word): The network takes the second word's embedding ($X_2$) and the previous hidden state ($H_1$). This is the memory! $H_1$ contains data about the first word. This gives the network context. It then produces a new output ($Y_2$) and an updated hidden state ($H_2$).
3. And so on...: This process repeats for every word in the sequence. Each step receives the current word and the memory of all preceding words.

Shared Weights: The Key to Learning

A important concept in RNNs is shared weights. The same set of calculations (the same "brain") is used at every single time step. This is incredibly efficient. Instead of learning a new set of rules for the first word, second word, etc., the RNN learns a single set of rules for how to update its memory based on a new word and its previous memory. The "Parameters" calculation in the visualization shows the size of these shared weights.

One word at a time, carrying a state

A recurrent network processes a sequence step by step. At each step it combines the current input with what it remembers from before:

hᵗ = tanh(Wₕₕ hᵗ₋₁ + Wₓₕ xᵗ + b)

hₜ is the hidden state — a fixed-size vector summarising all of it seen so far. xₜ is the current word's embedding. The same weight matrices are used at every step, which is parameter sharing along the time axis.

Walking through "the cat sat":

Each state is a function of the current word and the previous state. So data propagates forward by being repeatedly transformed.

That single design choice — one weight matrix reused at every step — is what lets an RNN handle sequences of any length with a fixed number of parameters.

The architectures it supports

For classification, use the final hidden state — or better, pool over all states. Since the final one is biased towards the end of the sequence.

For an encoder-decoder, the encoder's final state is the "thought vector" passed to the decoder. That design is exactly where attention was invented. Compressing a whole sentence into one fixed vector is a bottleneck, and letting the decoder look back at every encoder state instead removed it.

Two structural limitations

Vanishing gradients. Backpropagation through 50 steps multiplies 50 Jacobian factors. If they average below 1, the gradient reaching step 1 is in effect zero. So the network cannot learn that step 1 mattered. Tanh's derivative is at most 1 and usually well below it. So decay is the default.

No parallelism. Step t needs hₜ₋₁, so the steps must run in order. A GPU's thousands of cores sit mostly idle, and training time scales with sequence length. This is a hardware-utilisation problem, and it is the reason transformers displaced RNNs at scale rather than any single quality issue.

The first was addressed by gating (LSTM, GRU). The second was not addressable within the recurrent framework at all — which is why the field moved to attention.

The loop, and the four shapes it can take

An RNN reads one token at a time and updates one state. Where you take the output from decides whether you have a classifier, a tagger, a generator or a translator. Four architectures from the same loop.

import numpy as np

rng = np.random.default_rng(0)
vocab = ["<pad>", "the", "film", "was", "not", "very", "good", "bad", "."]
idx = {w: i for i, w in enumerate(vocab)}
V, D, H = len(vocab), 6, 5

E = rng.normal(0, 0.5, (V, D))
Wx = rng.normal(0, 0.5, (D, H))
Wh = rng.normal(0, 0.5, (H, H))
b = np.zeros(H)
Wy = rng.normal(0, 0.5, (H, 2))          # 2 outputs, for illustration

def step(x, h):
    return np.tanh(x @ Wx + h @ Wh + b)

sent = "the film was not very good .".split()
ids = [idx[w] for w in sent]
print("the pipeline, end to end:")
print("   text   : %r" % " ".join(sent))
print("   tokens : %s" % sent)
print("   ids    : %s" % ids)
print("   vectors: %d x %d (one row per token)" % (len(ids), D))
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

h = np.zeros(H)
states = []
print("then the loop. one token in, one state out, %d times:" % len(ids))
print("%10s %8s %30s" % ("token", "id", "hidden state"))
for w, i in zip(sent, ids):
    h = step(E[i], h)
    states.append(h.copy())
    print("%10s %8d %30s" % (w, i, np.round(h, 4)))
print()
print("the state after the last token has seen the whole sentence. the state")
print("after 'not' has seen only the first four words -- and that asymmetry")
print("is what makes the four shapes below different.")
print()

print("SHAPE 1 -- MANY TO ONE. take only the final state:")
logits = states[-1] @ Wy
print("   final state -> %s -> class %d"
      % (np.round(logits, 4), int(logits.argmax())))
print("   used for: sentiment, topic classification, spam detection.")
print("   the whole sentence is compressed into %d numbers before any" % H)
print("   decision is made.")
print()

print("SHAPE 2 -- MANY TO MANY, aligned. take EVERY state:")
print("%10s %26s %10s" % ("token", "state", "tag"))
for w, s in zip(sent, states):
    print("%10s %26s %10d" % (w, np.round(s[:3], 3), int((s @ Wy).argmax())))
print("   used for: part-of-speech tagging, named entity recognition.")
print("   one output per input, and each one only sees the tokens BEFORE")
print("   it -- which is why taggers are usually bidirectional.")
print()

The following block covers the left operations.

import numpy as np

print("SHAPE 3 -- ONE TO MANY. one input, then feed each output back in:")
h = step(E[idx["the"]], np.zeros(H))
generated = ["the"]
for _ in range(4):
    probs = np.exp(h @ Wy - (h @ Wy).max())
    nxt = vocab[1 + int(np.argmax(np.abs(h)) % (V - 1))]   # a stand-in decoder
    generated.append(nxt)
    h = step(E[idx[nxt]], h)
print("   %s" % " ".join(generated))
print("   (these weights are random and the decoder is a stand-in, so the")
print("   output is gibberish. the SHAPE is the point -- output becomes")
print("   input, and the loop keeps going as long as you let it.)")
print("   used for: text generation, image captioning.")
print("   note the model's own output becomes its next input, so an early")
print("   mistake propagates -- that is exposure bias, and it is why")
print("   generation quality degrades with length.")
print()

print("SHAPE 4 -- MANY TO MANY, unaligned (encoder-decoder). read the whole")
print("input, then start writing:")

The next block continues with the left demonstrations.

print("   encoder: %s -> final state %s" % (" ".join(sent), np.round(states[-1][:3], 3)))
print("   decoder: starts from that state and produces its own sequence")
print("   used for: translation, summarisation.")
print("   the input and output lengths need not match, which none of the")
print("   first three shapes allow.")
print()

print("AND THE PROBLEM THE FIRST AND FOURTH SHARE. everything the model")
print("knows is in %d numbers, whatever the input length:" % H)
for L in (4, 10, 30):
    a = [idx[vocab[1 + (i % (V - 1))]] for i in range(L)]
    hh = np.zeros(H)
    for i in a:
        hh = step(E[i], hh)
    aa = list(a); aa[0] = idx["bad"] if aa[0] != idx["bad"] else idx["good"]
    hh2 = np.zeros(H)
    for i in aa:
        hh2 = step(E[i], hh2)
    print("   length %2d: changing the FIRST token moves the final state by %.6f"
          % (L, np.linalg.norm(hh - hh2)))
print("   by 30 tokens the first word is gone. for classification that may")
print("   be survivable; for translation it is not, and that is exactly the")
print("   bottleneck attention was invented to remove.")

Tests to try

Use the simulation to see how an RNN performs sentiment analysis (classifying a sentence as positive or negative):

1. Set the Architecture: The default is a "Many-to-One" RNN. This is perfect for sentiment analysis because we read many words but only need one final output (the total sentiment).
2. Enter a Sentence: Type a sentence like "AI is very cool" in the "Sentiment Text" box. The "Seq Length" will by itself update to the number of words.
3. Simulate the Flow: Click the "Simulate Flow" button. Watch as the hidden state (the green arrow) is passed from one time step to the next. The network reads the sentence left-to-right, updating its memory at each word.
4. Final Prediction: In this Many-to-One setup, only the final output from the very last time step is used. This output is passed to a classifier (represented by the "Output Size" of 2: one for positive, one for negative) to make the final sentiment prediction. The network has "read" the whole sentence and is now making a judgment.
5. Check Parameters: Click the "Parameters" button. Now, change the "Hidden State" size from 8 to 16. Calculate again. Notice how the number of parameters increases much. A larger hidden state allows the network to store more complex data in its memory. But it also makes it more computationally expensive.

Where that leaves you

RNNs are designed for sequential data like text. Their core feature is the hidden state, a memory that is passed through time. This allows the network to remember previous inputs and understand context. By using shared weights, they can fast process sequences of any length. This makes them a cornerstone of modern Natural Language Handling.

Backpropagation through time

Training an RNN means unrolling it. A 50-word sentence becomes a 50-layer feed-forward network that happens to share weights, and ordinary backpropagation is applied to that unrolled graph.

The gradient for a shared weight is the sum of its gradients from every time step. Since it was used at all of them.

Two practical results:

Memory grows with sequence length, because every step's activations must be stored for the backward pass. Long sequences exhaust memory quickly.

Truncated BPTT is the standard remedy: backpropagate only k steps back (usually 20–50) rather than to the beginning. It bounds memory and compute, at the cost of not learning dependencies longer than k.

Gradient clipping is in effect mandatory here. The same multiplication that vanishes gradients can explode them. And a single large gradient destroys the weights in one step.

Practical notes

import torch.nn as nn

rnn = nn.LSTM(300, 256, batch_first=True)      # prefer LSTM/GRU over nn.RNN
out, (h, c) = rnn(packed_input)

nn.RNN — the plain version — is almost never the right choice; use an LSTM or GRU. This cost little more and train far better.

Three details that matter with real text:

Pack variable-length sequences. pack_padded_sequence tells the RNN to stop at each sequence's real end. Without it, the final hidden state summarises the padding rather than the sentence.

Sort or bucket by length when batching, so batches contain similar lengths and less padding is wasted.

Consider bidirectionality for classification and tagging — running one RNN forward and another backward and concatenating gives each position both left and right context. Not available for generation or streaming.

Questions people ask

Are RNNs still used? Yes, in streaming inference with bounded memory, on-device models, small datasets and classical time-series work. Not for large-scale language modelling.

Why does the same weight matrix apply at every step? Parameter sharing along time. The same reason a convolution shares filters across space. It lets one model handle any length.

How long a sequence can an LSTM handle? Hundreds of steps in practice; thousands is unreliable. Transformers handle far longer, at quadratic cost.

What is the hidden state size? A hyperparameter, usually 128–512. Larger means more memory capacity and more parameters.

Can I stack RNN layers? Yes, and two is usually the practical limit. Deeper stacks are hard to train.

Why did transformers replace them? Parallelism across the sequence and constant path length between positions. Both are structural gains that gating cannot provide.

Recap in one screen

• An RNN carries a fixed-size hidden state, updated at each step from the input and the previous state.
• The same weights are reused at every step, so one model handles any sequence length.
• Training unrolls the sequence and sums each weight's gradient across all steps.
• Vanishing gradients limit range (fixed by gating). The sequential dependency prevents parallelism (not fixable).
• Pack variable-length batches, clip gradients, and prefer LSTM or GRU over the plain cell.