A recurrent cell is a neuron with memory: it reads one token at a time and carries a hidden state forward. Step through a sentence and watch the memory change.
Overview
The problem with feeding a sentence to a normal network
A feedforward network has a fixed number of inputs. A sentence does not have a fixed number of words. You can pad everything to a maximum length, but then the model has no notion that word 3 comes before word 4 — each position gets its own independent weights, so “dog bites man” and “man bites dog” are unrelated inputs as far as it is concerned.
A recurrent cell solves both problems with one idea: process one token at a time, and carry a running summary forward.
Input Sequence
The Update Rule
hₜ = tanh(W·xₜ + U·hₜ₋₁ + b)
xₜ — embedding of the current token
hₜ₋₁ — memory carried from the previous step
W, U, b — the same weights, reused at every step
The Cell
t = 0
Hidden state hₜ — the network's memory (4 dims)
Memory Timeline
t
Token
Hidden state after step
Memory starts at h₀ = [0, 0, 0, 0]. Feed the first token.
What is a Recurrent Cell?: A Practical Guide
A recurrent cell is a small network that runs once per token and passes a summary of everything it has seen to its future self. That loop is what lets a fixed-size model read a sequence of any length.
The recurrence, written out
At each timestep t the cell takes the current input xt and the previous hidden state ht−1, and produces a new hidden state:
ht = tanh(Wxxt + Whht−1 + b)
Three things are worth noticing. The hidden state is the memory — a fixed-size vector holding everything the cell has decided is worth keeping from the sequence so far. The weights Wx, Wh and b are the same at every timestep; the cell is one small network applied repeatedly, not a chain of different networks. And because the weights do not depend on position, the same cell handles a sequence of 5 tokens or 500.
That weight sharing is what makes the parameter count independent of sequence length — and it is exactly the same trick a convolution uses across space.
The unit that carries state
A recurrent cell is the piece of a recurrent network that runs once per time step. It takes two things — the current input and its own state from the previous step — and produces a new state.
hᵗ = cell(xᵗ, hᵗ₋₁)
For the simplest cell:
hᵗ = tanh(Wₕₕ hᵗ₋₁ + Wₓₕ xᵗ + b)
The state hₜ is a fixed-size vector that summarises everything seen so far. Its size is a hyperparameter — 128, 256, 512 — and it does not grow with the sequence, which is both the cell's defining strength and its defining limit.
The same weights are used at every step. That is parameter sharing along time, and it is what lets one cell handle a sequence of any length with a fixed parameter count. A network with distinct weights per position would need a separate set for every possible length.
Three cells, in order of sophistication
Cell
State
Gates
Parameters per layer
Simple RNN
h
None
1 matrix
GRU
h
2 (update, reset)
3 matrices
LSTM
h and c
3 (forget, input, output)
4 matrices
The simple cell overwrites its state at every step. Whatever it knew is repeatedly transformed, and after fifty steps almost nothing survives. Gradients decay along the same path, so it cannot learn long-range dependencies. It is a teaching device rather than a practical choice.
The GRU adds an update gate that interpolates between keeping the old state and replacing it, plus a reset gate controlling how much history the candidate sees. One state, two gates.
The LSTM adds a separate cell state updated additively, with three gates deciding what to drop, what to store and what to expose. Two states, three gates.
The pattern across the three is the same insight applied more thoroughly: give the state an additive path so information and gradients can survive many steps.
Why gating changes everything
For the simple cell, the gradient across one step involves the derivative of tanh times the recurrent weight matrix — a quantity nothing keeps near 1, and usually below it. Over 50 steps the product collapses.
For an LSTM, ∂cₜ/∂cₜ₋₁ is exactly the forget gate. When the network has learned to preserve something, that value is near 1 and the gradient passes back essentially undamped.
That single derivative is the difference between failing at 20 steps and succeeding at 200. And it is structurally the same trick as a residual connection in a deep feed-forward network: an additive path with a derivative of 1.
The practical guidance follows directly: use an LSTM or GRU, never nn.RNN. The extra cost is small and the difference in what can be learned is large.
One interface, three implementations
A recurrent cell is any function that takes a state and an input and returns a new state. Writing the RNN, LSTM and GRU cells against that one signature shows what they share and exactly where they differ.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
X, H = 3, 4
def sigmoid(z):
return 1 / (1 + np.exp(-np.clip(z, -30, 30)))
print("the interface, and it is the whole abstraction:")
print(" new_state = cell(input, old_state)")
print()
print("everything else -- how many gates, whether there is a separate")
print("memory -- is an implementation detail behind that signature. which")
print("is why frameworks let you swap RNN for LSTM by changing one word.")
print()
# --- the three cells, same signature
Wx = rng.normal(0, 0.5, (X, H)); Wh = rng.normal(0, 0.5, (H, H))
def rnn_cell(x, state):
h = state
return np.tanh(x @ Wx + h @ Wh)
Wg = {k: rng.normal(0, 0.5, (X + H, H)) for k in "fiog"}
bg = {k: (np.ones(H) if k == "f" else np.zeros(H)) for k in "fiog"}
def lstm_cell(x, state):
h, c = state
z = np.concatenate([x, h])
f = sigmoid(z @ Wg["f"] + bg["f"])
i = sigmoid(z @ Wg["i"] + bg["i"])
g = np.tanh(z @ Wg["g"] + bg["g"])
o = sigmoid(z @ Wg["o"] + bg["o"])
c_new = f * c + i * g
return o * np.tanh(c_new), c_new
Wz = rng.normal(0, 0.5, (X + H, H))
Wr = rng.normal(0, 0.5, (X + H, H))
Wc = rng.normal(0, 0.5, (X + H, H))
def gru_cell(x, state):
h = state
zz = np.concatenate([x, h])
z = sigmoid(zz @ Wz)
r = sigmoid(zz @ Wr)
cand = np.tanh(np.concatenate([x, r * h]) @ Wc)
return (1 - z) * h + z * cand
print("%14s %12s %10s %14s"
% ("cell", "state", "gates", "parameters"))
print("%14s %12s %10d %14d" % ("RNN", "h", 0, Wx.size + Wh.size))
print("%14s %12s %10d %14d" % ("LSTM", "h and c", 4, 4 * (X + H) * H))
print("%14s %12s %10d %14d" % ("GRU", "h", 3, 3 * (X + H) * H))
print()
print("run all three over the same 5 inputs. only the LSTM's state is a")
print("pair; the loop that drives them is otherwise identical:")
seq = rng.normal(0, 1.0, (5, X))
print("%8s %30s %30s %30s" % ("step", "RNN h[0:2]", "LSTM h[0:2]", "GRU h[0:2]"))
h_r = np.zeros(H)
s_l = (np.zeros(H), np.zeros(H))
h_g = np.zeros(H)
for t, x in enumerate(seq):
h_r = rnn_cell(x, h_r)
s_l = lstm_cell(x, s_l)
h_g = gru_cell(x, h_g)
print("%8d %30s %30s %30s"
% (t, np.round(h_r[:2], 4), np.round(s_l[0][:2], 4), np.round(h_g[:2], 4)))
print()
print("THE LOOP IS THE SAME CODE for all three:")
print(" state = initial_state")
print(" for x in sequence:")
print(" state = cell(x, state)")
print("that is what makes a cell a useful abstraction, and it is why")
print("'RNN' loosely means both the cell and the loop around it.")
print()
print("WHAT THE STATE HAS TO DO. it is a fixed-size summary of everything")
print("seen so far, and everything the model will ever know about the past")
print("has to fit in it:")
for h_size in (4, 128, 1024):
print(" hidden size %5d -> %5d numbers carry the entire history"
% (h_size, h_size))
print(" that bound does not grow with the sequence. a 10-token sequence")
print(" and a 10,000-token one are summarised into the same %d numbers."
% H)
print()
print(" here is that limit biting. feed a long sequence and ask how much")
print(" of step 0 is still recoverable from the state:")
for L in (2, 5, 10, 25, 50):
a = rng.normal(0, 1.0, (L, X))
b = a.copy(); b[0] = rng.normal(0, 1.0, X)
ha, hb = np.zeros(H), np.zeros(H)
for xa, xb in zip(a, b):
ha, hb = rnn_cell(xa, ha), rnn_cell(xb, hb)
print(" length %3d: changing step 0 moves the final state by %.6f"
% (L, np.linalg.norm(ha - hb)))
print(" the LSTM and GRU exist to make that number fall more slowly. they")
print(" do not remove the bound -- the state is still fixed size.")
print()
print("STACKING. a cell's output is a valid input to another cell, so you")
print("can layer them:")
Wx2 = rng.normal(0, 0.5, (H, H))
def rnn_cell2(x, h):
return np.tanh(x @ Wx2 + h @ Wh)
h1, h2 = np.zeros(H), np.zeros(H)
print("%8s %26s %26s" % ("step", "layer 1 h[0:2]", "layer 2 h[0:2]"))
for t, x in enumerate(seq):
h1 = rnn_cell(x, h1)
h2 = rnn_cell2(h1, h2)
print("%8d %26s %26s" % (t, np.round(h1[:2], 4), np.round(h2[:2], 4)))
print(" layer 2's input at step t is layer 1's output at step t. depth in")
print(" this direction is separate from depth through time, and both make")
print(" the gradient path longer.")
Output
Things to try
Feed one token at a time. Press Feed Next Token and watch the hidden state change. Each press mixes new input into the existing memory rather than replacing it.
Watch old information fade. Keep pressing and follow the contribution of the first token. It does not vanish at once; it is progressively diluted as later tokens are folded in. That gradual decay is exactly why plain recurrent cells struggle with long-range dependencies.
Reset and compare. Press Reset Memory and feed the same tokens in a different order. The final state differs — unlike a bag-of-words model, order genuinely changes the representation.
Run it continuously. Press Auto Run and note that no new parameters appear as the sequence lengthens. One cell, applied repeatedly.
Why the simple cell is not enough
The hidden state is overwritten at every step. Information from token 1 survives to token 50 only by passing through 49 successive multiplications by Wh and 49 tanh nonlinearities.
That is the vanishing gradient problem in its original setting. The gradient reaching timestep 1 is a product of 49 terms, and since tanh’s derivative is at most 1 and typically well below it, the product decays geometrically. In practice a simple recurrent cell reliably remembers about 10 timesteps and loses anything much further back.
The fix is to give the cell an explicit, additive memory channel with learned gates controlling what enters and leaves it — which is precisely what LSTM and GRU do.
Common mistakes
Expecting long-range memory. A vanilla recurrent cell does not have it. If dependencies span more than a handful of tokens, use an LSTM, a GRU, or attention.
Exploding gradients. If Wh has eigenvalues above 1 the repeated multiplication amplifies instead of decaying, and the loss becomes NaN. Gradient clipping is standard practice for recurrent models for exactly this reason.
Confusing the hidden state with the output. They coincide in the simplest cell, but in an LSTM the cell state and hidden state are different objects, and mixing them up is a common implementation bug.
Forgetting to reset state between sequences. Carrying the hidden state from one unrelated example into the next leaks information and quietly corrupts training.
In one line
A recurrent cell applies one small network at every timestep, combining the current token with a hidden state that summarises everything before it, using the same weights throughout. That gives a fixed-size model the ability to read arbitrary-length sequences and to be sensitive to order. Its weakness is that memory is overwritten multiplicatively at each step, so information decays after roughly ten timesteps — the limitation that gated cells were invented to remove.
The cell versus the layer
Frameworks distinguish two levels, and knowing which you want saves confusion.
The cell processes one time step. nn.LSTMCell takes (x, (h, c)) and returns the new states. You write the loop.
The layer processes a whole sequence. nn.LSTM takes a (batch, seq, features) tensor and runs the loop internally, in optimised C or cuDNN.
# the layer - what you almost always want
lstm = nn.LSTM(300, 256, batch_first=True)
out, (h, c) = lstm(x) # x: (batch, seq, 300)
# the cell - when you need per-step control
cell = nn.LSTMCell(300, 256)
h, c = torch.zeros(b, 256), torch.zeros(b, 256)
for t in range(seq_len):
h, c = cell(x[:, t], (h, c))
The layer is dramatically faster — often ten times or more — because it fuses the four gate matrices into one multiplication and uses hardware-optimised kernels. Write the loop only when you need something the layer cannot express: inspecting gates, applying custom logic between steps, scheduled sampling during training.
Two states, two purposes
For an LSTM specifically, the distinction between the states is worth restating because it is the design's whole point.
The cell state is private long-term memory. It is unbounded, updated additively, and passed only to the next time step.
The hidden state is the working output. It is bounded by tanh, filtered by the output gate, and passed to three places: the next layer, the prediction head, and the next step's gate computations.
That separation lets a cell hold information quietly without reporting it. A GRU, having only a hidden state, cannot make the distinction — whatever it remembers is what it outputs.
Where recurrent cells still fit
Transformers displaced them for large-scale language work. The cases where a recurrent cell remains the better engineering choice are specific and real:
Streaming with bounded memory. One fixed-size state regardless of how much has been processed, where a transformer's KV cache grows with every token.
On-device inference. Small, fast, no attention kernels required.
Small datasets. The inductive bias helps where a transformer trained from scratch overfits.
Classical time series. Numeric sequences of moderate length, where windowing plus a GRU is a strong baseline.
Inside larger systems, as state trackers or encoders.
State-space models such as Mamba are worth noting: they revisit recurrence with linear scaling in sequence length and are competitive with transformers on long sequences, which suggests the idea is not finished.
Questions people ask
What is the difference between a cell and a layer? The cell handles one step; the layer runs the loop over a whole sequence and is much faster.
Should I ever use nn.RNN? Only to demonstrate the problem it has. Use LSTM or GRU.
How big should the hidden state be? 128–512 typically. Larger means more memory capacity and more parameters.
Can I stack cells? Yes — each layer's outputs are the next layer's inputs. Two layers is the practical limit.
Does the cell see the whole sequence? Not at once. It sees one element at a time, carrying a summary of the rest.
Why can it not be parallelised? Step t needs hₜ₋₁, so the steps are strictly ordered. This is the structural reason transformers won at scale.
Recap in one screen
A recurrent cell maps the current input and the previous state to a new state, using the same weights every step.
The state is a fixed-size summary that does not grow with the sequence.
A simple cell overwrites its state and cannot learn long range; gating adds an additive path that preserves both information and gradients.
Use the layer (nn.LSTM) rather than the cell unless you need per-step control.
Still the right tool for streaming, on-device and small-data sequence work.
Recall check
0 of 4
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Streaming with bounded memory” here?
One fixed-size state regardless of how much has been processed, where a transformer's KV cache grows with every token.
What is meant by “On-device inference” here?
Small, fast, no attention kernels required.
What is meant by “Small datasets” here?
The inductive bias helps where a transformer trained from scratch overfits.
What is meant by “Classical time series” here?
Numeric sequences of moderate length, where windowing plus a GRU is a strong baseline.
Cheat sheet
What is a Recurrent Cell?
A recurrent cell is a neuron with memory: it reads one token at a time and carries a hidden state forward. Step through a sentence and watch the memory change.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.