What is LSTM?

Two states, not one

A simple recurrent cell has one hidden state, and it is fully rewritten at every timestep. An LSTM has two:

• Cell state C_t — the long-term memory. It is modified only by addition and elementwise multiplication by a gate, never by a full matrix multiply.
• Hidden state h_t — the working output, a filtered view of the cell state, passed to the next layer and the next timestep.

The cell state is the important one. Because it is updated additively, information placed there at timestep 1 can reach timestep 500 in effect untouched. There is no repeated matrix multiplication to shrink it. Three gates control what flows through this channel. The forget gate decides how much of the old cell state to keep (a sigmoid output times elementwise). The input gate decides how much of a candidate update to write. The output gate decides how much of the cell state to expose as the hidden state. Each gate is a learned sigmoid layer, so the network learns *when* to remember and *when* to forget, rather than having memory decay forced by architecture. The GRU simplifies this to two gates and merges the cell and hidden states, trading some capacity for fewer parameters and faster training. This page shows each gate's activation as the LSTM reads a sentence, lets you trace how a specific piece of information survives across timesteps, and compares the LSTM and GRU architectures side by side.

The three gates

Each gate is a small sigmoid layer producing values in [0, 1]. This then multiply a vector elementwise — 0 blocks fully, 1 passes fully, and everything between is a partial pass.

f_t = σ(W_f·[h_t−1, x_t] + b_f)   forget

i_t = σ(W_i·[h_t−1, x_t] + b_i)   input

o_t = σ(W_o·[h_t−1, x_t] + b_o)   output

The cell state update is then two operations — erase, then write:

C_t = f_t ⊙ C_t−1 + i_t ⊙ C̃_t

h_t = o_t ⊙ tanh(C_t)

In a language model the forget gate might drop the previous subject’s gender when a new subject appears, the input gate writes the new one. And the output gate exposes it only when a pronoun actually needs to agree.

Why the gradient survives

The whole design rests on the additive update. Differentiating C_t about C_t−1 gives f_t. The forget gate itself, not a weight matrix.

So the gradient flowing back through the cell state is times by the forget gate at each step. When the network learns to keep something, the forget gate sits near 1, and multiplying by roughly 1 many times preserves the gradient. Compare that with a simple RNN, where the same path is times by W_h and a tanh derivative every step and decays geometrically.

This is the same principle as a residual connection: give the gradient an uninterrupted additive route, and depth stops destroying it.

A cell with a memory it can protect

A plain recurrent network overwrites its hidden state at every step. Whatever it knew about word one is repeatedly transformed, and after fifty steps almost nothing survives. Gradients decay along the same path, so it cannot learn long-range dependencies either.

An LSTM adds a second, apart protected state: the cell state. This travels along the sequence with only additive updates and multiplicative gates deciding what enters and leaves.

Two states, two jobs:

• Cell state c — long-term memory. Modified by addition, not by rewriting.
• Hidden state h — the working output at each step, derived from the cell state.

Three gates control the flow, each a small neural network with a sigmoid output between 0 and 1 — 0 meaning "block fully", 1 meaning "let everything through":

The equations, read as sentences

fᵗ = σ(Wᶠ·[hᵗ₋₁, xᵗ] + bᶠ)    forget gate

iᵗ = σ(Wᵢ·[hᵗ₋₁, xᵗ] + bᵢ)    input gate

gᵗ = tanh(Wᵕ·[hᵗ₋₁, xᵗ] + bᵕ)   candidate memory

cᵗ = fᵗ ⊙ cᵗ₋₁ + iᵗ ⊙ gᵗ    update the cell

oᵗ = σ(Wₒ·[hᵗ₋₁, xᵗ] + bₒ)    output gate

hᵗ = oᵗ ⊙ tanh(cᵗ)    produce the output

The line that matters most is the cell update. It is old memory scaled by the forget gate, plus new memory scaled by the input gate. Both operations are elementwise, so each dimension of the cell state is managed independently. One dimension can hold grammatical number while another tracks sentiment.

Note the two activations doing different jobs. Sigmoid produces gates, because a gate needs to be a fraction between 0 and 1. Tanh produces content, because content should be able to be negative.

Why this fixes vanishing gradients

In a plain RNN, the gradient flowing back through k steps is a product of k Jacobian factors. If those average below 1, the product decays exponentially.

The LSTM's cell state has a path where the derivative of cₜ about cₜ₋₁ is simply the forget gate fₜ. If the forget gate is near 1 — the network has learned to keep this information — the gradient passes back in effect unchanged.

That is the same structural idea as a residual connection: an additive path with a derivative near 1, protecting the gradient from the multiplicative decay. It is why LSTMs handled sequences of hundreds of steps where plain RNNs managed ten.

It reduces the problem rather than removing it. Very long dependencies remain hard, and exploding gradients still occur. This is why gradient clipping is standard when training recurrent models.

The whole cell, one timestep at a time

An LSTM cell is four small networks and one running memory. All four are computed here on a real sequence, with the cell state printed at every step so you can see what it carries.

import numpy as np

rng = np.random.default_rng(0)

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -30, 30)))

H, X = 4, 3          # hidden size, input size
Wf = rng.normal(0, 0.5, (X + H, H)); bf = np.ones(H)      # forget bias at 1
Wi = rng.normal(0, 0.5, (X + H, H)); bi = np.zeros(H)
Wc = rng.normal(0, 0.5, (X + H, H)); bc = np.zeros(H)
Wo = rng.normal(0, 0.5, (X + H, H)); bo = np.zeros(H)

def step(x, h, c):
    z = np.concatenate([x, h])
    f = sigmoid(z @ Wf + bf)          # forget:  what to drop from memory
    i = sigmoid(z @ Wi + bi)          # input:   how much of the new to admit
    g = np.tanh(z @ Wc + bc)          # candidate: what the new thing IS
    o = sigmoid(z @ Wo + bo)          # output:  what to reveal this step
    c_new = f * c + i * g
    h_new = o * np.tanh(c_new)
    return h_new, c_new, (f, i, g, o)
print("done")

Building on the same setup, the next block shows the remaining cases.

import numpy as np

print("the cell has TWO states, and the distinction is the whole design:")
print("   c -- the cell state. long-term memory. edited by addition.")
print("   h -- the hidden state. what the rest of the network sees.")
print()
print("and four gates, each a small layer over [x, h] concatenated:")
print("   forget    sigmoid -> 0..1, multiplies the old memory")
print("   input     sigmoid -> 0..1, multiplies the candidate")
print("   candidate tanh    -> -1..1, the new content itself")
print("   output    sigmoid -> 0..1, how much memory to expose as h")
print()
print("parameters: 4 gates x (%d + %d) x %d weights + %d biases = %d"
      % (X, H, H, 4 * H, 4 * ((X + H) * H + H)))
print("   four times a plain RNN of the same size, which is what the")
print("   gating costs.")
print()

seq = rng.normal(0, 1.0, (6, X))
h, c = np.zeros(H), np.zeros(H)
print("run %d timesteps. watch c accumulate while h moves around:" % len(seq))
print("%6s %30s %30s" % ("step", "cell state c", "hidden state h"))
print("%6s %30s %30s" % ("init", np.round(c, 3), np.round(h, 3)))
for t, x in enumerate(seq):
    h, c, gates = step(x, h, c)
    print("%6d %30s %30s" % (t, np.round(c, 3), np.round(h, 3)))
print()

The following block covers the remaining operations.

import numpy as np

h, c = np.zeros(H), np.zeros(H)
print("the gate values at each step, for hidden unit 0:")
print("%6s %10s %10s %12s %10s %14s %12s"
      % ("step", "forget", "input", "candidate", "output", "c[0] after", "h[0]"))
for t, x in enumerate(seq):
    h, c, (f, i, g, o) = step(x, h, c)
    print("%6d %10.4f %10.4f %12.4f %10.4f %14.4f %12.4f"
          % (t, f[0], i[0], g[0], o[0], c[0], h[0]))
print()
print("read the update: c = forget * c_old + input * candidate.")
print("   forget near 1 and input near 0 -> memory passes through untouched")
print("   forget near 0 and input near 1 -> memory is replaced")
print("   both near 1                    -> memory accumulates")
print()

The next block continues with the remaining demonstrations.

import numpy as np

print("WHY THE ADDITION MATTERS. a plain RNN multiplies its state by a")
print("matrix at every step, so the gradient is a product of %d matrices."
      % len(seq))
print("an LSTM's cell state is updated by ADDITION, so the gradient through")
print("it is a product of forget gates instead:")
for fval, label in ((0.99, "forget gate ~0.99"), (0.9, "0.90"), (0.5, "0.50")):
    print("   %-20s over 100 steps: %.4e" % (label, fval ** 100))
print("   a forget gate the model has learned to keep near 1 gives an almost")
print("   unobstructed path for the gradient. that is the entire reason an")
print("   LSTM can learn dependencies a plain RNN cannot.")
print()
print("and it is why the forget bias is initialised to 1 rather than 0:")
print("   bias 0 -> sigmoid(0) = %.4f, so memory halves at every step"
      % sigmoid(0.0))
print("   bias 1 -> sigmoid(1) = %.4f, so it leaks far more slowly"
      % sigmoid(1.0))
print("   over 20 steps: %.2e survives against %.2e -- a factor of %.0f."
      % (sigmoid(1.0) ** 20, sigmoid(0.0) ** 20,
         (sigmoid(1.0) / sigmoid(0.0)) ** 20))
print("   neither is 'remembering' on its own. the bias just decides where")
print("   the cell STARTS, before training moves it -- and starting shut is")
print("   starting with no gradient path to learn from.")
print()

Building on the same setup, the next block shows the remaining cases.

import numpy as np

print("a concrete memory test -- can the cell hold one bit across a gap?")
def carry_test(gap, forget_bias):
    bf_local = np.full(H, forget_bias)
    h2, c2 = np.zeros(H), np.zeros(H)
    signal = np.zeros(X); signal[0] = 3.0
    for t in range(gap + 1):
        x = signal if t == 0 else np.zeros(X)
        z = np.concatenate([x, h2])
        f = sigmoid(z @ Wf + bf_local)
        i = sigmoid(z @ Wi + bi)
        g = np.tanh(z @ Wc + bc)
        o = sigmoid(z @ Wo + bo)
        c2 = f * c2 + i * g
        h2 = o * np.tanh(c2)
    return np.abs(c2).max()

print("%12s %18s %18s" % ("gap", "forget bias 0", "forget bias 3"))
for gap in (1, 5, 20, 60):
    print("%12d %18.6f %18.6f" % (gap, carry_test(gap, 0.0), carry_test(gap, 3.0)))
print("   biased shut, the trace is gone within a handful of steps. biased")
print("   open, the state persists -- and in fact drifts upward, because the")
print("   input gate is never fully closed either and keeps adding a little")
print("   at every step. an untrained cell leaks in both directions.")
print()
print("   in a trained model both biases are learned per unit, which is how")
print("   some units end up as long-term registers and others as fast")
print("   scratch space. nobody assigns those roles; the gradient does.")

Experiments to try

1. Count the cost of gating. Press Parameters and compare with a plain RNN of the same size. An LSTM has four weight matrices instead of one — three gates plus the candidate — so it needs roughly four times the parameters.
2. Grow the state. Raise State Size and watch the count climb. Each of the four blocks is (d + h) × h. So the total is 4[(d + h)h + h].
3. Follow one timestep. Press Simulate Cell Operation and trace the cell state along the top. It passes straight through, touched only by a multiply and an add — that unbroken line is the long-term memory path.
4. Watch the gates open and close. During the simulation note that gate values are between 0 and 1 rather than binary. Gating is continuous, which is what makes it differentiable and so learnable.

Traps worth knowing

• Setting up the forget-gate bias at zero. That puts the gate at 0.5, halving the cell state every step and destroying memory before training can fix it. Initialise it to 1 or 2 so the cell starts out remembering.
• Reaching for an LSTM by default. A GRU merges the forget and input gates into one, uses about 25% fewer parameters, trains faster, and performs comparably on most tasks. Try it first.
• Confusing cell state with hidden state. PyTorch returns both as a tuple; passing the wrong one to the next layer is a common and quiet bug.
• Using an LSTM where attention belongs. For long documents a transformer both remembers further and parallelises, which an LSTM cannot.

In one line

An LSTM separates long-term memory (the cell state) from working output (the hidden state) and controls the flow between them with three learned sigmoid gates. Because the cell state is updated by addition and gated multiplication rather than a matrix multiply, the gradient travels back through it times only by the forget gate. So when the model chooses to remember, it genuinely can, for hundreds of steps. The cost is four times the parameters of a simple cell. And a GRU usually gets most of the benefit for less.

Using one, and the shapes involved

import torch.nn as nn

lstm = nn.LSTM(input_size=300,       # embedding dimension
               hidden_size=256,
               num_layers=2,
               batch_first=True,
               bidirectional=False,
               dropout=0.2)          # applies between layers only

out, (h, c) = lstm(x)                # x: (batch, seq, 300)
# out: (batch, seq, 256)   -- the hidden state at every step
# h, c: (layers, batch, 256) -- the final states

Which output you use depends on the task. For classification, take the final hidden state (or a pooled version of all steps). For tagging or sequence-to-sequence, use the full out tensor.

Parameter count is worth knowing, because it explains why LSTMs are heavier than they look: four gates each need a weight matrix over the concatenated input and hidden state, giving 4 × ((input + hidden) × hidden + hidden). For 300-dimensional input and 256 hidden units that is about 570,000 parameters per layer.

For variable-length batches, use pack_padded_sequence so the LSTM does not process padding. Otherwise the final hidden state reflects the padding rather than the last real token.

LSTM, GRU or transformer?

GRU merges the forget and input gates into a single update gate and drops the separate cell state. Fewer parameters, slightly faster, and empirically comparable on most tasks. The choice between them is usually not worth agonising over.

Transformers won for large-scale language work because they parallelise across the sequence and have constant path length between positions. But LSTMs remain a sensible choice for streaming inference with bounded memory, for small datasets, for on-device models, and for classical time-series work.

Questions people ask

Are LSTMs obsolete? Superseded for large-scale language modelling; still practical for streaming, small data, embedded use and time series.

What is the difference between the cell state and the hidden state? The cell state is protected long-term memory updated additively; the hidden state is the filtered output exposed at each step.

How many layers? One or two for most tasks. Deep stacks of LSTMs are hard to train and rarely pay off.

Why tanh and sigmoid rather than ReLU? Sigmoid bounds gates to 0–1. This is what a gate requires. Tanh bounds content to −1–1, which keeps the repeatedly-applied recurrence stable.

Do I need gradient clipping? Yes — recurrent models are the classic case for it.

Should I use a bidirectional LSTM? For classification and tagging where the whole input is available, yes. It usually helps. Never for generation or streaming, where the future is not available.

Recap in one screen

• Two states: a protected cell state for long-term memory, a hidden state for the working output.
• Three sigmoid gates decide what to forget, what to store and what to expose.
• The cell updates additively, so ∂cₜ/∂cₜ₋₁ is the forget gate. Near 1 means the gradient survives.
• That additive path is the same idea as a residual connection. And it is what fixed vanishing gradients.
• Superseded by transformers at scale; still the right tool for streaming, small data and on-device work.