What is LSTM?

The three gates

Each gate is a small sigmoid layer producing values in [0, 1], which then multiply a vector elementwise — 0 blocks completely, 1 passes completely, and everything between is a partial pass.

f_t = σ(W_f·[h_t−1, x_t] + b_f)   forget

i_t = σ(W_i·[h_t−1, x_t] + b_i)   input

o_t = σ(W_o·[h_t−1, x_t] + b_o)   output

The cell state update is then two operations — erase, then write:

C_t = f_t ⊙ C_t−1 + i_t ⊙ C̃_t

h_t = o_t ⊙ tanh(C_t)

In a language model the forget gate might drop the previous subject’s gender when a new subject appears, the input gate writes the new one, and the output gate exposes it only when a pronoun actually needs to agree.

Why the gradient survives

The whole design rests on the additive update. Differentiating C_t with respect to C_t−1 gives f_t — the forget gate itself, not a weight matrix.

So the gradient flowing back through the cell state is multiplied by the forget gate at each step. When the network learns to keep something, the forget gate sits near 1, and multiplying by roughly 1 many times preserves the gradient. Compare that with a simple RNN, where the same path is multiplied by W_h and a tanh derivative every step and decays geometrically.

This is the same principle as a residual connection: give the gradient an uninterrupted additive route, and depth stops destroying it.

A cell with a memory it can protect

A plain recurrent network overwrites its hidden state at every step. Whatever it knew about word one is repeatedly transformed, and after fifty steps almost nothing survives — gradients decay along the same path, so it cannot learn long-range dependencies either.

An LSTM adds a second, separately protected state: the cell state, which travels along the sequence with only additive updates and multiplicative gates deciding what enters and leaves.

Two states, two jobs:

• Cell state c — long-term memory. Modified by addition, not by rewriting.
• Hidden state h — the working output at each step, derived from the cell state.

Three gates control the flow, each a small neural network with a sigmoid output between 0 and 1 — 0 meaning "block completely", 1 meaning "let everything through":

The equations, read as sentences

fᵗ = σ(Wᶠ·[hᵗ₋₁, xᵗ] + bᶠ)    forget gate

iᵗ = σ(Wᵢ·[hᵗ₋₁, xᵗ] + bᵢ)    input gate

gᵗ = tanh(Wᵕ·[hᵗ₋₁, xᵗ] + bᵕ)   candidate memory

cᵗ = fᵗ ⊙ cᵗ₋₁ + iᵗ ⊙ gᵗ    update the cell

oᵗ = σ(Wₒ·[hᵗ₋₁, xᵗ] + bₒ)    output gate

hᵗ = oᵗ ⊙ tanh(cᵗ)    produce the output

The line that matters most is the cell update. It is old memory scaled by the forget gate, plus new memory scaled by the input gate. Both operations are elementwise, so each dimension of the cell state is managed independently — one dimension can hold grammatical number while another tracks sentiment.

Note the two activations doing different jobs. Sigmoid produces gates, because a gate needs to be a fraction between 0 and 1. Tanh produces content, because content should be able to be negative.

Why this fixes vanishing gradients

In a plain RNN, the gradient flowing back through k steps is a product of k Jacobian factors. If those average below 1, the product decays exponentially.

The LSTM's cell state has a path where the derivative of cₜ with respect to cₜ₋₁ is simply the forget gate fₜ. If the forget gate is near 1 — the network has learned to keep this information — the gradient passes back essentially unchanged.

That is the same structural idea as a residual connection: an additive path with a derivative near 1, protecting the gradient from the multiplicative decay. It is why LSTMs handled sequences of hundreds of steps where plain RNNs managed ten.

It reduces the problem rather than eliminating it. Very long dependencies remain hard, and exploding gradients still occur, which is why gradient clipping is standard when training recurrent models.

The whole cell, one timestep at a time

An LSTM cell is four small networks and one running memory. All four are computed here on a real sequence, with the cell state printed at every step so you can see what it carries.

import numpy as np

rng = np.random.default_rng(0)

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -30, 30)))

H, X = 4, 3          # hidden size, input size
Wf = rng.normal(0, 0.5, (X + H, H)); bf = np.ones(H)      # forget bias at 1
Wi = rng.normal(0, 0.5, (X + H, H)); bi = np.zeros(H)
Wc = rng.normal(0, 0.5, (X + H, H)); bc = np.zeros(H)
Wo = rng.normal(0, 0.5, (X + H, H)); bo = np.zeros(H)

def step(x, h, c):
    z = np.concatenate([x, h])
    f = sigmoid(z @ Wf + bf)          # forget:  what to drop from memory
    i = sigmoid(z @ Wi + bi)          # input:   how much of the new to admit
    g = np.tanh(z @ Wc + bc)          # candidate: what the new thing IS
    o = sigmoid(z @ Wo + bo)          # output:  what to reveal this step
    c_new = f * c + i * g
    h_new = o * np.tanh(c_new)
    return h_new, c_new, (f, i, g, o)

print("the cell has TWO states, and the distinction is the whole design:")
print("   c -- the cell state. long-term memory. edited by addition.")
print("   h -- the hidden state. what the rest of the network sees.")
print()
print("and four gates, each a small layer over [x, h] concatenated:")
print("   forget    sigmoid -> 0..1, multiplies the old memory")
print("   input     sigmoid -> 0..1, multiplies the candidate")
print("   candidate tanh    -> -1..1, the new content itself")
print("   output    sigmoid -> 0..1, how much memory to expose as h")
print()
print("parameters: 4 gates x (%d + %d) x %d weights + %d biases = %d"
      % (X, H, H, 4 * H, 4 * ((X + H) * H + H)))
print("   four times a plain RNN of the same size, which is what the")
print("   gating costs.")
print()

seq = rng.normal(0, 1.0, (6, X))
h, c = np.zeros(H), np.zeros(H)
print("run %d timesteps. watch c accumulate while h moves around:" % len(seq))
print("%6s %30s %30s" % ("step", "cell state c", "hidden state h"))
print("%6s %30s %30s" % ("init", np.round(c, 3), np.round(h, 3)))
for t, x in enumerate(seq):
    h, c, gates = step(x, h, c)
    print("%6d %30s %30s" % (t, np.round(c, 3), np.round(h, 3)))
print()

h, c = np.zeros(H), np.zeros(H)
print("the gate values at each step, for hidden unit 0:")
print("%6s %10s %10s %12s %10s %14s %12s"
      % ("step", "forget", "input", "candidate", "output", "c[0] after", "h[0]"))
for t, x in enumerate(seq):
    h, c, (f, i, g, o) = step(x, h, c)
    print("%6d %10.4f %10.4f %12.4f %10.4f %14.4f %12.4f"
          % (t, f[0], i[0], g[0], o[0], c[0], h[0]))
print()
print("read the update: c = forget * c_old + input * candidate.")
print("   forget near 1 and input near 0 -> memory passes through untouched")
print("   forget near 0 and input near 1 -> memory is replaced")
print("   both near 1                    -> memory accumulates")
print()

print("WHY THE ADDITION MATTERS. a plain RNN multiplies its state by a")
print("matrix at every step, so the gradient is a product of %d matrices."
      % len(seq))
print("an LSTM's cell state is updated by ADDITION, so the gradient through")
print("it is a product of forget gates instead:")
for fval, label in ((0.99, "forget gate ~0.99"), (0.9, "0.90"), (0.5, "0.50")):
    print("   %-20s over 100 steps: %.4e" % (label, fval ** 100))
print("   a forget gate the model has learned to keep near 1 gives an almost")
print("   unobstructed path for the gradient. that is the entire reason an")
print("   LSTM can learn dependencies a plain RNN cannot.")
print()
print("and it is why the forget bias is initialised to 1 rather than 0:")
print("   bias 0 -> sigmoid(0) = %.4f, so memory halves at every step"
      % sigmoid(0.0))
print("   bias 1 -> sigmoid(1) = %.4f, so it leaks far more slowly"
      % sigmoid(1.0))
print("   over 20 steps: %.2e survives against %.2e -- a factor of %.0f."
      % (sigmoid(1.0) ** 20, sigmoid(0.0) ** 20,
         (sigmoid(1.0) / sigmoid(0.0)) ** 20))
print("   neither is 'remembering' on its own. the bias just decides where")
print("   the cell STARTS, before training moves it -- and starting shut is")
print("   starting with no gradient path to learn from.")
print()

print("a concrete memory test -- can the cell hold one bit across a gap?")
def carry_test(gap, forget_bias):
    bf_local = np.full(H, forget_bias)
    h2, c2 = np.zeros(H), np.zeros(H)
    signal = np.zeros(X); signal[0] = 3.0
    for t in range(gap + 1):
        x = signal if t == 0 else np.zeros(X)
        z = np.concatenate([x, h2])
        f = sigmoid(z @ Wf + bf_local)
        i = sigmoid(z @ Wi + bi)
        g = np.tanh(z @ Wc + bc)
        o = sigmoid(z @ Wo + bo)
        c2 = f * c2 + i * g
        h2 = o * np.tanh(c2)
    return np.abs(c2).max()

print("%12s %18s %18s" % ("gap", "forget bias 0", "forget bias 3"))
for gap in (1, 5, 20, 60):
    print("%12d %18.6f %18.6f" % (gap, carry_test(gap, 0.0), carry_test(gap, 3.0)))
print("   biased shut, the trace is gone within a handful of steps. biased")
print("   open, the state persists -- and in fact drifts upward, because the")
print("   input gate is never fully closed either and keeps adding a little")
print("   at every step. an untrained cell leaks in both directions.")
print()
print("   in a trained model both biases are learned per unit, which is how")
print("   some units end up as long-term registers and others as fast")
print("   scratch space. nobody assigns those roles; the gradient does.")

Experiments to try

1. Count the cost of gating. Press Parameters and compare with a plain RNN of the same size. An LSTM has four weight matrices instead of one — three gates plus the candidate — so it needs roughly four times the parameters.
2. Grow the state. Raise State Size and watch the count climb. Each of the four blocks is (d + h) × h, so the total is 4[(d + h)h + h].
3. Follow one timestep. Press Simulate Cell Operation and trace the cell state along the top. It passes straight through, touched only by a multiply and an add — that unbroken line is the long-term memory path.
4. Watch the gates open and close. During the simulation note that gate values are between 0 and 1 rather than binary. Gating is continuous, which is what makes it differentiable and therefore learnable.

Traps worth knowing

• Initialising the forget-gate bias at zero. That puts the gate at 0.5, halving the cell state every step and destroying memory before training can fix it. Initialise it to 1 or 2 so the cell starts out remembering.
• Reaching for an LSTM by default. A GRU merges the forget and input gates into one, uses about 25% fewer parameters, trains faster, and performs comparably on most tasks. Try it first.
• Confusing cell state with hidden state. PyTorch returns both as a tuple; passing the wrong one to the next layer is a common and quiet bug.
• Using an LSTM where attention belongs. For long documents a transformer both remembers further and parallelises, which an LSTM cannot.

In one line

An LSTM separates long-term memory (the cell state) from working output (the hidden state) and controls the flow between them with three learned sigmoid gates. Because the cell state is updated by addition and gated multiplication rather than a matrix multiply, the gradient travels back through it multiplied only by the forget gate — so when the model chooses to remember, it genuinely can, for hundreds of steps. The cost is four times the parameters of a simple cell, and a GRU usually gets most of the benefit for less.

Using one, and the shapes involved

import torch.nn as nn

lstm = nn.LSTM(input_size=300,       # embedding dimension
               hidden_size=256,
               num_layers=2,
               batch_first=True,
               bidirectional=False,
               dropout=0.2)          # applies between layers only

out, (h, c) = lstm(x)                # x: (batch, seq, 300)
# out: (batch, seq, 256)   -- the hidden state at every step
# h, c: (layers, batch, 256) -- the final states

Which output you use depends on the task. For classification, take the final hidden state (or a pooled version of all steps). For tagging or sequence-to-sequence, use the full out tensor.

Parameter count is worth knowing, because it explains why LSTMs are heavier than they look: four gates each need a weight matrix over the concatenated input and hidden state, giving 4 × ((input + hidden) × hidden + hidden). For 300-dimensional input and 256 hidden units that is about 570,000 parameters per layer.

For variable-length batches, use pack_padded_sequence so the LSTM does not process padding — otherwise the final hidden state reflects the padding rather than the last real token.

LSTM, GRU or transformer?

GRU merges the forget and input gates into a single update gate and drops the separate cell state. Fewer parameters, slightly faster, and empirically comparable on most tasks — the choice between them is usually not worth agonising over.

Transformers won for large-scale language work because they parallelise across the sequence and have constant path length between positions. But LSTMs remain a sensible choice for streaming inference with bounded memory, for small datasets, for on-device models, and for classical time-series work.

Questions people ask

Are LSTMs obsolete? Superseded for large-scale language modelling; still practical for streaming, small data, embedded use and time series.

What is the difference between the cell state and the hidden state? The cell state is protected long-term memory updated additively; the hidden state is the filtered output exposed at each step.

How many layers? One or two for most tasks. Deep stacks of LSTMs are hard to train and rarely pay off.

Why tanh and sigmoid rather than ReLU? Sigmoid bounds gates to 0–1, which is what a gate requires. Tanh bounds content to −1–1, which keeps the repeatedly-applied recurrence stable.

Do I need gradient clipping? Yes — recurrent models are the classic case for it.

Should I use a bidirectional LSTM? For classification and tagging where the whole input is available, yes — it usually helps. Never for generation or streaming, where the future is not available.

Recap in one screen

• Two states: a protected cell state for long-term memory, a hidden state for the working output.
• Three sigmoid gates decide what to forget, what to store and what to expose.
• The cell updates additively, so ∂cₜ/∂cₜ₋₁ is the forget gate — near 1 means the gradient survives.
• That additive path is the same idea as a residual connection, and it is what fixed vanishing gradients.
• Superseded by transformers at scale; still the right tool for streaming, small data and on-device work.