Zoom in on the first gate of the LSTM. The Forget Gate ($f_t$) looks at the new input and the previous output, then decides — one number per memory slot — how much of the old cell state survives.
The Forget Gate is a small neural layer with a sigmoid output. It reads the merged vector $[H_{t-1}, X_t]$ and produces one number between 0 and 1 for every slot in the cell state:
$$f_t = \sigma(W_f \cdot [H_{t-1}, X_t] + b_f)$$
Those numbers are then multiplied element-by-element into the old memory: $f_t \odot C_{t-1}$. A slot that receives 1 is copied forward perfectly. A slot that receives 0 is erased. Everything in between is partial fading.
Before an LSTM writes anything new, it decides what of the past is still worth keeping. That single decision is what makes long-term memory possible.
The word "gate" suggests open or closed, but the sigmoid makes it a dimmer per dimension. In a real trained LSTM the forget vector might look like $[0.98, 0.03, 0.71, 0.99, \dots]$ — the cell holds on to some facts with near-perfect fidelity while dumping others in the same time step.
This is the crucial difference from a plain RNN. An RNN squashes its entire state through $\tanh$ at every step, so everything decays together. The LSTM decides slot by slot, so a subject noun can survive twenty words while a comma is discarded immediately.
Memory does not decay once, it decays repeatedly. If the gate emits roughly the same value $f$ each step, the fraction of the original signal still present after $n$ steps is:
$$\text{survival} = f^{\,n}$$
Because this is exponential, small differences in $f$ produce enormous differences in reach. Play with the slider in the Forget Gate Lab and watch the two survival bars:
That last case is the reason LSTMs defeat the vanishing gradient. When $f_t \approx 1$, the update becomes $C_t \approx C_{t-1} + \text{something}$ — pure addition. Gradients flowing backwards along this path are multiplied by $f_t \approx 1$ instead of being squashed, so they survive the trip.
At initialisation the weights are near zero, so $f_t \approx \sigma(0) = 0.5$ — the network starts out forgetting half its memory every single step and struggles to learn anything long-range. The standard fix is to initialise the forget-gate bias $b_f$ to +1 or +2, which pushes the gate to roughly 0.73–0.88 from the start. The cell begins life inclined to remember, and learns what to discard from there.
The forget gate is the first thing an LSTM cell computes, and it answers one question: of everything currently in memory, how much of each part should survive this step?
fᵗ = σ(Wᶠ · [hᵗ₋₁, xᵗ] + bᶠ)
It takes the previous hidden state and the current input, produces a vector the same width as the cell state, and squashes every element through a sigmoid to between 0 and 1. Then:
cᵗ = fᵗ ⊙ cᵗ₋₁ + (new information)
The multiplication is elementwise, which is the crucial detail. Each dimension of the cell state is scaled independently, so the gate can keep one dimension entirely (a coefficient of 1) while erasing another (a coefficient of 0) in the same step.
| Gate value | Effect on that dimension |
|---|---|
| 1.0 | Keep completely |
| 0.7 | Keep most of it, decay slowly |
| 0.3 | Mostly discard |
| 0.0 | Erase entirely |
Concretely, in a language model, the forget gate learns to clear state at boundaries.
Suppose a dimension of the cell state tracks the grammatical number of the current subject, so that the verb can agree with it. When a new sentence begins, that information is no longer relevant — it would actively mislead. A trained forget gate learns to output near 0 for that dimension when it sees a full stop, and near 1 while the sentence continues.
Other observed behaviours: clearing after a closing bracket, resetting on a topic shift, and dropping a tracked entity once a new one is introduced.
The gate does not "know" any of this. It is a learned function of the previous state and the current input, and these behaviours are what minimising the loss produced.
The forget gate is also the reason LSTMs handle long sequences, and the mechanism is a single derivative.
Differentiating the cell update with respect to the previous cell state gives:
∂cᵗ / ∂cᵗ₋₁ = fᵗ
That is it — the forget gate itself. So the gradient flowing back through k steps is multiplied by the product of k forget-gate values.
If the network has learned to keep a piece of information, those values are near 1, and the product stays near 1 across many steps. The gradient reaches back essentially undamped, and the network can learn that something 100 steps ago mattered.
Compare a plain RNN, where the per-step factor is the derivative of a tanh times a weight matrix — a quantity nothing constrains to be near 1, and which is usually below it. That is the whole difference between failing at 20 steps and succeeding at 200.
There is a practical corollary: the forget gate bias is often initialised to 1 (rather than 0), so the gate starts open and information is preserved by default. The network then learns when to close it. Initialising it at 0 means the gate starts half-closed, memory decays from the first step, and training is measurably slower.
The forget gate multiplies the old memory by a number between 0 and 1. That single multiplication is what makes an LSTM's gradient survive, and what makes it possible to clear the memory on purpose.
The Forget Gate is the only component that can remove information from an LSTM's long-term memory, and it does so multiplicatively and permanently. Keeping it near 1 turns the cell state into a protected highway across time; letting it drift toward 0 collapses the LSTM back into a forgetful RNN. Next, see how new information gets written in: the Input Gate.
| Gate | Activation | Question | Applied to |
|---|---|---|---|
| Forget | Sigmoid | What should I drop? | The old cell state |
| Input | Sigmoid | What new information is worth storing? | The candidate |
| Output | Sigmoid | What should I expose now? | tanh of the cell state |
All three use a sigmoid, because a gate must be a fraction between 0 and 1. The content they control — the candidate memory — uses tanh, because content should be able to be negative.
Note that forget and input are separate gates in an LSTM, which means the cell can keep everything and add new information at full strength in the same step. A GRU merges them into one update gate with coefficients z and 1 − z, so keeping more necessarily means adding less. That is the expressiveness the GRU trades away for a quarter fewer parameters.
Forget-gate activations are unusually interpretable for a neural network internal, and they are worth looking at when debugging a sequence model.
# PyTorch's fused LSTM does not expose gates; a manual cell does
class Cell(nn.Module):
def forward(self, x, state):
h, c = state
gates = self.W(torch.cat([h, x], dim=-1))
f, i, g, o = gates.chunk(4, dim=-1)
f, i, o = f.sigmoid(), i.sigmoid(), o.sigmoid()
c = f * c + i * g.tanh()
h = o * c.tanh()
return h, (h, c), f # return f to inspect itWhat to look for:
Forget values stuck near 0 across the whole sequence mean memory is being wiped every step, and the cell is behaving like a plain feed-forward layer. Check the bias initialisation.
Forget values stuck near 1 mean nothing is ever cleared, so the state accumulates without bound and old information crowds out new.
Sharp drops at sentence or clause boundaries are the healthy pattern — the gate has learned where context resets.
Why is it called "forget" when 1 means keep? The value is the retention coefficient. The name refers to the gate's purpose, not to its numeric direction — a genuine source of confusion.
Can it forget completely? A sigmoid never reaches exactly 0, so it asymptotes rather than erasing outright. In practice values of 0.01 are effectively erasure.
Why initialise the bias to 1? So the gate starts open and memory is preserved by default, which trains faster and better than starting half-closed.
Does the forget gate cause vanishing gradients? It is the reason they are avoided — when it is near 1, the gradient passes back undamped. Consistently small values would cause decay, which is what the bias initialisation guards against.
Is it shared across dimensions? No — it produces one value per cell-state dimension, so each can be managed independently.
Do GRUs have a forget gate? Not separately. Its update gate does the job of both forget and input, with coupled coefficients.
∂cₜ/∂cₜ₋₁ is exactly the forget gate — near 1 means the gradient survives many steps.Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
The Forget Gate is the only component that can remove information from an LSTM's long-term memory, and it does so multiplicatively and permanently. Keeping it near 1 turns the cell state into a protected highway across time; letting it drift toward 0 collapses the LSTM back into a forgetful RNN. Next, see how new information gets written in: the Input Gate .
What does this module say about “What the Gate Actually Is”?
The Forget Gate is a small neural layer with a sigmoid output. It reads the merged vector $[H_{t-1}, X_t]$ and produces one number between 0 and 1 for every slot in the cell state :
What does this module say about “It Is a Dimmer, Not a Switch”?
The word "gate" suggests open or closed, but the sigmoid makes it a dimmer per dimension . In a real trained LSTM the forget vector might look like $[0.98, 0.03, 0.71, 0.99, \dots]$ — the cell holds on to some facts with near-perfect fidelity while dumping others in the same time step.
Zoom in on the first gate of the LSTM. The Forget Gate ($f_t$) looks at the new input and the previous output, then decides — one number per memory slot — how much of the old cell state survives.