Zoom in on the first gate of the LSTM. The Forget Gate ($f_t$) looks at the new input and the previous output, then decides — one number per memory slot — how much of the old cell state survives.
The Forget Gate is a small neural layer with a sigmoid output. It reads the merged vector $[H_{t-1}, X_t]$ and produces one number between 0 and 1 for every slot in the cell state:
$$f_t = \sigma(W_f \cdot [H_{t-1}, X_t] + b_f)$$
Those numbers are then times element-by-element into the old memory: $f_t \odot C_{t-1}$. A slot that receives 1 is copied forward perfectly. A slot that receives 0 is erased. Everything in between is partial fading. The sigmoid activation is the right choice precisely because its output is bounded between 0 and 1, making it a natural gate: each slot in the cell state is independently scaled by a retention fraction. Contrast this with the candidate layer, which uses tanh because it needs to propose both positive and negative updates. The forget gate's bias is usually set up to a large positive value (1 or 2) so that the gate starts near 1 and the network begins by remembering everything. Learning to forget specific slots over time is easier than learning to remember from scratch. Without this initialisation trick, LSTMs often fail to learn long-range dependencies at all. The output gate controls what is revealed from memory, but the forget gate controls what survives in the first place — making it the most critical gate for long-term memory retention. This page animates the forget gate at each timestep, shows which memory slots are retained and which are erased, and covers the bias initialisation trick.
Zoom in on the first gate of the LSTM.
Before an LSTM writes anything new, it decides what of the past is still worth keeping. That single decision is what makes long-term memory possible.
The Forget Gate is a small neural layer with a sigmoid output. It reads the merged vector $[H_{t-1}, X_t]$ and produces one number between 0 and 1 for every slot in the cell state:
$$f_t = \sigma(W_f \cdot [H_{t-1}, X_t] + b_f)$$
Those numbers are then times element-by-element into the old memory: $f_t \odot C_{t-1}$. A slot that receives 1 is copied forward perfectly. A slot that receives 0 is erased. Everything in between is partial fading. The sigmoid activation is the right choice precisely because its output is bounded between 0 and 1, making it a natural gate: each slot in the cell state is independently scaled by a retention fraction. Contrast this with the candidate layer, which uses tanh because it needs to propose both positive and negative updates. The forget gate's bias is usually set up to a large positive value (1 or 2) so that the gate starts near 1 and the network begins by remembering everything. Learning to forget specific slots over time is easier than learning to remember from scratch. Without this initialisation trick, LSTMs often fail to learn long-range dependencies at all. The output gate controls what is revealed from memory, but the forget gate controls what survives in the first place — making it the most critical gate for long-term memory retention. This page animates the forget gate at each timestep, shows which memory slots are retained and which are erased, and covers the bias initialisation trick.
The word "gate" suggests open or closed, but the sigmoid makes it a dimmer per dimension. In a real trained LSTM the forget vector might look like $[0.98, 0.03, 0.71, 0.99, \dots]$. The cell holds on to some facts with near-perfect fidelity while dumping others in the same time step.
This is the important difference from a plain RNN. An RNN squashes its entire state through $\tanh$ at every step. So everything decays together. The LSTM decides slot by slot, so a subject noun can survive twenty words while a comma is discarded at once.
Memory does not decay once, it decays repeatedly. If the gate emits roughly the same value $f$ each step, the fraction of the original signal still present after $n$ steps is:
$$\text{survival} = f^{\,n}$$
Because this is exponential, small differences in $f$ produce enormous differences in reach. Play with the slider in the Forget Gate Lab and watch the two survival bars:
That last case is the reason LSTMs defeat the vanishing gradient. When $f_t \approx 1$, the update becomes $C_t \approx C_{t-1} + \text{something}$ — pure addition. Gradients flowing backwards along this path are times by $f_t \approx 1$ instead of being squashed. So they survive the trip.
At initialisation the weights are near zero, so $f_t \approx \sigma(0) = 0.5$. The network starts out forgetting half its memory every single step and struggles to learn anything long-range. The standard fix is to initialise the forget-gate bias $b_f$ to +1 or +2. This pushes the gate to roughly 0.73–0.88 from the start. The cell begins life inclined to remember, and learns what to discard from there.
The forget gate is the first thing an LSTM cell computes. And it answers one question: of everything currently in memory, how much of each part should survive this step?
fᵗ = σ(Wᶠ · [hᵗ₋₁, xᵗ] + bᶠ)
It takes the previous hidden state and the current input, produces a vector the same width as the cell state, and squashes every element through a sigmoid to between 0 and 1. Then:
cᵗ = fᵗ ⊙ cᵗ₋₁ + (new information)
The multiplication is elementwise, which is the important detail. Each dimension of the cell state is scaled independently, so the gate can keep one dimension entirely (a coefficient of 1) while erasing another (a coefficient of 0) in the same step.
| Gate value | Effect on that dimension |
|---|---|
| 1.0 | Keep completely |
| 0.7 | Keep most of it, decay slowly |
| 0.3 | Mostly discard |
| 0.0 | Erase entirely |
Concretely, in a language model, the forget gate learns to clear state at boundaries.
Suppose a dimension of the cell state tracks the grammatical number of the current subject. So that the verb can agree with it. When a new sentence begins, that information is no longer relevant. It would actively mislead. A trained forget gate learns to output near 0 for that dimension when it sees a full stop, and near 1 while the sentence continues.
Other observed behaviours: clearing after a closing bracket, resetting on a topic shift, and dropping a tracked entity once a new one is introduced.
The gate does not "know" any of this. It is a learned function of the previous state and the current input. And these behaviours are what minimising the loss produced.
The forget gate is also the reason LSTMs handle long sequences. And the mechanism is a single derivative.
Differentiating the cell update about the previous cell state gives:
∂cᵗ / ∂cᵗ₋₁ = fᵗ
That is it — the forget gate itself. So the gradient flowing back through k steps is times by the product of k forget-gate values.
If the network has learned to keep a piece of information, those values are near 1. And the product stays near 1 across many steps. The gradient reaches back in effect undamped, and the network can learn that something 100 steps ago mattered.
Compare a plain RNN, where the per-step factor is the derivative of a tanh times a weight matrix. A quantity nothing constrains to be near 1, and which is usually below it. That is the whole difference between failing at 20 steps and succeeding at 200.
There is a practical corollary: the forget gate bias is often set up to 1 (rather than 0). So the gate starts open and information is preserved by default. The network then learns when to close it. Setting up it at 0 means the gate starts half-closed, memory decays from the first step, and training is measurably slower.
The forget gate multiplies the old memory by a number between 0 and 1. That single multiplication is what makes an LSTM's gradient survive, and what makes it possible to clear the memory on purpose.
Building on the same setup, the next block shows the remaining cases.
The following block covers the remaining operations.
The Forget Gate is the only component that can remove information from an LSTM's long-term memory. And it does so multiplicatively and permanently. Keeping it near 1 turns the cell state into a protected highway across time; letting it drift toward 0 collapses the LSTM back into a forgetful RNN. Next, see how new information gets written in: the Input Gate.
| Gate | Activation | Question | Applied to |
|---|---|---|---|
| Forget | Sigmoid | What should I drop? | The old cell state |
| Input | Sigmoid | What new information is worth storing? | The candidate |
| Output | Sigmoid | What should I expose now? | tanh of the cell state |
All three use a sigmoid, because a gate must be a fraction between 0 and 1. The content they control — the candidate memory — uses tanh, because content should be able to be negative.
Note that forget and input are separate gates in an LSTM. This means the cell can keep everything and add new information at full strength in the same step. A GRU merges them into one update gate with coefficients z and 1 − z. So keeping more necessarily means adding less. That is the expressiveness the GRU trades away for a quarter fewer parameters.
Forget-gate activations are unusually interpretable for a neural network internal, and they are worth looking at when debugging a sequence model.
# PyTorch's fused LSTM does not expose gates; a manual cell does
class Cell(nn.Module):
def forward(self, x, state):
h, c = state
gates = self.W(torch.cat([h, x], dim=-1))
f, i, g, o = gates.chunk(4, dim=-1)
f, i, o = f.sigmoid(), i.sigmoid(), o.sigmoid()
c = f * c + i * g.tanh()
h = o * c.tanh()
return h, (h, c), f # return f to inspect itWhat to look for:
Forget values stuck near 0 across the whole sequence mean memory is being wiped every step. And the cell is behaving like a plain feed-forward layer. Check the bias initialisation.
Forget values stuck near 1 mean nothing is ever cleared, so the state builds up without bound and old information crowds out new.
Sharp drops at sentence or clause boundaries are the healthy pattern. The gate has learned where context resets.
Why is it called "forget" when 1 means keep? The value is the retention coefficient. The name refers to the gate's purpose, not to its numeric direction. A genuine source of confusion.
Can it forget fully? A sigmoid never reaches exactly 0, so it asymptotes rather than erasing outright. In practice values of 0.01 are in effect erasure.
Why initialise the bias to 1? So the gate starts open and memory is preserved by default. This trains faster and better than starting half-closed.
Does the forget gate cause vanishing gradients? It is the reason they are avoided. When it is near 1, the gradient passes back undamped. Consistently small values would cause decay, which is what the bias initialisation guards against.
Is it shared across dimensions? No — it produces one value per cell-state dimension. So each can be managed independently.
Do GRUs have a forget gate? Not apart. Its update gate does the job of both forget and input, with coupled coefficients.
∂cₜ/∂cₜ₋₁ is exactly the forget gate — near 1 means the gradient survives many steps.Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “What the Gate Actually Is”?
What does this module say about “It Is a Dimmer, Not a Switch”?
See how a sigmoid between 0 and 1 erases or preserves the cell state, and how forgetting compounds over successive time steps.