Zoom in on the first gate of the LSTM. The Forget Gate ($f_t$) looks at the new input and the previous output, then decides — one number per memory slot — how much of the old cell state survives.
Before an LSTM writes anything new, it decides what of the past is still worth keeping. That single decision is what makes long-term memory possible.
The Forget Gate is a small neural layer with a sigmoid output. It reads the merged vector $[H_{t-1}, X_t]$ and produces one number between 0 and 1 for every slot in the cell state:
$$f_t = \sigma(W_f \cdot [H_{t-1}, X_t] + b_f)$$
Those numbers are then multiplied element-by-element into the old memory: $f_t \odot C_{t-1}$. A slot that receives 1 is copied forward perfectly. A slot that receives 0 is erased. Everything in between is partial fading.
The word "gate" suggests open or closed, but the sigmoid makes it a dimmer per dimension. In a real trained LSTM the forget vector might look like $[0.98, 0.03, 0.71, 0.99, \dots]$ — the cell holds on to some facts with near-perfect fidelity while dumping others in the same time step.
This is the crucial difference from a plain RNN. An RNN squashes its entire state through $\tanh$ at every step, so everything decays together. The LSTM decides slot by slot, so a subject noun can survive twenty words while a comma is discarded immediately.
Memory does not decay once, it decays repeatedly. If the gate emits roughly the same value $f$ each step, the fraction of the original signal still present after $n$ steps is:
$$\text{survival} = f^{\,n}$$
Because this is exponential, small differences in $f$ produce enormous differences in reach. Play with the slider in the Forget Gate Lab and watch the two survival bars:
That last case is the reason LSTMs defeat the vanishing gradient. When $f_t \approx 1$, the update becomes $C_t \approx C_{t-1} + \text{something}$ — pure addition. Gradients flowing backwards along this path are multiplied by $f_t \approx 1$ instead of being squashed, so they survive the trip.
At initialisation the weights are near zero, so $f_t \approx \sigma(0) = 0.5$ — the network starts out forgetting half its memory every single step and struggles to learn anything long-range. The standard fix is to initialise the forget-gate bias $b_f$ to +1 or +2, which pushes the gate to roughly 0.73–0.88 from the start. The cell begins life inclined to remember, and learns what to discard from there.
The Forget Gate is the only component that can remove information from an LSTM's long-term memory, and it does so multiplicatively and permanently. Keeping it near 1 turns the cell state into a protected highway across time; letting it drift toward 0 collapses the LSTM back into a forgetful RNN. Next, see how new information gets written in: the Input Gate.
Zoom in on the first gate of the LSTM. The Forget Gate ($f_t$) looks at the new input and the previous output, then decides — one number per memory slot — how much of the old cell state survives.