Zoom in on the final gate. The Output Gate ($o_t$) decides how much of the cell's private memory becomes the visible hidden state $H_t$ — the only thing the rest of the network ever sees.
A third sigmoid layer, computed from the same merged input as the others:
$$o_t = \sigma(W_o \cdot [H_{t-1}, X_t] + b_o)$$
It is applied only at the very end, to the already updated cell state, after that state has been squashed by $\tanh$:
$$H_t = o_t \odot \tanh(C_t)$$
Note the ordering carefully: the memory is updated first, then filtered. The Output Gate never changes what is stored — it only changes what is shown.
The cell state is private. The hidden state is public. The Output Gate is the filter between them — and that separation is one of the most under-appreciated ideas in the LSTM.
This gate is the reason an LSTM carries two state vectors instead of one:
Because of this split, the cell can remember something without acting on it. Suppose the LSTM reads a plural subject: it stores that fact in $C_t$ and keeps the output gate closed on that slot for many steps. Only when the verb finally arrives does the gate open, releasing "the subject was plural" exactly when it is needed to choose are over is. A single-state RNN cannot do this — anything it remembers, it must also expose.
The cell state is a running sum, so it can drift well outside $[-1, 1]$ after many additions. Feeding that raw value to the next layer would produce unstable activations. Squashing with $\tanh$ rescales it into a bounded, zero-centred range before the gate scales it further.
There is a side effect worth seeing in the lab: once $|C_t|$ is large, $\tanh$ flattens out. Piling more into memory stops changing the output. Use the $C_t$ slider beyond $\pm 2.5$ and watch the revealed bar barely move.
$H_t$ is not just the cell's answer — it is also half of the input to the next time step, where it will help compute all four layers again. So the Output Gate does double duty: it decides what the outside world sees, and it decides what the cell tells its own future self. A closed output gate leaves the next step reasoning almost entirely from the incoming token.
The cell state holds everything the LSTM is carrying. The output gate decides how much of it to reveal as this step's output.
oᵗ = σ(Wₒ · [hᵗ₋₁, xᵗ] + bₒ)
hᵗ = oᵗ ⊙ tanh(cᵗ)
Two things happen on that second line. The cell state is squashed through tanh into the range −1 to 1, and then multiplied elementwise by the gate.
The squashing matters: the cell state is unbounded and can grow large through accumulation, so exposing it raw would produce enormous hidden states that destabilise the next layer. The tanh bounds it.
The gate then filters. The cell may be tracking six things and only two of them are relevant to the current output, so the gate opens for those dimensions and closes for the rest.
This is the design decision worth understanding, and it is what distinguishes an LSTM from a GRU.
The cell state is long-term storage. It should keep information for as long as it is needed, whether or not it is relevant at this exact step.
The hidden state is the working output. It feeds the next layer, the prediction head, and the gate computations of the following step.
Those are different requirements. Reading "The keys to the cabinet ___", a language model must remember that the subject is plural (keys, not cabinet) in order to choose "are". While reading the intervening words, that information should stay in memory and should not dominate the output — the output at "cabinet" is about cabinets.
Separating them lets the cell hold information quietly and surface it when needed. A GRU, which has only a hidden state, cannot make that distinction: whatever it remembers is also what it outputs.
Cell state c | Hidden state h | |
|---|---|---|
| Role | Long-term memory | Working output |
| Range | Unbounded | −1 to 1, gated |
| Updated by | Forget and input gates, additively | Output gate, from c |
| Passed to | The next time step only | The next layer, the head, and the next step's gates |
hₜ goes to three places, which is why the output gate matters more than "just the output" suggests:
[hₜ, xₜ₊₁] as input.That third path is the subtle one. The output gate does not only control what leaves the cell — it controls what the cell's own future gate decisions are based on. Close it entirely and the next step's forget, input and output gates see nothing but the new input, effectively cutting the recurrence.
That is a real mechanism rather than a curiosity: a trained cell can use it to isolate a segment of the sequence from what came before.
The output gate is the only one that does not touch memory. It controls what the rest of the network gets to see -- which is why an LSTM can hold something for fifty steps and act on it once.
The Output Gate separates having information from using it. Together with the Forget Gate (what to erase), the Input Gate (what to write), and the Candidate (what the content is), it completes the four-layer machine that makes an LSTM cell work — and explains why it costs four times a simple RNN's parameters.
The output gate sits on the path from the loss back into the cell state, and the tanh in that path is where gradient trouble can appear.
Differentiating hₜ = oₜ ⊙ tanh(cₜ) with respect to cₜ gives oₜ ⊙ (1 − tanh²(cₜ)). Two ways that becomes small:
The gate is closed. If oₜ is near 0, little gradient flows back into the cell state from this step's loss.
The cell state is large. If |cₜ| is big, tanh(cₜ) saturates and 1 − tanh² approaches 0.
The second is the one to watch. A cell whose state has accumulated to values in the tens or hundreds — forget gate near 1 and input gate near 1 for many steps — has a saturated output path and receives almost no gradient through it.
Note this does not affect the main long-range path. The cell-to-cell gradient ∂cₜ/∂cₜ₋₁ = fₜ bypasses the tanh entirely, which is precisely why the LSTM's memory survives distance. The output path is a separate, shorter route.
The practical remedies are the usual ones: layer normalisation inside the cell, gradient clipping, and keeping inputs scaled so the state does not run away.
# manual cell exposing all gates
gates = self.W(torch.cat([h, x], dim=-1))
f, i, g, o = gates.chunk(4, dim=-1)
f, i, o = f.sigmoid(), i.sigmoid(), o.sigmoid()
c = f * c + i * g.tanh()
h = o * c.tanh()What healthy output gates look like: a mean somewhere in the middle of the range, real variation across dimensions, and variation across time steps.
Two failure signatures:
Always open (near 1). The hidden state is just tanh(c), so the gate is doing nothing and the model has lost the memory/output distinction — effectively a GRU with extra parameters.
Always closed (near 0). The hidden state is near zero, the recurrence is broken, and the model is operating on the current input alone. Usually a learning rate or initialisation problem.
Also worth checking the cell state's magnitude directly. If |c| routinely exceeds about 5, the tanh is saturated and the output path is barely differentiable.
Why squash the cell state with tanh before gating? The cell state is unbounded, and passing large values to the next layer destabilises it. Tanh bounds the output to −1 to 1.
Why is the hidden state, not the cell state, passed to the next layer? Because the cell state is private long-term memory; the hidden state is the filtered, bounded view intended for consumption.
Does the output gate affect long-range memory? Not the main path — ∂cₜ/∂cₜ₋₁ is the forget gate and bypasses the output entirely. It affects how much gradient reaches the cell from this step's loss.
What if the output gate is always 1? The gate stops doing anything, and the model loses the separation between what it remembers and what it reports.
Do GRUs have an output gate? No — that is the other simplification, alongside merging forget and input. Their hidden state is exposed directly.
Should its bias be initialised specially? Zero is standard. Only the forget gate benefits from a bias of 1.
tanh(cₜ) to produce the hidden state, one coefficient per dimension.Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Why $\tanh(C_t)$ First”?
The cell state is a running sum, so it can drift well outside $[-1, 1]$ after many additions. Feeding that raw value to the next layer would produce unstable activations. Squashing with $\tanh$ rescales it into a bounded, zero-centred range before the gate scales it further.
What does this module say about “It Closes the Loop”?
$H_t$ is not just the cell's answer — it is also half of the input to the next time step, where it will help compute all four layers again. So the Output Gate does double duty: it decides what the outside world sees, and it decides what the cell tells its own future self. A closed output gate leaves the next step reasoning almost entirely from the incoming token.
What does this module say about “Deciding what to expose”?
The cell state holds everything the LSTM is carrying. The output gate decides how much of it to reveal as this step's output.
Zoom in on the final gate. The Output Gate ($o_t$) decides how much of the cell's private memory becomes the visible hidden state $H_t$ — the only thing the rest of the network ever sees.