Output Gate in LSTM

The Output Gate: Deciding What to Reveal

The cell state is private. The hidden state is public. The Output Gate is the filter between them — and that separation is one of the most under-appreciated ideas in the LSTM.

What the Gate Actually Is

A third sigmoid layer, computed from the same merged input as the others:

$$o_t = \sigma(W_o \cdot [H_{t-1}, X_t] + b_o)$$

It is applied only at the very end, to the already updated cell state, after that state has been squashed by $\tanh$:

$$H_t = o_t \odot \tanh(C_t)$$

Note the ordering carefully: the memory is updated first, then filtered. The Output Gate never changes what is stored — it only changes what is shown.

Two Memories: Private and Public

This gate is the reason an LSTM carries two state vectors instead of one:

$C_t$ — the cell state. Long-term, internal. It never leaves the cell directly and is never seen by the next layer.
$H_t$ — the hidden state. Short-term, external. It is passed to the next layer, used for the prediction, and fed back into this same cell at the next time step.

Because of this split, the cell can remember something without acting on it. Suppose the LSTM reads a plural subject: it stores that fact in $C_t$ and keeps the output gate closed on that slot for many steps. Only when the verb finally arrives does the gate open, releasing "the subject was plural" exactly when it is needed to choose are over is. A single-state RNN cannot do this — anything it remembers, it must also expose.

Why $\tanh(C_t)$ First?

The cell state is a running sum, so it can drift well outside $[-1, 1]$ after many additions. Feeding that raw value to the next layer would produce unstable activations. Squashing with $\tanh$ rescales it into a bounded, zero-centred range before the gate scales it further.

There is a side effect worth seeing in the lab: once $|C_t|$ is large, $\tanh$ flattens out. Piling more into memory stops changing the output. Use the $C_t$ slider beyond $\pm 2.5$ and watch the revealed bar barely move.

It Closes the Loop

$H_t$ is not just the cell's answer — it is also half of the input to the next time step, where it will help compute all four layers again. So the Output Gate does double duty: it decides what the outside world sees, and it decides what the cell tells its own future self. A closed output gate leaves the next step reasoning almost entirely from the incoming token.

Interactive Exploration Guide

Run the simulation. The highlighted route runs from the glowing $o_t$ box to the final $\times$, where it meets the cell state coming down through the $\tanh$ block — and only then splits out as $H_t$, both upward and to the right.
Set the gate to 0 with $C_t$ at 1.50. The blue memory bar stays full while the revealed bar drops to zero. The cell knows something and is deliberately saying nothing.
Now open the gate to 1.00. The same stored memory suddenly appears at the output, unchanged in the cell but fully exposed — that is a gate opening at the moment the information becomes relevant.
Push $C_t$ from 2.5 to 3.0. Memory grows visibly; the revealed value hardly moves. That is $\tanh$ saturation limiting how much any single slot can shout.

Key Takeaway

The Output Gate separates having information from using it. Together with the Forget Gate (what to erase), the Input Gate (what to write), and the Candidate (what the content is), it completes the four-layer machine that makes an LSTM cell work — and explains why it costs four times a simple RNN's parameters.