Zoom in on the one internal layer that is not a gate. The Candidate ($\tilde{C}_t$) uses $\tanh$ instead of a sigmoid because its job is to propose real, signed content — not to open or close a valve.
The candidate layer — written $\tilde{C}_t$ and read "C-tilde" — takes the same merged vector as every other layer and produces a fresh piece of content:
$$\tilde{C}_t = \tanh(W_C \cdot [H_{t-1}, X_t] + b_C)$$
It has no authority. It cannot write to memory on its own; the Input Gate decides how much of it survives. Think of it as an employee drafting a proposal and the input gate as the manager approving some, all, or none of it.
Three of the LSTM's four internal layers are sigmoid gates that control flow. This one is different: it is the only layer that manufactures the information itself.
This is the question that separates people who have memorised the diagram from people who understand it. The answer is sign.
Being zero-centred also helps optimisation: the updates do not carry a systematic positive bias that the rest of the network has to cancel out.
Because the cell state is repeatedly added to, an unbounded proposal would let $C_t$ grow without limit over a long sequence and blow up the activations. Clamping every proposal to $[-1, 1]$ means each step can move a memory slot by at most 1. Combined with the Forget Gate's decay, the cell state stays in a numerically comfortable range no matter how long the sequence runs.
The same squashing that keeps things stable creates a trap. The derivative of $\tanh$ is $1 - \tanh^2(z)$: it peaks at 1.0 when $z = 0$ and collapses toward 0 as $|z|$ grows. Once a unit is driven to $z = \pm 4$, its output is stuck near $\pm 1$ and almost no gradient flows back — the unit has effectively stopped learning.
This is why weight initialisation and normalisation matter so much. They keep pre-activations near the middle of the curve, where the layer is still responsive.
The candidate memory is what the cell could add to its state at this step. It is computed from the same inputs as the gates, with one important difference:
gᵗ = tanh(Wᵕ · [hᵗ₋₁, xᵗ] + bᵕ)
Note the tanh, not sigmoid. That is the whole distinction between content and gate:
Candidate g | Gates f, i, o | |
|---|---|---|
| Activation | tanh | Sigmoid |
| Range | −1 to 1 | 0 to 1 |
| Meaning | What to write | How much to write |
| Can be negative | Yes | No |
Content needs to be able to be negative, because a feature can be present, absent, or actively counter-indicated. A gate cannot be negative, because "write −30% of this" is meaningless.
The candidate is then scaled by the input gate before being added:
cᵗ = fᵗ ⊙ cᵗ₋₁ + iᵗ ⊙ gᵗ
So the candidate proposes and the input gate decides. Sometimes described as the pair being a writer and a valve, which is a fair reading.
The cell state accumulates across time steps. If the candidate were unbounded — a linear or ReLU output — then repeated additions could grow the state without limit.
That growth has two consequences. The output path tanh(cₜ) saturates, so its gradient approaches zero and the output gate stops learning. And the numbers themselves can overflow into inf over a long sequence.
Bounding the candidate to ±1 means each step adds at most 1 to any dimension, and the forget gate can subtract. The state can still grow over many steps — which is why cell values in the tens are possible and worth monitoring — and it grows linearly rather than explosively.
This is also why ReLU is not used inside recurrent cells. It is the default in feed-forward networks and the wrong choice here: an unbounded activation applied repeatedly through a recurrence produces instability that gating cannot contain.
Take a cell tracking sentiment while reading "the film was not very good".
At "good", the candidate for the sentiment dimension is strongly positive — that is what the word contributes in isolation. At "not", earlier, a different dimension carried a negation signal, and the state now reflects it.
What the trained cell learns is to use the gates to combine these correctly rather than letting the last strong candidate win. The candidate at "good" may be +0.9 while the input gate for that dimension is low, because the context makes a naive positive reading wrong.
That division of labour — the candidate reading the current word, the gates reading the context — is what the four-way structure buys, and it is why a single computation producing both content and strength would be less expressive.
Three of an LSTM's four small networks are sigmoids that act as valves. The candidate is a tanh, and it is the one that carries actual content -- which is why its activation function is different.
The candidate layer is the LSTM's content generator: bounded, signed, and completely powerless on its own. Content and control are deliberately separated — $\tanh$ proposes, sigmoid disposes. The last piece of the cell decides what the outside world gets to see: the Output Gate.
All four of an LSTM's transformations take the same input — the concatenation of the previous hidden state and the current input — and produce vectors of the same width. So implementations compute them as one matrix multiplication and split the result:
gates = x @ W_ih.T + h @ W_hh.T + b # one multiply, 4x hidden wide
i, f, g, o = gates.chunk(4, dim=-1) # split into the four parts
i, f, o = i.sigmoid(), f.sigmoid(), o.sigmoid()
g = g.tanh() # only the candidate uses tanh
c = f * c + i * g
h = o * c.tanh()That fusion is why nn.LSTM is dramatically faster than a hand-written Python loop: one large matrix multiplication per step rather than four small ones, with cuDNN kernels underneath.
The candidate's weight matrix is the same size as each gate's: (input + hidden) × hidden. With 300-dimensional input and 256 hidden units, about 143,000 parameters — a quarter of the cell's total.
Candidate saturated at ±1. If the pre-activation is consistently large, tanh is flat and its derivative is near zero, so the candidate's weights stop learning. Scaled inputs and sensible initialisation prevent it; layer normalisation inside the cell fixes it directly.
Candidate near zero everywhere. Nothing is ever proposed for storage, so memory never updates usefully. Usually a dead-weights or learning-rate problem.
Cell state growing without bound. Forget gate near 1 and input gate near 1 for many steps means candidates accumulate. Check |c|: values above about 5 mean the output tanh is saturated.
Monitoring is straightforward with a manual cell:
print(f"candidate: mean {g.mean():.3f} |max| {g.abs().max():.3f}")
print(f"cell state: |max| {c.abs().max():.3f}")A candidate whose absolute maximum is pinned at 1.0 across all dimensions is saturated. Healthy values are spread across the range.
Why tanh rather than sigmoid for the candidate? Content must be able to be negative. Sigmoid's 0-to-1 range could only ever add.
Why not ReLU? It is unbounded, and an unbounded activation added repeatedly through a recurrence destabilises the cell state.
Is the candidate the same as the new cell state? No — it is the proposal. The new cell state is the retained old state plus the gated candidate.
Can the candidate be ignored entirely? Yes, when the input gate is near 0 for that dimension. That is the mechanism for "this word is not relevant here".
Does it have its own weights? Yes — one matrix, the same size as each gate's, computed in the same fused multiplication.
What is g called in the literature? Variously the candidate, the candidate cell state, c̃ₜ, or the "new memory content". They all mean this.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
The candidate layer is the LSTM's content generator: bounded, signed, and completely powerless on its own. Content and control are deliberately separated — $\tanh$ proposes, sigmoid disposes. The last piece of the cell decides what the outside world gets to see: the Output Gate .
What does this module say about “A Proposal, Not a Decision”?
The candidate layer — written $\tilde{C}_t$ and read "C-tilde" — takes the same merged vector as every other layer and produces a fresh piece of content:
What does this module say about “Why $\tanh$ and Not a Sigmoid”?
This is the question that separates people who have memorised the diagram from people who understand it. The answer is sign .
Zoom in on the one internal layer that is not a gate. The Candidate ($\tilde{C}_t$) uses $\tanh$ instead of a sigmoid because its job is to propose real, signed content — not to open or close a valve.