Zoom in on the gate that performs the write. The Input Gate ($i_t$) does not decide what the new information is — it decides how much of it is allowed into the cell state.
Like the Forget Gate, the Input Gate is a sigmoid layer emitting one value per memory slot:
$$i_t = \sigma(W_i \cdot [H_{t-1}, X_t] + b_i)$$
But notice what it is multiplied against. It never touches the old memory. It scales the candidate vector $\tilde{C}_t$ — the freshly proposed content — before that content is added to the cell state:
$$C_t = \underbrace{f_t \odot C_{t-1}}_{\text{what survives}} + \underbrace{i_t \odot \tilde{C}_t}_{\text{what gets written}}$$
The input gate and the candidate layer form a division of labour: the candidate proposes what to write. And the input gate decides how much of it to accept. A gate value near 0 blocks a memory slot entirely; near 1, it writes the candidate's full proposal. This element-wise scaling means different slots can be updated by different amounts on the same timestep. The LSTM can at once write strongly to one slot and barely touch another. During backpropagation, the gradient to the candidate flows through the input gate, and the gradient to the input gate flows through the candidate. So training adjusts both what content is proposed and how aggressively it is written. The forget gate and input gate together control the memory's evolution: the forget gate erases selectively. And the input gate writes selectively. When the forget gate is near 1 and the input gate is near 0, the memory is copied forward unchanged — the LSTM is coasting. When both are active, the memory is being rewritten. This page animates the input gate and candidate at each timestep, shows which memory slots receive new content, and covers the forget-input interplay.
Zoom in on the gate that performs the write.
Having decided what to erase, the LSTM must decide what to store. The Input Gate is the valve on that write — and it works as one half of a two-part team.
Like the Forget Gate, the Input Gate is a sigmoid layer emitting one value per memory slot:
$$i_t = \sigma(W_i \cdot [H_{t-1}, X_t] + b_i)$$
But notice what it is multiplied against. It never touches the old memory. It scales the candidate vector $\tilde{C}_t$ — the freshly proposed content — before that content is added to the cell state:
$$C_t = \underbrace{f_t \odot C_{t-1}}_{\text{what survives}} + \underbrace{i_t \odot \tilde{C}_t}_{\text{what gets written}}$$
The input gate and the candidate layer form a division of labour: the candidate proposes what to write. And the input gate decides how much of it to accept. A gate value near 0 blocks a memory slot entirely; near 1, it writes the candidate's full proposal. This element-wise scaling means different slots can be updated by different amounts on the same timestep. The LSTM can at once write strongly to one slot and barely touch another. During backpropagation, the gradient to the candidate flows through the input gate, and the gradient to the input gate flows through the candidate. So training adjusts both what content is proposed and how aggressively it is written. The forget gate and input gate together control the memory's evolution: the forget gate erases selectively. And the input gate writes selectively. When the forget gate is near 1 and the input gate is near 0, the memory is copied forward unchanged — the LSTM is coasting. When both are active, the memory is being rewritten. This page animates the input gate and candidate at each timestep, shows which memory slots receive new content, and covers the forget-input interplay.
Beginners often merge these two ideas, so it is worth stating plainly:
Splitting the decision this way is what lets the cell be selective per dimension. The candidate can propose a strong update to eight memory slots while the gate admits it into only two of them.
The written term is added to what the Forget Gate let through. This matters enormously. Replacement would destroy old memory every time something new arrived; addition lets the cell build up. Keeping the subject of a sentence while layering on new adjectives.
It also protects the gradient. Addition passes gradients backwards unchanged, which is precisely why the cell state acts as a highway through time rather than a lossy relay.
A natural assumption is that $i_t = 1 - f_t$ — that whatever you forget, you replace. In a standard LSTM this is false: the two gates are independent layers with their own weights. And all four combinations are meaningful.
The cell can keep old memory and write new content ($f\!\approx\!1, i\!\approx\!1$), keep memory and write nothing ($f\!\approx\!1, i\!\approx\!0$), wipe and overwrite ($f\!\approx\!0, i\!\approx\!1$), or wipe and store nothing at all ($f\!\approx\!0, i\!\approx\!0$). The GRU is the variant that does tie them together with a single update gate — which is exactly how it saves a set of weights.
The forget gate decides what leaves memory. The input gate decides what enters it — and it works in a pair with the candidate memory.
iᵗ = σ(Wᵢ · [hᵗ₋₁, xᵗ] + bᵢ) the gate
gᵗ = tanh(Wᵕ · [hᵗ₋₁, xᵗ] + bᵕ) the candidate
cᵗ = fᵗ ⊙ cᵗ₋₁ + iᵗ ⊙ gᵗ
Two separate computations from the same inputs, with different activations and different jobs:
g uses tanh, so its values run from −1 to 1. It is what could be written — content, which may be positive or negative.i uses sigmoid, so its values run from 0 to 1. It is how much of that content to actually write.Their elementwise product is what gets added to the cell state. A dimension with a strong candidate and a closed gate contributes nothing; a dimension with a weak candidate and an open gate contributes a little.
The obvious question: why not have one computation produce both the content and its strength?
Because they are different decisions and depend on different things. Whether a piece of information is relevant now is a question about context; what that information is is a question about the current input. Separating them lets the cell learn "this input carries a strong signal, but it is not relevant at this point in the sequence" — which a single computation cannot express.
The practical example is selectivity. Reading "The cat, which was orange, sat", the cell wants to store the subject (cat) and update its state as the clause proceeds without letting "orange" overwrite the subject. Separate gate and candidate make that possible: the candidate can carry "orange" while the gate keeps it out of the dimensions holding the subject.
Gate i | Candidate g | |
|---|---|---|
| Activation | Sigmoid | Tanh |
| Range | 0 to 1 | −1 to 1 |
| Meaning | How much to write | What to write |
| Analogy | The valve | The fluid |
The cell update has two terms, and the two gates control one each. That independence is what distinguishes an LSTM from a GRU.
| Forget | Input | Result for that dimension |
|---|---|---|
| 1 | 0 | Keep old memory unchanged |
| 0 | 1 | Replace entirely with new |
| 1 | 1 | Add new on top of old — accumulate |
| 0 | 0 | Clear the dimension |
| 0.9 | 0.3 | Decay slowly while adding a little |
The third row is what a GRU cannot do. Its update gate uses z and 1 − z, so keeping the old state at full strength forces the new contribution to zero. An LSTM can do both, and that is the expressiveness it buys with its extra gate.
The 1 → 1 case matters in practice: it is how a cell builds up evidence over several steps, such as building up a sentiment reading across a sentence rather than replacing it word by word.
The input gate is what lets a cell see something and choose to ignore it. Without it every token would be written to memory, which is the defect it exists to fix.
The next block continues with the remaining demonstrations.
The input gate partners with the forget gate to give four distinct behaviours per step.
A GRU ties the forget and input gates into one dial, which costs expressiveness but saves parameters.
The Input Gate is the LSTM's write-enable line. It converts a proposal into an actual memory change, one dimension at a time, and it does so by adding rather than overwriting. To see where that proposal comes from, study the Candidate Memory layer next.
The gate stays closed. If i is consistently near 0, nothing new ever enters memory and the cell state is frozen. The model behaves as if it had no memory at all beyond the current input. Usually a symptom of a learning rate or initialisation problem, or a task where the model has found no use for memory.
The gate stays open. If i is consistently near 1 while the forget gate is also near 1, the cell state grows without bound as contributions build up. Values in the hundreds appear, tanh in the output path saturates, and gradients vanish through the output gate. Layer normalisation inside the cell is one remedy.
Saturated sigmoids. A gate pinned at 0 or 1 has a derivative near zero, so it stops learning when to open and close. Keeping pre-activations in a reasonable range — scaled inputs, sensible initialisation — prevents it.
Diagnosing these means logging the mean and spread of gate activations:
# with a manual cell that returns its gates
print(f"input gate: mean {i.mean():.3f} std {i.std():.3f}")Healthy gates have a mean somewhere in the middle of the range and genuine variation across dimensions and time steps. A mean of 0.99 with a standard deviation of 0.001 means the gate has stopped being a gate.
Each gate needs its own weight matrix over the concatenated previous hidden state and current input:
per gate: (input + hidden) × hidden + hidden
With 300-dimensional input and 256 hidden units that is 143,616 parameters per gate. An LSTM has four such matrices — forget, input, candidate and output — so about 574,000 per layer.
That is why LSTMs are heavier than they appear, and why a GRU with three matrices instead of four is roughly 25% smaller. In practice the four are computed as one fused matrix multiplication and then split, which is what nn.LSTM does internally and why it is much faster than a hand-written loop.
Why does the candidate use tanh and the gate use sigmoid? Content should be able to be negative, so tanh. A gate is a fraction, so sigmoid.
Is the input gate the same as the update gate in a GRU? Related but not the same. A GRU's update gate does the work of forget and input together, with coupled coefficients.
Can the input gate be negative? No — sigmoid output is strictly between 0 and 1. Negative content comes from the candidate.
What if the input gate is always 0? The cell never stores anything new, and memory is useless. Check initialisation and learning rate.
Should its bias be set up specially? Zero is standard. It is the forget gate that benefits from a bias of 1.
Are the four matrices computed separately? No — implementations concatenate them into one multiplication and split the result, which is considerably faster.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “What the Gate Actually Is”?
What does this module say about “Why Addition, Not Replacement”?
A sigmoid layer that scales the candidate vector element by element, deciding how much new information each memory slot accepts.