Input Gate in LSTM

The Input Gate: Deciding What to Write

Having decided what to erase, the LSTM must decide what to store. The Input Gate is the valve on that write — and it works as one half of a two-part team.

What the Gate Actually Is

Like the Forget Gate, the Input Gate is a sigmoid layer emitting one value per memory slot:

$$i_t = \sigma(W_i \cdot [H_{t-1}, X_t] + b_i)$$

But notice what it is multiplied against. It never touches the old memory. It scales the candidate vector $\tilde{C}_t$ — the freshly proposed content — before that content is added to the cell state:

$$C_t = \underbrace{f_t \odot C_{t-1}}_{\text{what survives}} + \underbrace{i_t \odot \tilde{C}_t}_{\text{what gets written}}$$

Two Layers, Two Different Jobs

Beginners often merge these two ideas, so it is worth stating plainly:

The candidate $\tilde{C}_t$ ($\tanh$) answers "what is the new information?" It can be positive or negative, because it is real content.
The input gate $i_t$ ($\sigma$) answers "how much of it do we actually want?" It is strictly between 0 and 1, because it is a volume knob, not content.

Splitting the decision this way is what lets the cell be selective per dimension. The candidate can propose a strong update to eight memory slots while the gate admits it into only two of them.

Why Addition, Not Replacement

The written term is added to what the Forget Gate let through. This matters enormously. Replacement would destroy old memory every time something new arrived; addition lets the cell accumulate — keeping the subject of a sentence while layering on new adjectives.

It also protects the gradient. Addition passes gradients backwards unchanged, which is precisely why the cell state acts as a highway through time rather than a lossy relay.

Forget and Input Are Not Opposites

A natural assumption is that $i_t = 1 - f_t$ — that whatever you forget, you replace. In a standard LSTM this is false: the two gates are independent layers with their own weights, and all four combinations are meaningful.

The cell can keep old memory and write new content ($f\!\approx\!1, i\!\approx\!1$), keep memory and write nothing ($f\!\approx\!1, i\!\approx\!0$), wipe and overwrite ($f\!\approx\!0, i\!\approx\!1$), or wipe and store nothing at all ($f\!\approx\!0, i\!\approx\!0$). The GRU is the variant that does tie them together with a single update gate — which is exactly how it saves a set of weights.

Interactive Exploration Guide

Run the simulation. The highlighted route shows $i_t$ leaving the glowing gate, meeting the candidate at the $\times$ operator, and only then rising to the $+$ on the conveyor belt.
Set the gate to 0 but leave the candidate at 0.80. The written amount drops to zero. The network did all the work of computing a rich candidate and then threw it away — the cell state simply passes through untouched.
Open the gate to 1.00 and sweep the candidate. Now the written amount tracks the candidate exactly, including negative values that pull the memory slot down.
Try gate 1.00 with candidate 0.00. Nothing is written even though the gate is wide open — proof that the gate controls volume, not content.

Key Takeaway

The Input Gate is the LSTM's write-enable line. It converts a proposal into an actual memory change, one dimension at a time, and it does so by adding rather than overwriting. To see where that proposal comes from, study the Candidate Memory layer next.