vizlearn
Home / Deep Learning

What is LSTM?

Take a microscopic look inside a single Long Short-Term Memory cell. Visualize the internal architecture, the conveyor belt, and the four gates that control memory.

Waiting for Data...

Internal Flow

$C_{t-1} \to C_t$ (Cell State)
$H_{t-1} \to H_t$ (Hidden State)
$X_t$ (New Input Data)
Neural Network Gates (W)
Mathematical Operations
$\times$
Pointwise Multiplication
$+$
Pointwise Addition

Inside the LSTM: Unpacking the Cell

The internal architecture of a Long Short-Term Memory cell is designed to act like a smart computer memory module. It decides what to remember, what to forget, and what to output.

The Core Idea: The Cell State

The key to LSTMs is the Cell State ($C_t$), visualized as the horizontal line running across the top of the diagram. It acts like a conveyor belt.

Information can flow along it easily with only minor linear interactions. The LSTM has the ability to remove or add information to this cell state, carefully regulated by structures called gates.

Step-by-Step Flow

When the simulation runs, watch the data flow through these sequential steps inside the cell:

  1. Concatenation: First, the new data ($X_t$) and the previous short-term memory ($H_{t-1}$) enter the cell from the bottom and are merged together. This combined vector will be fed into four distinct neural network layers (the gates).
  2. 1. The Forget Gate ($f_t$): This layer outputs numbers between 0 and 1 using a Sigmoid function ($\sigma$). It looks at the merged inputs and decides what information we should throw away from the past Cell State. A '0' means "completely forget this", while a '1' means "completely keep this".
    Math: $f_t = \sigma(W_f \cdot [H_{t-1}, X_t] + b_f)$
  3. 2. The Input Gate ($i_t$) & Candidate ($\tilde{C}_t$): Two layers work together to decide what new information to store. The Input Gate ($\sigma$) decides which values to update. The Candidate layer ($tanh$) creates a vector of new candidate values. They are multiplied together to prepare an update.
    Math: $i_t = \sigma(W_i \cdot [H_{t-1}, X_t] + b_i)$ and $\tilde{C}_t = \tanh(W_c \cdot [H_{t-1}, X_t] + b_c)$
  4. 3. Updating the Cell State: The old Cell State ($C_{t-1}$) is multiplied ($\times$) by the Forget Gate to drop old memories. Then, the new candidate data is added ($+$) to it. The conveyor belt has been successfully updated to $C_t$!
  5. 4. The Output Gate ($o_t$): Finally, the cell decides what to output as the new short-term memory ($H_t$). It runs the updated Cell State through a $tanh$ function (to push values between -1 and 1) and multiplies it by the Output Gate's decision.
    Math: $H_t = o_t \times \tanh(C_t)$

The Parameter Cost

Because there are four distinct neural network layers inside this one cell (Forget, Input, Candidate, Output), an LSTM is much more computationally heavy than a standard RNN. Check the Parameters button to see how the matrix dimensions are multiplied by 4.

Cheat sheet

What is LSTM?

Take a microscopic look inside a single Long Short-Term Memory cell. Visualize the internal architecture, the conveyor belt, and the four gates that control memory.

NLP · vizlearn.in/natural_language_processing/what_is_lstm.html

More in NLP

×fNLPForget Gate in LSTM+×iC~NLPInput Gate in LSTM1-1tanhNLPCandidate Memory in LSTMtanh×oHNLPOutput Gate in LSTM