Take a microscopic look inside a single Recurrent Neural Network block. Visualize how it simply merges past memory with new input data to generate a new context state.
The internal architecture of a standard Recurrent Neural Network block is wonderfully simple. It takes its past memory, merges it with the present, and squashes it through a single neural layer to create the future state.
Unlike standard feed-forward networks, RNNs pass a single vector of context forward through time. This is the Hidden State ($H_t$).
Inside the cell, there is no complex long-term conveyor belt (like in an LSTM). There is only this singular hidden state, serving as both the short-term working memory and the network's output representation for that specific step.
When the simulation runs, watch the data flow through these sequential steps inside the cell:
Because the standard RNN only has this simple $tanh$ calculation repeated over and over, it suffers heavily from Vanishing Gradients. When it tries to learn connections across long sequences (e.g., matching a word at the end of a long paragraph to one at the beginning), the learning signal multiplied repeatedly through the layers shrinks to almost zero. This is why more complex architectures like LSTMs were invented!
Take a microscopic look inside a single Recurrent Neural Network block. Visualize how it simply merges past memory with new input data to generate a new context state.