Understanding Recurrent Neural Networks
Unlike traditional feedforward networks which process data in single, independent snapshots, Recurrent Neural Networks (RNNs) are designed specifically for sequential data. They maintain a "memory" of previous inputs by passing information across time steps.
1. The Temporal Dimension (Unrolling)
In the 3D visualization above, the network is "unrolled" across time along the Z-axis. While it looks like a massive network, it is actually the exact same set of weights being applied repeatedly at each time step $t$. The $Z$-axis visually represents the passage of time.
2. Deep RNNs (Multiple Layers)
Just like standard neural networks benefit from depth, RNNs can be stacked into Deep RNNs. By setting the "Hidden Layers" dropdown to 2 or 3, you create a hierarchy. The first hidden layer extracts basic temporal features from the raw input sequence, while higher layers piece together those basic features to understand more complex, long-term abstractions. In deep RNNs, the output of Hidden Layer 1 at time $t$ serves as the input to Hidden Layer 2 at time $t$.
3. RNN Architectures
By varying which inputs and outputs we pay attention to across time steps, we can solve drastically different problems. You can switch between these architectures using the settings above:
- Many-to-Many: Sequence to Sequence (e.g., Machine Translation, generating a French sentence from an English sentence).
- Many-to-One: Sequence to Vector (e.g., Sentiment Analysis, reading a whole paragraph to output a single Positive/Negative score).
- One-to-Many: Vector to Sequence (e.g., Image Captioning, taking a single image and generating a sentence describing it).
- One-to-One: The classic Standard Neural Network. No memory over time, just 1 input to 1 output.
4. Sequence Length, Padding & Truncation
Neural networks generally require inputs of a fixed size. If you set the Sequence Length to 6, but type a sentence with only 4 words, the network must pad the remaining sequence. We do this by passing a blank [PAD] token (visually represented as faded inputs) to fill the remaining time steps. Conversely, if your sentence is too long, the excess words are truncated and ignored.
5. The Hidden State Math
The core feature of an RNN is its Hidden State ($h_t$). At any given time step $t$, a hidden layer $l$ receives two distinct inputs:
- The data coming from the layer below it at the same time step $t$.
- Its own hidden state from the previous time step: $\mathbf{h}^{(l)}_{t-1}$.