Understanding Recurrent Neural Networks

Natural Language Processing·By Ashish Jangra·Updated 11 September 2026

Unlike traditional feedforward networks which process data in single, independent snapshots, Recurrent Neural Networks (RNNs) are designed specifically for sequential data. They maintain a "memory" of previous inputs by passing information across time steps.

1. The Temporal Dimension (Unrolling)

In the 3D visualization above, the network is "unrolled" across time along the Z-axis. While it looks like a massive network, it is actually the exact same set of weights being applied repeatedly at each time step $t$. The $Z$-axis visually represents the passage of time.

2. Deep RNNs (Multiple Layers)

Just like standard neural networks benefit from depth, RNNs can be stacked into Deep RNNs. By setting the "Hidden Layers" dropdown to 2 or 3, you create a hierarchy. The first hidden layer extracts basic temporal features from the raw input sequence, while higher layers piece together those basic features to understand more complex, long-term abstractions. In deep RNNs, the output of Hidden Layer 1 at time $t$ serves as the input to Hidden Layer 2 at time $t$.

3. RNN Architectures

By varying which inputs and outputs we pay attention to across time steps, we can solve drastically different problems. You can switch between these architectures using the settings above:

• Many-to-Many: Sequence to Sequence (e.g., Machine Translation, generating a French sentence from an English sentence).
• Many-to-One: Sequence to Vector (e.g., Sentiment Analysis, reading a whole paragraph to output a single Positive/Negative score).
• One-to-Many: Vector to Sequence (e.g., Image Captioning, taking a single image and generating a sentence describing it).
• One-to-One: The classic Standard Neural Network. No memory over time, just 1 input to 1 output.

4. Sequence Length, Padding & Truncation

Neural networks generally require inputs of a fixed size. If you set the Sequence Length to 6, but type a sentence with only 4 words, the network must pad the remaining sequence. We do this by passing a blank [PAD] token (visually represented as faded inputs) to fill the remaining time steps. Conversely, if your sentence is too long, the excess words are truncated and ignored.

5. The Hidden State Math

The core feature of an RNN is its Hidden State ($h_t$). At any given time step $t$, a hidden layer $l$ receives two distinct inputs:

• The data coming from the layer below it at the same time step $t$.
• Its own hidden state from the previous time step: $\mathbf{h}^{(l)}_{t-1}$.

Read that equation as a sentence: the new memory is a squashed sum of what just arrived and what I already remembered. The $\tanh$ keeps the hidden state bounded between $-1$ and $1$, which prevents the values from growing without limit as they are fed back in step after step.

6. Counting the Weights

An RNN cell has three sets of learnable parameters, and every one of them is shared across all time steps. That sharing is what makes a recurrent network able to handle sequences of any length with a fixed number of weights.

• $\mathbf{W}_{forward}$ — shape $(h \times d)$, mapping the input at this step into hidden space.
• $\mathbf{W}_{recurrent}$ — shape $(h \times h)$, mapping the previous hidden state into the new one.
• $\mathbf{b}$ — one bias per hidden unit, shape $(h)$.

With an input size $d = 50$ and a hidden size $h = 100$, that is $100 \times (50 + 100 + 1) = 15{,}100$ weights — whether the sequence is 6 steps long or 600. The recurrent matrix dominates as the hidden size grows, since it scales with $h^2$ while the input matrix scales only with $h \times d$.

This is the direct analogue of parameter sharing in a convolutional network. There, one filter is reused at every position; here, one weight matrix is reused at every time step. Both assume that what is worth computing at one place in the data is worth computing everywhere in it.

7. Backpropagation Through Time

Training works by unrolling the network exactly as the visualisation above shows it, treating the result as a very deep feedforward network, and running ordinary backpropagation. The gradient for the shared weights is the sum of their contributions at every time step — which is why one weight matrix can be trained by many steps at once.

The complication is depth. A 100-step sequence unrolls into a 100-layer network, and the gradient must travel back through all of it. Because each step multiplies by the same recurrent matrix, that repeated multiplication is where recurrent networks get their notorious training problems.

Two practical techniques address the cost rather than the mathematics. Truncated backpropagation through time unrolls only the most recent $k$ steps, capping memory use at the price of never learning dependencies longer than $k$. Gradient clipping rescales any gradient whose norm exceeds a threshold, which is a crude fix and an extremely effective one.

8. Vanishing and Exploding Gradients

Propagating a gradient back through $n$ time steps involves multiplying by the recurrent weight matrix roughly $n$ times. The outcome depends on the magnitude of that matrix, and neither case is benign:

• Vanishing: if the relevant factors are smaller than 1, repeated multiplication drives the gradient towards zero. Early time steps receive almost no learning signal, so the network cannot associate a cause with an effect that arrives much later.
• Exploding: if they are larger than 1, the gradient grows exponentially, the weight update overshoots wildly, and the loss becomes NaN within a few batches.

Exploding gradients are loud and easy to fix — clip them. Vanishing gradients are the harder problem, because nothing crashes. Training appears to proceed, the loss falls a little, and the model simply never learns anything that depends on information from more than ten or twenty steps ago.

The classic demonstration is a sentence whose subject and verb are far apart: "The keys that I left on the table in the hallway upstairs are missing." Choosing "are" over "is" requires remembering that the subject was plural, eleven words earlier. A plain RNN reliably fails at this, and the failure is a property of the architecture rather than of insufficient training.

9. LSTM and GRU: Adding a Gate

Long Short-Term Memory cells solve the vanishing-gradient problem by adding a separate cell state that information can travel along almost unchanged, plus three learned gates that decide what happens to it:

• Forget gate: how much of the existing cell state to discard.
• Input gate: how much of the newly computed candidate to write in.
• Output gate: how much of the cell state to expose as this step's hidden state.

The important structural difference is that the cell state is updated by addition rather than by repeated matrix multiplication. Addition does not shrink a gradient, so a signal can survive hundreds of steps. This is the same trick as a residual connection in a deep convolutional network, arrived at independently and years earlier.

A GRU merges the forget and input gates into a single update gate and drops the separate cell state, giving roughly three-quarters of the parameters and comparable accuracy on most tasks. The practical advice is unglamorous: try a GRU first because it trains faster, and switch to an LSTM if the task genuinely needs longer memory.

10. Bidirectional and Stacked Variants

A plain RNN reads left to right, so its state at step $t$ knows nothing about what comes later. For many tasks that is an artificial handicap — when classifying the sentiment of a complete review, the end of the sentence is available and useful.

A bidirectional RNN runs two independent recurrences, one forward and one backward, and concatenates their hidden states. Every position then has context from both directions, at twice the parameters and twice the compute. It is the right default for classification and tagging, and it is impossible for genuine real-time prediction, since the backward pass needs the whole sequence up front.

Stacking, which you can switch on with the "Hidden Layers" control above, is the other axis. Two or three layers usually help; beyond that, returns diminish quickly and training difficulty rises. Depth in a recurrent network is far less productive than depth in a convolutional one, because the unrolled network is already extremely deep in the time direction.

11. What Replaced Them, and What Did Not

Since 2017, Transformers have displaced RNNs across most of natural language processing, for one decisive reason: an RNN must process step $t$ before step $t+1$, so training cannot be parallelised along the sequence. A Transformer's attention mechanism looks at all positions simultaneously, which maps directly onto GPU hardware and made training on internet-scale corpora feasible.

Attention also connects any two positions in one step rather than through a chain of $n$ intermediate states, which removes the long-range dependency problem outright instead of mitigating it.

Recurrent models remain preferable in specific circumstances, and they are worth knowing rather than being treated as history:

• Streaming inference. An RNN carries a fixed-size state and processes one step at a time, so cost per token is constant. Attention over a growing context is not.
• Small models and small data. Attention has weaker built-in assumptions and needs more examples to compensate.
• Embedded and low-memory settings, where a Transformer's attention matrix does not fit.
• Very long sequences, where full attention costs $O(n^2)$ and a recurrence costs $O(n)$ — which is exactly why recent state-space models such as Mamba revisit the recurrent formulation.

12. Recap

• An RNN processes a sequence one step at a time, carrying a hidden state that summarises everything seen so far.
• The same weights are applied at every time step, so parameter count is $h(d + h + 1)$ regardless of sequence length.
• Unrolling turns the network into a deep feedforward graph; gradients are summed across steps during backpropagation through time.
• Repeated multiplication by the recurrent matrix causes vanishing or exploding gradients — clip the second, and change architecture for the first.
• LSTMs and GRUs add gates and an additive path for information, which is what makes long-range dependencies learnable.
• Bidirectional layers add future context and rule out streaming use; stacking helps up to about three layers.
• Transformers superseded RNNs mainly because they parallelise across the sequence, but constant-cost streaming keeps recurrence relevant.