vizlearn
Home / Natural Language Processing

What is a GRU?

The Gated Recurrent Unit is the LSTM's streamlined cousin: two gates instead of three, no separate cell state, ~25% fewer parameters — and nearly the same power. Step through a sentence and watch the gates blend old and new.

Input Sequence

Update-Gate Experiment

The update gate z blends old and new per dimension. Forcing it to an extreme shows what a “memory dial” it really is. Changing mode resets the run.

Parameter Diet vs LSTM

hidden size H = 128 · embedding E = 128

LSTM · 4(EH + H² + H)
GRU · 3(EH + H² + H)
GRU saves

The GRU Cell

t = 0
GRU Cell z update r reset xₜ (input) hₜ hₜ₋₁ loops back one state · two gates · no separate cell state
UPDATE z̄
–
RESET r̄
–

z̄ near 1 = keep old memory · z̄ near 0 = accept new candidate · r̄ near 0 = ignore the past when proposing it

Hidden state hₜ — the single memory (4 dims)

Gate Timeline

tTokenz̄r̄Hidden state hₜ
Memory starts at h₀ = [0, 0, 0, 0]. Feed the first token.

GRU: The LSTM on a Diet

Two gates, one state, three-quarters of the parameters — and most of the power.

Simplify Without Breaking

The Gated Recurrent Unit (Cho et al., 2014) asked a sharp question: does an LSTM really need three gates and two separate memory tracks? The GRU's answer: merge the cell state and hidden state into one vector, merge forget+input into a single update gate z, and add a reset gate r for proposing new content. Same gating idea, leaner machine.

How the Two Gates Cooperate

zₜ = σ(W⃂xₜ + U⃂hₜ₋₁)  ·  rₜ = σ(Wᵣxₜ + Uᵣhₜ₋₁)
h̃ₜ = tanh(W·xₜ + U·(rₜ⊙hₜ₋₁))
hₜ = zₜ⊙hₜ₋₁ + (1 − zₜ)⊙h̃ₜ

  • Reset gate r — when proposing the new candidate h̃, decides how much of the old memory to consult. r→0 means “start fresh, the past is irrelevant here.”
  • Update gate z — the final blend: each dimension of the new state is a weighted average between the old value (weight z) and the new candidate (weight 1−z). One dial does the job of the LSTM's forget and input gates.
  • The interpolation is the gradient highway. When z sits near 1, hₜ ≈ hₜ₋₁ — the state (and its gradient) passes through the step almost untouched, exactly like the LSTM's conveyor.

LSTM vs GRU at a Glance

LSTMGRU
Gates3 (forget, input, output)2 (update, reset)
Memory tracks2 (cell state + hidden)1 (hidden only)
Weight matrices4 sets3 sets (~25% fewer params)
When to prefervery long dependencies, large datasmaller data, faster training

In practice the two trade blows: no consistent winner across tasks. GRUs train faster and overfit less on small datasets; LSTMs sometimes edge ahead when sequences are very long.

Interactive Exploration Guide

  1. Auto-run with learned gates and watch z̄ hover in the middle — the cell constantly negotiates between keeping and rewriting.
  2. Force z → 1. The hidden state freezes at zero no matter what you feed in: with the update gate fully closed to new content, the GRU is a perfect (if useless) memory.
  3. Force z → 0. Now the state is rewritten from scratch at every token — the network becomes almost memoryless. The learned mode is powerful precisely because it sits between these extremes, per dimension, per step.
  4. Drag the hidden-size slider and watch the LSTM/GRU parameter gap widen — at H = 512 the GRU saves over a million weights in a single layer.

Key Takeaway

The GRU keeps the essential insight of gating — memory updates as learned, per-dimension decisions — while cutting a gate, a state track, and a quarter of the parameters. Both LSTM and GRU still read strictly left-to-right, though. What if the meaning of a word depends on what comes after it? That is the next module: bidirectional processing.

Cheat sheet

What is a GRU?

The Gated Recurrent Unit is the LSTM's streamlined cousin: two gates instead of three, no separate cell state, ~25% fewer parameters — and nearly the same power. Step through a sentence and watch the gates blend old and new.

NLP · vizlearn.in/natural_language_processing/what_is_a_gru.html

More in NLP

→ forward · backward ←NLPWhat is a Bidirectional Layer?'A'65NLPASCII Character CodestextNLPHow Neural Networks Process TextNLPHow RNN Processes Text