The Gated Recurrent Unit is the LSTM's streamlined cousin: two gates instead of three, no separate cell state, ~25% fewer parameters — and nearly the same power. Step through a sentence and watch the gates blend old and new.
Overview
Simplify Without Breaking
The Gated Recurrent Unit (Cho et al., 2014) asked a sharp question: does an LSTM really need three gates and two separate memory tracks? The GRU's answer: merge the cell state and hidden state into one vector, merge forget+input into a single update gate z, and add a reset gate r for proposing new content. Same gating idea, leaner machine.
Input Sequence
Update-Gate Experiment
The update gate z blends old and new per dimension. Forcing it to an extreme shows what a “memory dial” it really is. Changing mode resets the run.
Parameter Diet vs LSTM
hidden size H = 128 · embedding E = 128
LSTM · 4(EH + H² + H)
GRU · 3(EH + H² + H)
GRU saves
The GRU Cell
t = 0
UPDATE z̄
–
RESET r̄
–
z̄ near 1 = keep old memory · z̄ near 0 = accept new candidate · r̄ near 0 = ignore the past when proposing it
Hidden state hₜ — the single memory (4 dims)
Gate Timeline
t
Token
z̄
r̄
Hidden state hₜ
Memory starts at h₀ = [0, 0, 0, 0]. Feed the first token.
GRU: The LSTM on a Diet
Two gates, one state, three-quarters of the parameters — and most of the power.
Reset gate r — when proposing the new candidate h̃, decides how much of the old memory to consult. r→0 means “start fresh, the past is irrelevant here.”
Update gate z — the final blend: each dimension of the new state is a weighted average between the old value (weight z) and the new candidate (weight 1−z). One dial does the job of the LSTM's forget and input gates.
The interpolation is the gradient highway. When z sits near 1, hₜ ≈ hₜ₋₁ — the state (and its gradient) passes through the step almost untouched, exactly like the LSTM's conveyor.
LSTM vs GRU at a Glance
LSTM
GRU
Gates
3 (forget, input, output)
2 (update, reset)
Memory tracks
2 (cell state + hidden)
1 (hidden only)
Weight matrices
4 sets
3 sets (~25% fewer params)
When to prefer
very long dependencies, large data
smaller data, faster training
In practice the two trade blows: no consistent winner across tasks. GRUs train faster and overfit less on small datasets; LSTMs sometimes edge ahead when sequences are very long.
Two gates instead of three
A GRU is an LSTM with the design simplified. It merges the forget and input gates into a single update gate, and it drops the separate cell state — there is only the hidden state.
zᵗ = σ(Wᶫ·[hᵗ₋₁, xᵗ]) update gate
rᵗ = σ(Wᵣ·[hᵗ₋₁, xᵗ]) reset gate
h̃ᵗ = tanh(W·[rᵗ ⊙ hᵗ₋₁, xᵗ]) candidate state
hᵗ = (1 − zᵗ) ⊙ hᵗ₋₁ + zᵗ ⊙ h̃ᵗ blend
The last line is the whole idea: the new state is an interpolation between the old state and a candidate, with the update gate choosing the mix. When z is near 0 the state is carried forward unchanged; when near 1 it is replaced.
The reset gate does something subtler: it controls how much of the previous state the candidate is allowed to see. Near 0, the candidate is computed almost from the current input alone — which is how the cell starts fresh at a sentence boundary or a topic change.
Because the two coefficients are z and 1 − z, the GRU cannot both keep the old state and add new information at full strength. An LSTM's separate forget and input gates can. That is the expressiveness the simplification gives up.
GRU or LSTM?
LSTM
GRU
Gates
3 (forget, input, output)
2 (update, reset)
States
Cell and hidden
Hidden only
Parameters
4 weight matrices
3 — about 25% fewer
Speed
Slower
Faster
Long sequences
Slight edge
Slightly behind
Small data
Comparable
Sometimes better — fewer parameters
The honest summary from the literature: they perform comparably on most tasks, with no consistent winner. Comparisons find each ahead on some datasets and behind on others, usually by small margins.
Practical guidance: try the GRU first — fewer parameters, faster, less to overfit — and switch to an LSTM if long-range dependencies are central and you can afford the extra cost. Do not spend a week on the comparison; spend it on the data.
The additive path, again
The GRU keeps the property that made LSTMs work. Because hₜ includes (1 − zₜ) ⊙ hₜ₋₁, the derivative of hₜ with respect to hₜ₋₁ includes the term (1 − zₜ). When the update gate is near 0 — the network has learned to preserve this information — that coefficient is near 1 and the gradient passes back nearly unchanged.
That is the same structural trick as the LSTM's cell state and as a residual connection: an additive path with a derivative near 1, protecting the gradient from multiplicative decay through many steps.
It reduces vanishing gradients rather than eliminating them, and exploding gradients remain possible — so gradient clipping is standard here too.
Exploration guide
Auto-run with learned gates and watch z̄ hover in the middle — the cell constantly negotiates between keeping and rewriting.
Force z → 1. The hidden state freezes at zero no matter what you feed in: with the update gate fully closed to new content, the GRU is a perfect (if useless) memory.
Force z → 0. Now the state is rewritten from scratch at every token — the network becomes almost memoryless. The learned mode is powerful precisely because it sits between these extremes, per dimension, per step.
Drag the hidden-size slider and watch the LSTM/GRU parameter gap widen — at H = 512 the GRU saves over a million weights in a single layer.
The same job with two gates instead of three
A GRU ties the forget and input gates together and drops the cell state entirely. Both simplifications are run side by side against an LSTM here, along with what the GRU consequently cannot express.
example_01.pyNumPy
import numpy as np
def sigmoid(z):
return 1 / (1 + np.exp(-np.clip(z, -30, 30)))
H, X = 4, 3
rng = np.random.default_rng(0)
# --- GRU
Wz = rng.normal(0, 0.5, (X + H, H)); bz = np.zeros(H) # update gate
Wr = rng.normal(0, 0.5, (X + H, H)); br = np.zeros(H) # reset gate
Wh = rng.normal(0, 0.5, (X + H, H)); bh = np.zeros(H) # candidate
def gru_step(x, h):
z = sigmoid(np.concatenate([x, h]) @ Wz + bz)
r = sigmoid(np.concatenate([x, h]) @ Wr + br)
cand = np.tanh(np.concatenate([x, r * h]) @ Wh + bh) # note r * h
return (1 - z) * h + z * cand, (z, r, cand)
print("a GRU has ONE state, not two:")
print(" LSTM: c (memory) and h (what is exposed)")
print(" GRU : h only")
print()
print("and two gates:")
print(" update z -- how much of h to replace")
print(" reset r -- how much of h the candidate is allowed to see")
print()
print("the update rule is the tied one:")
print(" h = (1 - z) * h_old + z * candidate")
print("whatever it forgets, it writes exactly that much back.")
print()
print("parameters, against an LSTM of the same width:")
gru_p = 3 * ((X + H) * H + H)
lstm_p = 4 * ((X + H) * H + H)
print(" GRU : 3 gates x ((%d+%d) x %d + %d) = %d" % (X, H, H, H, gru_p))
print(" LSTM : 4 gates x ((%d+%d) x %d + %d) = %d" % (X, H, H, H, lstm_p))
print(" %.0f%% fewer parameters, and one fewer state to carry."
% (100 * (1 - gru_p / lstm_p)))
print()
print("WHAT THE TYING COSTS. an LSTM's forget and input gates are")
print("independent, so it has four behaviours. a GRU has one dial:")
print("%12s %14s %16s %s" % ("z", "keeps", "writes", "LSTM equivalent"))
for z in (0.0, 0.3, 0.7, 1.0):
print("%12.1f %14.2f %16.2f %s" % (z, 1 - z, z, "f=%.1f, i=%.1f" % (1 - z, z)))
print()
print(" the LSTM can also do:")
print(" f=0.99, i=0.95 -- accumulate: keep everything AND add more")
print(" f=0.05, i=0.01 -- clear: throw it away and store nothing")
print(" a GRU cannot reach either. z would have to be simultaneously")
print(" near 0 and near 1.")
print()
c_lstm, h_gru = 1.0, 1.0
print(" accumulating from 1.0, ten steps, candidate 0.5 each time:")
for t in range(1, 11):
c_lstm = 0.99 * c_lstm + 0.95 * 0.5 # LSTM: f and i both high
h_gru = (1 - 0.95) * h_gru + 0.95 * 0.5 # GRU: z high means forget too
if t in (1, 3, 10):
print(" step %2d: LSTM cell %8.4f GRU state %8.4f" % (t, c_lstm, h_gru))
print(" the LSTM builds a running total. the GRU converges to the")
print(" candidate value and stays there -- it cannot accumulate, because")
print(" writing means forgetting.")
print()
print("THE RESET GATE is the piece with no LSTM counterpart. it controls")
print("what the CANDIDATE is allowed to see:")
print(" candidate = tanh(W @ [x, r * h])")
print("%12s %s" % ("reset r", "what the candidate is computed from"))
for r in (0.0, 0.5, 1.0):
print("%12.1f %s" % (r, "x alone -- the past is ignored" if r == 0 else
("x and %.0f%% of h" % (100 * r) if r < 1 else
"x and the full previous state")))
print(" r near 0 lets a unit start fresh -- useful at a sentence boundary,")
print(" or when the new input makes the accumulated context irrelevant.")
print(" it is a different kind of forgetting from z: z drops the STATE,")
print(" r drops the state's INFLUENCE on what comes next.")
print()
seq = rng.normal(0, 1.0, (8, X))
h = np.zeros(H)
print("run 8 steps. watch z and r move independently:")
print("%6s %12s %12s %14s" % ("step", "mean z", "mean r", "|h| change"))
for t, x in enumerate(seq):
prev = h.copy()
h, (z, r, cand) = gru_step(x, h)
print("%6d %12.4f %12.4f %14.4f"
% (t, z.mean(), r.mean(), np.linalg.norm(h - prev)))
print()
print("WHICH TO USE. the honest answer from a decade of benchmarks:")
print(" they perform about the same on most tasks.")
print(" a GRU is faster and smaller, so it wins when data or compute is")
print(" limited.")
print(" an LSTM's separate cell state and independent gates occasionally")
print(" win on very long sequences and on tasks needing counting or")
print(" accumulation -- exactly the behaviour shown above.")
print(" and in practice the question is now usually moot, because a")
print(" transformer beats both when the sequence fits in a context")
print(" window.")
print()
print("the reason to know both is that the design question they answer --")
print("how much control over memory is worth its cost in parameters --")
print("keeps reappearing, most recently in state-space models.")
Output
Summing up
The GRU keeps the essential insight of gating — memory updates as learned, per-dimension decisions — while cutting a gate, a state track, and a quarter of the parameters. Both LSTM and GRU still read strictly left-to-right, though. What if the meaning of a word depends on what comes after it? That is the next module: bidirectional processing.
Using one
import torch.nn as nn
gru = nn.GRU(input_size=300, hidden_size=256, num_layers=2,
batch_first=True, bidirectional=True, dropout=0.2)
out, h = gru(x) # note: one state returned, not two
# out: (batch, seq, 512) with bidirectional=True
# h: (layers*2, batch, 256)
The interface differs from nn.LSTM in one visible way: a GRU returns a single hidden state rather than a (h, c) tuple, because there is no cell state. Code written for one needs that adjustment.
Everything else transfers: pack variable-length sequences so the final state reflects the last real token rather than padding, clip gradients, and keep the layer count to one or two.
Parameter count is 3 × ((input + hidden) × hidden + hidden) per direction per layer — for 300-dimensional input and 256 hidden units, about 428,000, against roughly 570,000 for the equivalent LSTM.
Where recurrent cells still make sense
Transformers displaced both for large-scale language work, and there are situations where a GRU remains the better engineering choice:
Streaming with bounded memory. A GRU carries one fixed-size state regardless of how much has been processed. A transformer's KV cache grows with every token.
On-device inference. Small, fast, low memory, and no attention kernels needed.
Small datasets. With a few thousand examples, a GRU's inductive bias and small parameter count often beat a transformer trained from scratch.
Classical time series. Sensor data, demand forecasting and control problems, where sequences are numeric and moderate in length.
As a component. GRUs appear inside larger systems as encoders or state trackers.
Note also that the recurrent idea is not finished: state-space models such as Mamba revisit it with linear scaling in sequence length and competitive quality on long sequences.
Questions people ask
Is a GRU always faster than an LSTM? Usually by roughly 20–30% per step, from having one fewer gate and one fewer state.
Which should I choose? GRU first. Switch to LSTM if long-range memory is central and the extra cost is affordable.
Does a GRU have a cell state? No — that is the main structural simplification.
What does the reset gate do that the update gate does not? It controls how much history the candidate sees, which lets the cell effectively restart at a boundary.
Are GRUs obsolete? For large language models, yes. For streaming, embedded and small-data work, no.
Can I stack GRUs deeply? Two layers is the practical limit; deeper recurrent stacks are difficult to train and rarely help.
Recap in one screen
Two gates: update decides how much of the state to replace, reset decides how much history the candidate sees.
One state, not two — the LSTM's cell state is gone.
The new state is an interpolation between old and candidate, which gives the same gradient-preserving additive path.
About 25% fewer parameters than an LSTM and comparable accuracy; try it first.
Still the right choice for streaming, on-device and small-data sequence work.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Simplify Without Breaking”?
The Gated Recurrent Unit (Cho et al., 2014) asked a sharp question: does an LSTM really need three gates and two separate memory tracks? The GRU's answer: merge the cell state and hidden state into one vector, merge forget+input into a single update gate z , and add a reset gate r for proposing new content. Same gating idea, leaner machine.
What does this module say about “How the Two Gates Cooperate”?
What does this module say about “LSTM vs GRU at a Glance”?
A GRU is an LSTM with the design simplified. It merges the forget and input gates into a single update gate , and it drops the separate cell state — there is only the hidden state.
Cheat sheet
What is a GRU?
The Gated Recurrent Unit is the LSTM's streamlined cousin: two gates instead of three, no separate cell state, ~25% fewer parameters — and nearly the same power. Step through a sentence and watch the gates blend old and new.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.