Why a new architecture was needed
A plain recurrent network maps a sequence to a sequence of the same length. It tags parts of speech well and it labels named entities well, because those tasks have one output per input.
Translation doesn't. "I don't speak French" is five English words and four French ones. The verb moves. The negation is one word in one language and two in the other. The output length isn't known until the output is made, and the alignment isn't monotonic.
Sutskever, Vinyals and Le's answer in 2014 was to split the problem. One network reads the source and stops, leaving its final hidden state. A second network writes the target, starting from that state, one token at a time, each token conditioned on the ones already emitted. Reading and writing are decoupled, so the lengths need not match.
The training signal is the same one a language model uses: at each decoder step, predict the next target token, with the loss being cross-entropy against the real one. During training the decoder is fed the *true* previous token rather than its own guess. teacher forcing — which makes the grad well behaved and creates the exposure-bias problem, because at inference the decoder must consume its own mistakes.
The bottleneck, measured
The objection is immediate. A fifty-word sentence and a five-word sentence both have to be compressed into the same fixed vector. Where does the extra info go?
Most explanations stop at asserting that it's lost. The explorer measures it.
The encoder is a real recurrent network with fixed weights, so its hidden states are genuine found values. To find out how much a given input token still matters at the end, the widget flips that token's embedding, re-runs the encoder, and measures how far the final state moved. That distance is the bar under each word.
Watch two things:
- Push the sentence length up. The bars at the front of the sentence flatten toward nothing. The statistic reports the ratio of mean sway in the last third against the first third. At length 7 it's close to 1, and by length 40 it's large.
- Push the hidden size down. The same decay happens faster, because there are fewer directions in the state to keep things separate in.
The mechanism isn't mysterious. The state is overwritten at each token, and what an early word contributed has been times'd by the recurrent weight mat once per later token. If that mat contracts — and a stable recurrence often does — then a hundred times is annihilation. LSTM and GRU gates were designed to hold a value against just this, and they push the horizon out a lot. But They don't drop it.
The first paper's own workaround is worth knowing because it's so blunt: they reversed the source sentence. Feeding "sentence the reversed" puts the first source words nearest the end of the encoding, next to the first target words the decoder must give, and this alone raised BLEU by a few points. That a trick like that works is the clearest possible evidence that the bottleneck was real.
Attention, as a small change
Turn Focus on in the explorer and look at what really changed. The encoder is the same. It still reads left to right, still makes one hidden state per token.
The difference is that those per-step states are kept rather than discarded. At each decoder step, the decoder finds a score against each encoder state, softmaxes the scores into weights that sum to 1, and reads a weighted mean — the *context vector* — which it uses alongside its own state.
score_j = f(decoder state, encoder state j)
weight_j = softmax over j of score_j
context = sum over j of weight_j * encoder state j
Nothing has to live to the end of the sentence any more, because nothing has to travel. The channel between the two networks stops being H numbers and turns n×H numbers, growing with the input rather than being fixed against it. The statistic in the explorer reports both.
The focus bars for the current decoder step show which source words that step is reading. The weights sum to 1 by build. So Focus is always a question of *allocation*: attending more to one word always means attending less to another.
What this became
Read the focus equation again with the encoder removed and it's the whole of a transformer. Bahdanau's focus finds a compatibility between one query and a set of keys, softmaxes it, and means the values. Scaled dot-product focus is that with f fixed to a dot product and cut by √dₖ, and self-focus is that with the query coming from the same sequence as the keys.
The 2017 paper's title — "Focus Is All You Need" — is a claim about this page: that once you've focus, the recurrence it was bolted onto isn't carrying its weight. Dropping it makes the whole sequence computable in parallel rather than one step at a time. This Is the change that made scale possible.
import torch
import torch.nn as nn
class Encoder(nn.Module):
def __init__(self, vocab, emb=256, hidden=512):
super().__init__()
self.embed = nn.Embedding(vocab, emb)
self.rnn = nn.GRU(emb, hidden, batch_first=True, bidirectional=True)
def forward(self, src):
# outputs: the per-step states attention needs, [B, T, 2H]
# h: the final state, all the no-attention decoder ever gets, [2, B, H]
outputs, h = self.rnn(self.embed(src))
return outputs, h
class Attention(nn.Module):
def __init__(self, hidden=512):
super().__init__()
self.project = nn.Linear(hidden * 2, hidden)
def forward(self, query, keys, mask):
# query [B, H], keys [B, T, 2H]
scores = torch.bmm(self.project(keys), query.unsqueeze(2)).squeeze(2)
scores = scores.masked_fill(mask == 0, float("-inf")) # ignore padding
weights = torch.softmax(scores, dim=1)
return torch.bmm(weights.unsqueeze(1), keys).squeeze(1), weights
The masked_fill line is the one that is quietly key. Batched sequences are padded to equal length, and without the mask the softmax spreads chance onto padding tokens — the model learns to attend to nothing, and the bug shows up as mysteriously poor translation of short sentences in a batch of long ones.
Teacher forcing and the gap it leaves
One detail of how these models are trained shapes how they fail, and it's invisible in the architecture diagram.
During training the decoder is fed the true previous target token at each step. Feeding it its own previous guess instead would mean that early in training it rules on nonsense, and learning never gets started. Teacher forcing avoids that and makes each step's grad free of the others, which is also what lets the whole target sequence be processed in parallel.
At inference there's no true previous token. The decoder must consume its own output, which means it's operating on a distribution of prefixes it never saw during training — exposure bias. One mistake shifts the context away from any of it familiar, and the errors compound: a model that is 95% accurate per token is far worse than 95% accurate per twenty-token sentence.
The mitigations are all partial. Scheduled sampling mixes in the model's own guesses during training with a chance that rises over time. Sequence-level training optimises a metric like BLEU on made output right, using reinforcement learning because the metric isn't differentiable. And beam search helps more than either, because keeping a few hypotheses alive means one bad token doesn't irrecoverably find the rest — which is the subject of the next module.
What to carry forward
Three ideas from this architecture outlived it fully. Encoder-decoder as a shape, for any task where input and output are both sequences of unrelated length. Teacher forcing and its exposure-bias problem, which is why scheduled sampling and reinforcement-learning fine-tuning exist. And focus, which started as a patch for a fixed vector that was too small and ended up replacing the network it was patching.
Questions people ask
Why not just use a bigger hidden state? Because that moves the horizon without dropping it. The state is overwritten at each token and times'd by the recurrent mat once per token after it. A contracting mat applied a hundred times annihilates what an early word contributed, but many dims it had. Focus drops the need for any of it to live the trip at all.
LSTMs fixed vanishing grads — is the bottleneck not already solved? They're two different problems. Gates let a value persist across many steps and push the usable horizon out a lot. The channel between encoder and decoder is still H numbers no matter how long the input is. Focus changes that channel from H to n×H. That is a change in kind rather than degree.
Why did reversing the source sentence help so much? It puts the first source words nearest the end of the encoding, next to the first target words the decoder has to give. This Shortens the path for the tokens made earliest. That a trick this blunt bought a few BLEU points is the clearest evidence available that the bottleneck was real and severe.
Is focus the same thing as alignment? It behaves like alignment without being trained as it. The weights come from "predict the next token" and nothing else, yet they focus where a translator would point. The difference that matters. Focus is a soft allocation summing to 1. It always points a place, including when no source word matches to the token being emitted.
If teacher forcing causes exposure bias, why use it? Because without it the decoder spends early training conditioned on its own nonsense and never gets started. It also makes each step's grad free of the others. This Is what lets the whole target sequence be processed in parallel. Scheduled sampling and sequence-level training are partial mitigations. beam search helps more than either.
Do I really need the padding mask? Yes. Batches are padded to equal length. Without the mask the softmax spreads chance onto padding tokens. The model learns to attend to nothing. The symptom is short sentences translating badly when they share a batch with long ones. This Is a long way from the cause.
Recap in one screen
- Split reading from writing: one network consumes the source and stops, another emits the target conditioned on what it has already made. The two lengths never have to match.
- The fixed vector is a real bottleneck. It's measurable. Flip one input token, re-run the encoder. See how far the final state moves.
- Focus keeps the per-step encoder states instead of discarding them and reads a weighted mean. The channel grows with the input rather than staying fixed against it.
- The weights sum to 1. This Makes focus a question of allocation. Attending more to one word is attending less to another.
- Teacher forcing during training and self-consumption at inference is exposure bias. It compounds. 95% fit per token is far worse than 95% per sentence.
- Take the recurrence out of the focus equation and what is left is a transformer.