The Attention Mechanism
A fixed-size context vector forgets. Attention lets the decoder look back at every input word and weigh them, one output word at a time.
Decoding
inverse temperature on the softmax
force one fixed context vector, as seq2seq did before attention
Alignment
—Line thickness and opacity are the attention weight. They always sum to 1.
Full Alignment Matrix
This Step
Weights
The Attention Mechanism: A Practical Guide
The idea that removed the bottleneck, and then removed recurrence altogether.
Quick Context
The encoder–decoder models that came before attention worked like this: an RNN read the whole input sentence and squeezed it into a single fixed-length vector, and a second RNN generated the output from that vector alone.
The flaw is visible as soon as you say it out loud. Every sentence, whether five words or fifty, has to fit in the same number of floats. The encoder's final state is dominated by whatever it read last, and the beginning of a long sentence is simply gone by the time decoding starts. Translation quality fell off a cliff past about twenty words.
The fix
Attention throws away the assumption that the decoder needs one summary. Instead, keep every encoder state, and at each output step let the decoder build a fresh context vector by weighting them:
- Score. Compare the decoder's current state against every encoder state. One number per input word — how relevant is this word, right now?
- Normalise. Push the scores through softmax so they are positive and sum to 1. These are the attention weights.
- Blend. Take the weighted average of the encoder states. That is the context vector for this step, and it is different for every output word.
Nothing is discarded and nothing is compressed. The decoder reaches back into the full input every single time it produces a word.
Why the weights are interpretable
Because they sum to 1 and are all positive, the weights read as a distribution of interest: "to produce this word, I looked 70% at cat and 20% at the". Plotting them for every output word against every input word gives the alignment matrix in the second panel, and on a translation task it recovers word alignment without ever having been told what alignment is.
That interpretability is genuinely useful, but treat it carefully — attention weights show where information was drawn from, which is not quite the same as showing why the model decided what it decided.
Interactive Exploration Guide
- Walk the output. Set the Output Step slider to 1, then step it up one at a time. The bright connection moves across the source sentence as each output word is produced — the decoder is looking somewhere different every time.
- See the bottleneck it replaced. Tick Bottleneck Mode. Now every output step gets the same flat average of the input, exactly as a fixed context vector would give it. The weights stop moving, focus collapses, and the model has nothing left to distinguish one step from another.
- Sharpen the focus. Set the Sharpness slider to 4. The weights concentrate almost entirely on one source word and the entropy readout falls toward zero — this is a hard alignment.
- Blur it. Set the Sharpness slider to 0.2. The weights spread almost evenly and entropy rises toward its maximum. The context vector becomes a bland average, which is the bottleneck failure again, arrived at from a different direction.
- Watch the whole matrix. Look at the second panel while you change sharpness. The bright diagonal-ish band is the alignment the model has learned; blurring smears it into a uniform grey.
Scoring functions
Step 1 needs a way to compare two vectors. Two became standard:
- Additive (Bahdanau, 2014): push both vectors through a small feed-forward layer. Flexible, works when the two have different sizes, but adds parameters and is slower.
- Dot-product (Luong, 2015): just take the dot product. No parameters at all, and it is a matrix multiplication, which is exactly what accelerators are built for.
Dot-product won, and that decision is the reason transformers are fast. Its one wrinkle — that dot products grow with vector size and push softmax into saturation — is fixed by dividing by the square root of the dimension, the "scaled" in scaled dot-product attention.
What came next
Attention was invented as a patch for RNNs, and for three years that is what it was. Then came the observation that gave the paper Attention Is All You Need its title: if the decoder can reach any input position directly, the recurrence is no longer doing anything essential. Remove it, apply attention to the sequence against itself, and you get self-attention — which trains in parallel across all positions instead of stepping through them one at a time.
The next module generalises the three steps above into the query, key and value framing that every transformer is written in.
What usually goes wrong
- Treating attention weights as explanation. They show where information came from, not why the output was chosen. Different weightings can produce the same prediction.
- Forgetting the cost. Every output position scores against every input position, so the work grows with the product of the two lengths. On long sequences this quadratic term, not the model size, is what runs out of memory.
- Expecting clean alignments everywhere. Attention is often diffuse, and heads in a real model frequently attend to punctuation or to the first token for reasons that have nothing to do with meaning.
Key Takeaway
Attention replaces the single fixed context vector with a fresh weighted average of every encoder state, recomputed at each output step: score the decoder state against each input position, softmax the scores into weights that sum to 1, and blend. Because nothing is compressed, long sentences stop degrading, and because the weights are a proper distribution they can be read as an alignment. Dot-product scoring made it fast enough to be worth doing everywhere, and once the decoder could reach any position directly, recurrence turned out to be optional.