Self-Attention
Point the sentence at itself. Every word looks at every other word at once — which is how "it" finds out what it refers to.
Sentence
one word changes; watch what "it" attends to
layer 2 queries come from layer 1's output
block every position from seeing the future, as GPT does
self-attention alone cannot tell the difference
Attention Matrix
10 × 10Row = the word doing the looking. Each row sums to 1.
The Focused Row
Resolution
Cost
An RNN needs one step per word to carry information across a sentence. Self-attention needs one, whatever the distance — but pays for it with an n² number of scores.
Self-Attention: A Practical Guide
The step that made recurrence optional, and the two things it cost.
Quick Context
In encoder–decoder attention the queries come from one sequence and the keys and values from another. Self-attention is the same machinery with a single change: all three come from the same sequence. The sentence attends to itself.
That sounds like a small variation. It is the change that produced the transformer.
What a word is actually doing
Every position emits a query, a key and a value. Each query is scored against every key — including its own — giving an n × n grid of scores. Softmax each row, and you have the matrix above: row i is how word i distributes its attention across the whole sentence.
The output for that word is the weighted blend of all the values. So after one self-attention layer, every word's representation has been rewritten as a mixture of the words it found relevant. The word "it" stops being a generic pronoun vector and becomes something closer to whatever it refers to.
The example worth staring at
"The animal didn't cross the street because it was too tired." What does "it" refer to?
Change one word — "because it was too wide" — and the answer flips from the animal to the street. Nothing about the grammar changed. The only way to resolve it is to notice which noun makes sense with the adjective, which means "it" has to be able to look at both nouns and at the adjective, all at once.
An RNN reading left to right has to carry that decision in its hidden state across six words. Self-attention just looks — but not, it turns out, in one go.
One layer cannot do it. In layer 1 each word's query is a fixed function of that word alone, so the query "it" sends out is identical in both sentences. It scores the same against "animal" and against "street" whichever adjective the sentence ends on, and the panel confirms this: at layer 1 the answer is "animal" both times, to three decimal places.
Layer 2 is where it resolves. By then "it" has absorbed a little of everything it attended to in layer 1, including the adjective, so the query it sends the second time is no longer generic — and that is enough to separate the two nouns. Switch to layer 2 and the reference follows the adjective: "tired" keeps it on the animal, "wide" moves it to the street.
This is the real reason transformers are deep. A single attention layer mixes information; it takes another to use what was mixed.
Interactive Exploration Guide
- Watch one layer fail. Set the Layer slider to 1 and the Focus Row to it. Note what it attends to, then switch the Example to the "wide" sentence. The number does not move at all — a single layer's query for "it" cannot depend on a word elsewhere in the sentence.
- Then watch depth fix it. Set the Layer slider to 2 and switch between the two examples again. Now the reference follows the adjective: "tired" holds it on the animal, "wide" moves it to the street. Layer 2's query was built from what layer 1 collected.
- Confirm every row is a distribution. Change the Focus Row to any word and check Row Sum. It is always 1 — softmax runs across each row independently.
- Blur the whole thing. Set the Sharpness slider to 0.2. The matrix goes flat grey: every word attends equally to everything, which is the same as attending to nothing. Set it to 5 and the matrix turns into a near-binary lookup.
- Cut off the future. Tick Causal Mask. The upper triangle goes black — each position can now only see itself and what came before. This single change is the whole architectural difference between BERT and GPT.
- Prove it is order-blind. Tick Shuffle Word Order. The words move, but each word keeps exactly the same attention pattern toward the same partners, because the mechanism never consulted position at all. This is precisely why positional encoding has to exist.
- Read the price. Watch Scores Computed. Ten words need 100 scores; double the sentence and it quadruples. Path Length stays at 1 no matter what — that is the trade self-attention makes.
Why this killed recurrence
| RNN / LSTM | Self-attention | |
|---|---|---|
| Steps between two words | their distance | 1, always |
| Training | sequential | fully parallel |
| Work per layer | grows with n | grows with n² |
| Knows word order | inherently | only if told |
The first two rows are why transformers took over. A recurrent network cannot start step 50 until step 49 is done, so training time scales with sentence length and gradients have to survive fifty multiplications. Self-attention computes every position simultaneously, which is what let models grow to the size they are now.
The last two rows are the bill. The quadratic cost is why context windows were small for years, and the order-blindness is a genuine hole that has to be patched.
What usually goes wrong
- Forgetting positional information. Without it, "dog bites man" and "man bites dog" are the same input. Self-attention is permutation-equivariant by construction.
- Assuming the matrix is symmetric. It is not. Word A attending to B says nothing about B attending to A, because queries and keys are different projections.
- Reading heads as syntax. A trained model's heads often attend to punctuation or dump weight on the first token when they have nothing useful to do. Clean linguistic patterns are the exception, not the rule.
- Underestimating n². Doubling the context quadruples both time and memory for attention. Almost every "efficient transformer" paper is an attack on that one term.
Key Takeaway
Self-attention is attention with queries, keys and values all drawn from the same sequence, so every word is rewritten as a blend of the words it finds relevant — which is how a pronoun can resolve to a noun six words away in a single step. Because every position is computed independently, training parallelises completely and the path between any two words is length 1 regardless of distance, and those two properties are what displaced recurrence. One layer only mixes information, though — resolving a pronoun against a distant adjective takes a second layer to use what the first collected, which is why the architecture is stacked. The costs are real and specific: the number of scores grows with the square of the sequence length, and the mechanism is entirely blind to word order until positional information is added.