Multi-Head Attention

Quick Context

A single self-attention layer produces one set of weights per word, and those weights must sum to 1. That is a hard constraint: attention spent on one word is attention taken from another.

But a word usually relates to several others in different ways at once. In "the tired cat drank the cold milk", the verb "drank" has a subject, an object, and a tense — three relationships, three different words. One distribution has to compromise between them, and a compromise between three answers is not a good answer to any of them.

The idea

Run several attention operations in parallel. Each gets its own query, key and value projections, so each is free to compare words along a different axis and produce a completely different weighting. Then concatenate their outputs and pass the result through one more linear layer to mix them back together.

MultiHead(Q,K,V) = Concat(head₁, …, head_h) W^O

That final projection matters. Without it the heads' outputs would sit in separate slices of the vector, never interacting; W^O is what lets the layer combine what different heads found.

The part that surprises people

Heads are not stacked on top of the model dimension — the model dimension is divided among them. With d_model = 512 and 8 heads, each head works in 64 dimensions, not 512.

So going from 1 head to 8 does not multiply the cost by eight. The projection matrices are the same total size either way, and the parameter count is unchanged. What changes is the shape of the computation: eight narrow attention patterns instead of one wide one.

This is close to a free lunch, and it is why every transformer uses many heads. The cost is that each head is individually less expressive, which is why very high head counts eventually stop helping — at some point d_k gets too small to represent anything useful.

Interactive Exploration Guide

1. Collapse to one head. Set the Number of Heads slider to 1 and pick "drank" as the Focus Word. A single distribution has to cover subject, object and everything else at once, and Coverage drops.Run this
2. Open them up. Set the Number of Heads slider to 4. The heads separate: one tracks the subject, another the object, another the neighbouring determiner. Distinct Targets rises — the same layer is now representing several relationships simultaneously.Run this
3. Prove the diversity is the point. Tick Make Heads Identical. Every head now shares one projection, so all four rows become the same row, Head Disagreement falls to zero and Distinct Targets drops to 1. Four copies of one opinion are worth exactly one opinion.
4. Watch the width shrink. With Model Dimension at 64, step the Number of Heads slider from 1 to 4 and watch Dimension per Head fall from 64 to 16 while Total Parameters does not move at all. The heads are splitting the budget, not adding to it.Run this
5. Make the heads too narrow. Set the Model Dimension slider to 16 with 4 heads. Each head now has 4 dimensions to work with — barely enough to express a relationship, which is the practical limit on how many heads are worth having.Run this

What heads actually learn

Analyses of trained models have found heads that track syntactic dependencies, heads that attend to the previous token, heads that follow coreference. That is the encouraging half.

The other half is that many heads learn nothing legible. A large fraction attend overwhelmingly to the first token or to punctuation — behaviour usually read as a "no-op", somewhere to park attention when a head has nothing to contribute for this input. Several papers have shown that a majority of heads can be pruned after training with little loss, which suggests the redundancy is real and that heads matter more during training than at inference.

So treat clean per-head interpretations with care. The tidy examples in papers are selected; most heads are not that tidy.

What usually goes wrong

• Assuming more heads means more capacity. It means the same capacity, divided. Adding heads without raising d_model makes each one narrower.
• Choosing a head count that does not divide d_model. The dimension must split evenly; most implementations simply refuse otherwise.
• Forgetting W^O. Concatenation alone leaves the heads' outputs in disjoint slices. The output projection is what merges them.
• Reading one head's pattern as the model's reasoning. The layer's output is a mixture of all heads, then projected. No single head is the answer.

Key Takeaway

Multi-head attention runs several attention operations in parallel, each with its own projections, so a layer can represent several relationships at once instead of compromising between them in a single distribution that must sum to 1. Crucially the model dimension is split across heads rather than multiplied by them, so more heads cost no extra parameters — they buy diversity at the price of making each head narrower, which is why the count cannot be raised indefinitely. The concatenated outputs are mixed by a final projection, and in trained models many heads turn out to be redundant or parked on punctuation, so individual head patterns should be read with caution.