Query, Key and Value

Quick Context

The previous module described attention as score, normalise, blend. That description works, but it leaves one thing vague: what exactly is being compared against what?

The query–key–value framing answers that, and it is worth learning properly because every transformer paper, diagram and implementation is written in it.

A soft dictionary lookup

Think of a Python dictionary. You supply a key, it matches one stored key exactly, and you get back its value. Attention is the same operation with the hard edges removed:

• the query is what you are looking up;
• every entry has a key, which decides how well it matches;
• every entry has a value, which is what you get back.

The difference is that instead of one key matching and the rest failing, every key matches to some degree, and you get back a weighted blend of all the values. A dictionary lookup is attention with one weight set to 1 and the rest to 0.

Where they come from

All three are linear projections of the same input embedding:

Q = X W_Q    K = X W_K    V = X W_V

The three weight matrices are learned. This is the part people find surprising: the same word produces three different vectors, because it plays three different roles. As a query it asks a question; as a key it advertises what it can answer; as a value it supplies content.

And the whole mechanism is one formula:

Attention(Q, K, V) = softmax( QK^T / √d_k ) V

Why keys and values are separate

It is reasonable to ask why we need both — why not match against the values directly and save a matrix?

Because how easily something is found is a different question from what it contains. A word might be highly relevant to a query while the information it should pass on is quite unrelated to why it matched. Splitting the two lets the model learn addressing and content independently, and in practice that separation is what allows different heads to specialise.

Interactive Exploration Guide

1. Follow one lookup end to end. Set the Stage slider to 1 and step it up to 4. You see the raw dot products appear, get divided by the square root of the head dimension, become weights through softmax, and finally blend the value vectors into one output.Run this
2. Change who is asking. Set the Query Word to drank. The scores rearrange completely — the same six keys, a different question, a different answer.Run this
3. Break the scaling. Set the Head Dimension slider to 64 and untick Scale by 1/√d_k. The raw scores grow large, softmax saturates, and the weights collapse onto a single key with entropy near zero. In a real model that means a vanishing gradient and a layer that stops learning.Run this
4. Then fix it. Tick Scale by 1/√d_k again with the dimension still at 64. The scores come back into a sane range and the distribution is usable again. That single division is the entire reason the mechanism is called scaled dot-product attention.
5. Check the low-dimension case. Set the Head Dimension slider to 1. Now the scaling barely matters, because there was never anything to blow up. The correction only earns its keep as dimension grows.Run this

Why divide by √d_k

If the components of the query and key are roughly independent with unit variance, their dot product is a sum of d_k such products, so its variance grows with d_k and its typical magnitude grows with √d_k. At d_k = 64 the scores are around eight times larger than at d_k = 1.

Softmax is exponential, so those larger gaps translate into near-one and near-zero weights. That is bad in itself — the layer stops blending — and worse for training, because softmax's gradient is almost zero once it saturates. Dividing by √d_k cancels the growth exactly and keeps the scores in the range where softmax is still responsive.

The three kinds of attention

The same formula covers every attention in a transformer; only where Q, K and V come from changes:

Cross-attention is the classic encoder–decoder attention of the previous module. When query, key and value all come from the same sequence you get self-attention, which is the next module.

What usually goes wrong

• Assuming Q, K and V are the same vector. They are three different projections of it, with three different learned matrices, and conflating them makes the rest of the architecture impossible to follow.
• Dropping the scaling. It looks like a detail and is not. Without it, wide heads saturate and training stalls.
• Thinking the weights are a similarity between words. They are a similarity between a query projection of one word and a key projection of another. Those projections are learned, so two synonyms need not attend to each other at all.
• Forgetting the last multiply. Softmax gives weights, not output. The output is those weights applied to V, and V is a different projection again.

Key Takeaway

Attention is a soft dictionary lookup: the query says what a position is looking for, each key says how well that position matches, and each value says what it contributes once matched, with all three being separate learned projections of the same input. Every key matches to some degree, so the output is a weighted blend rather than a single retrieved entry, and the whole mechanism is softmax(QKᵀ/√d_k)V. Keys and values are kept apart because findability and content are different things, and the √d_k division exists because dot products grow with dimension and would otherwise saturate the softmax into a hard, ungradient-able argmax.