Query, Key and Value

A soft dictionary lookup

Attention is easiest to read as a lookup that returns a blend rather than one entry.

In an ordinary dictionary you present a key and get back exactly one value. In attention you present a query, compare it against every key, and get back a weighted mixture of all the values — weighted by how well each key matched.

• Query — what this position is looking for.
• Key — the label each position advertises.
• Value — what each position actually contributes when selected.

All three are linear projections of the same input, with separate learned matrices:

Q = XW_Q   K = XW_K   V = XW_V

That is where every parameter in the mechanism lives. Attention itself has no weights — it is a fixed formula applied to these three projections.

Where they come from

All three are linear projections of the same input embedding:

Q = X W_Q    K = X W_K    V = X W_V

The three weight matrices are learned. This is the part people find surprising: the same word produces three different vectors, because it plays three different roles. As a query it asks a question; as a key it advertises what it can answer; as a value it supplies content.

And the whole mechanism is one formula:

Attention(Q, K, V) = softmax( QK^T / √d_k ) V

Why keys and values are separate

It is reasonable to ask why we need both — why not match against the values directly and save a matrix?

Because how easily something is found is a different question from what it contains. A word might be highly relevant to a query while the information it should pass on is quite unrelated to why it matched. Splitting the two lets the model learn addressing and content independently, and in practice that separation is what allows different heads to specialise.

Why three separate projections

The obvious question: if all three come from the same input, why not use the input directly for all of them?

Because the three roles want different information. What a token is looking for is not the same as how it wants to be found, and neither is the same as what it should contribute.

Take "it" in "The cat sat on the mat because it was warm". As a query, "it" needs to search for a nearby noun that could be warm. As a key, "mat" needs to advertise itself as a warmable surface. As a value, "mat" contributes its full semantic content. Separate matrices let the model learn those three functions independently.

The clearest demonstration is that Q and K are used only to compute compatibility, and V only to produce output. A token can be highly attended to for one reason and contribute something quite different — which a single shared projection could not express.

The computation, step by step

Attention(Q, K, V) = softmax(QKᵀ / √d_k) V

For a sequence of 4 tokens with head dimension 64:

Row i of the 4×4 matrix holds token i's attention weights over all four tokens. Multiplying by V replaces token i's representation with the weighted mixture.

The √dₖ division is not cosmetic. Dot products of d-dimensional vectors grow with d, so without scaling the scores reach the tens or hundreds, the softmax becomes nearly one-hot, and the gradient through it approaches zero. Dividing by √dₖ keeps the variance of the scores near 1 regardless of dimension.

Why three projections and not one

Q, K and V are three different linear maps of the same input. Collapsing any two of them breaks something specific, and this shows exactly what.

import numpy as np

rng = np.random.default_rng(0)
words = ["i", "poured", "milk", "into", "the", "cup"]
T, D = len(words), 6
E = rng.normal(0, 1.0, (T, D))

def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def attend(Q, K, V):
    A = softmax(Q @ K.T / np.sqrt(Q.shape[1]))
    return A, A @ V

print("the analogy, stated once and then dropped:")
print("   QUERY  -- what this token is looking for")
print("   KEY    -- what each token offers, for matching purposes")
print("   VALUE  -- what each token actually contributes if attended to")
print()
print("keys and values are separated because 'what makes you findable' and")
print("'what you are worth once found' are different things. a library")
print("catalogue card is the key; the book is the value.")
print()

Wq = rng.normal(0, 0.5, (D, D))
Wk = rng.normal(0, 0.5, (D, D))
Wv = rng.normal(0, 0.5, (D, D))
A_full, out_full = attend(E @ Wq, E @ Wk, E @ Wv)
print("full attention -- three separate projections:")
print("%10s %s" % ("", "".join("%9s" % w for w in words)))
for i, w in enumerate(words):
    print("%10s %s" % (w, "".join("%9.4f" % a for a in A_full[i])))
print()

print("NOW TIE Q AND K TOGETHER (use the same matrix for both):")
A_tied, _ = attend(E @ Wk, E @ Wk, E @ Wv)
print("   the score matrix becomes symmetric:")
print("      max |score[i][j] - score[j][i]| with separate Q,K: %.4f"
      % np.abs((E @ Wq) @ (E @ Wk).T - ((E @ Wq) @ (E @ Wk).T).T).max())
print("      with Q and K tied:                                %.4f"
      % np.abs((E @ Wk) @ (E @ Wk).T - ((E @ Wk) @ (E @ Wk).T).T).max())
print("   so 'milk attends to poured' and 'poured attends to milk' are now")
print("   forced to be equally strong. in the full version they are not:")
i, j = words.index("milk"), words.index("poured")
print("      milk -> poured : %.4f      poured -> milk : %.4f"
      % (A_full[i][j], A_full[j][i]))
print("   language is full of asymmetric relations -- an adjective modifies")
print("   a noun far more than the reverse -- and a symmetric score matrix")
print("   cannot express one.")
print()

print("NOW TIE K AND V TOGETHER:")
A_kv, out_kv = attend(E @ Wq, E @ Wk, E @ Wk)
print("   attention weights are unchanged (%.2e difference) because only V"
      % np.abs(A_kv - A_full).max())
print("   moved -- but the OUTPUT is now forced into the same space that")
print("   matching happens in:")
print("      output[2] with separate V: %s" % np.round(out_full[2][:4], 4))
print("      output[2] with V = K     : %s" % np.round(out_kv[2][:4], 4))
print("   a token now has to be findable and useful in the same encoding.")
print("   there is only one set of weights to satisfy both jobs.")
print()

print("and the dimensions do not all have to match. d_k and d_v are")
print("independent, which is what makes multi-head attention affordable:")
for dk, dv in ((6, 6), (2, 6), (6, 2)):
    Wq2 = rng.normal(0, 0.5, (D, dk))
    Wk2 = rng.normal(0, 0.5, (D, dk))
    Wv2 = rng.normal(0, 0.5, (D, dv))
    A2, o2 = attend(E @ Wq2, E @ Wk2, E @ Wv2)
    print("   d_k=%d, d_v=%d -> scores %s, output %s"
          % (dk, dv, A2.shape, o2.shape))
print("   the score matrix is always T x T whatever d_k is. only the cost of")
print("   COMPUTING it changes, which is why heads use a small d_k each.")
print()

print("cross-attention is the same three lines with Q from one sequence and")
print("K, V from another -- which is how a decoder reads an encoder:")
S = 4
enc = rng.normal(0, 1.0, (S, D))
A_cross, out_cross = attend(E @ Wq, enc @ Wk, enc @ Wv)
print("   %d decoder queries against %d encoder positions -> weights %s"
      % (T, S, A_cross.shape))
print("   each decoder token distributes 1.0 of attention over the encoder:")
print("      row sums:", np.round(A_cross.sum(1), 6))
print("   self-attention is just the case where those two sequences are")
print("   the same one.")

Things to try

1. Follow one lookup end to end. Set the Stage slider to 1 and step it up to 4. You see the raw dot products appear, get divided by the square root of the head dimension, become weights through softmax, and finally blend the value vectors into one output.
2. Change who is asking. Set the Query Word to drank. The scores rearrange completely — the same six keys, a different question, a different answer.
3. Break the scaling. Set the Head Dimension slider to 64 and untick Scale by 1/√d_k. The raw scores grow large, softmax saturates, and the weights collapse onto a single key with entropy near zero. In a real model that means a vanishing gradient and a layer that stops learning.
4. Then fix it. Tick Scale by 1/√d_k again with the dimension still at 64. The scores come back into a sane range and the distribution is usable again. That single division is the entire reason the mechanism is called scaled dot-product attention.
5. Check the low-dimension case. Set the Head Dimension slider to 1. Now the scaling barely matters, because there was never anything to blow up. The correction only earns its keep as dimension grows.

Why divide by √d k

If the components of the query and key are roughly independent with unit variance, their dot product is a sum of d_k such products, so its variance grows with d_k and its typical magnitude grows with √d_k. At d_k = 64 the scores are around eight times larger than at d_k = 1.

Softmax is exponential, so those larger gaps translate into near-one and near-zero weights. That is bad in itself — the layer stops blending — and worse for training, because softmax's gradient is almost zero once it saturates. Dividing by √d_k cancels the growth exactly and keeps the scores in the range where softmax is still responsive.

The three kinds of attention

The same formula covers every attention in a transformer; only where Q, K and V come from changes:

Cross-attention is the classic encoder–decoder attention of the previous module. When query, key and value all come from the same sequence you get self-attention, which is the next module.

Traps worth knowing

• Assuming Q, K and V are the same vector. They are three different projections of it, with three different learned matrices, and conflating them makes the rest of the architecture impossible to follow.
• Dropping the scaling. It looks like a detail and is not. Without it, wide heads saturate and training stalls.
• Thinking the weights are a similarity between words. They are a similarity between a query projection of one word and a key projection of another. Those projections are learned, so two synonyms need not attend to each other at all.
• Forgetting the last multiply. Softmax gives weights, not output. The output is those weights applied to V, and V is a different projection again.

Summing up

Attention is a soft dictionary lookup: the query says what a position is looking for, each key says how well that position matches, and each value says what it contributes once matched, with all three being separate learned projections of the same input. Every key matches to some degree, so the output is a weighted blend rather than a single retrieved entry, and the whole mechanism is softmax(QKᵀ/√d_k)V. Keys and values are kept apart because findability and content are different things, and the √d_k division exists because dot products grow with dimension and would otherwise saturate the softmax into a hard, ungradient-able argmax.

Multiple heads, and where they come from

Multi-head attention runs several of these computations in parallel with different learned projections, then concatenates the outputs and applies one more projection.

import torch, torch.nn.functional as F

def attention(x, Wq, Wk, Wv, mask=None):
    Q, K, V = x @ Wq, x @ Wk, x @ Wv
    scores = Q @ K.transpose(-2, -1) / (K.size(-1) ** 0.5)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float("-inf"))
    return F.softmax(scores, dim=-1) @ V

With a model dimension of 512 and 8 heads, each head projects to 64 dimensions, so the total work is roughly the same as one 512-dimensional head — and the model gets 8 independent views instead of 1.

The masking line is how causal generation and padding are both handled: setting a score to negative infinity makes its softmax weight exactly zero.

Cross-attention, and the caching trick

In self-attention Q, K and V all come from the same sequence. In cross-attention the queries come from one sequence and the keys and values from another — a decoder querying an encoder's output, or a text query attending to image patches.

That asymmetry has a practical consequence during generation, and it is the basis of the most important inference optimisation in language models.

When generating token by token, the keys and values for all previous tokens do not change — only the new token's query is new. So they can be computed once and cached:

Without a KV cache, generating the 1,000th token requires recomputing keys and values for all 999 previous tokens. Generation is quadratic in output length.

With a KV cache, each new token computes its own K and V, appends them, and attends against the stored ones. Generation becomes linear.

The cost is memory: the cache holds two tensors per layer per token, which for a large model and a long context runs to many gigabytes. That is exactly what multi-query and grouped-query attention address, by sharing key and value projections across heads and shrinking the cache several-fold.

Questions people ask

Why is it called query, key, value? By analogy with a database lookup. The analogy is genuinely useful and the mechanism is soft rather than exact.

Are Q, K and V the same size? Q and K must match, since they are dotted together. V can differ in principle and is usually the same size in practice.

Do they have separate weight matrices? Yes — three per head, plus one output projection per layer.

What is dₖ? The dimension of a single head's key vectors, typically 64. It is what the scaling factor is derived from.

Why not use cosine similarity instead of a dot product? The dot product is cheaper, and the scaling plus LayerNorm already keeps magnitudes controlled. Some architectures do normalise the queries and keys.

Can K and V come from a different source than Q? That is precisely cross-attention, and it is how decoders read encoders and how multimodal models connect modalities.

Recap in one screen

• Query is what a position seeks, key is how a position advertises itself, value is what it contributes.
• All three are learned linear projections of the input; attention itself has no parameters.
• Score with QKᵀ, scale by √dₖ, softmax, then weight the values.
• The scaling prevents softmax saturation and vanishing gradients in high dimensions.
• Keys and values do not change during generation, which is what makes KV caching — and linear-time decoding — possible.