What are Queries, Keys and Values in an LLM?
Three learned projections of the same token embedding. The query is what this token is looking for; the key is what it advertises to others; the value is what it hands over when attended to. Scores come from query · key; the output is a weighted sum of values.
Overview
Why three and not one
If a token used the same vector to search with and to be found by, attention would collapse into plain similarity: tokens would attend to tokens like themselves. Separating query from key lets a token look for something different from itself — a verb seeking its subject, a pronoun seeking its referent.
Separating value from key matters just as much. What makes a token worth attending to is not necessarily what it should contribute once attended to. The key is the address; the value is the payload.
Parameters
Visualisation
—Readout
What to watch
- All three come from one embedding through three different matrices.
- The score uses K; the output uses V — they are deliberately separate.
- K and V for a past token are fixed forever, which is the cache's premise.
What are Queries, Keys and Values in an LLM?: A Practical Guide
What do Q, K and V actually mean in attention, and why are they three separate things?
The mechanism in one line
softmax(QKT / √d) V. The dot products score every query against every key; the division keeps the softmax out of its saturated region, where gradients vanish; softmax turns scores into weights summing to one; and those weights are applied to the values.
In self-attention all three come from the same sequence. In cross-attention the queries come from one sequence and the keys and values from another — which is how a decoder reads an encoder, and the same shape as retrieval: a query against a set of keys.
Why this is a serving question too
Generating token n needs the keys and values of all previous tokens — and those never change, because each depends only on tokens before it. So they are computed once and kept: the KV cache.
Without it every new token would recompute K and V for the whole prefix, making generation quadratic. With it, each step is linear in the context length. The cost is memory: the cache grows with sequence length × layers × heads, and it is usually what limits how many requests a GPU can serve at once.
Things to try
- Attend from sat. It puts most of its weight on cat, its subject — which a token searching with its own embedding could never do.
- Raise the score scale to 4. The softmax collapses to nearly one-hot; that saturation is what dividing by √d exists to prevent.
- Switch the focus token and watch the whole distribution move. Each token's query is asking a different question of the same keys.
What to remember
Q, K and V are three learned projections of one embedding. The query is what a token looks for, the key is what it advertises, the value is what it contributes. Scores come from Q·K and the output is a weighted sum of V. Because K and V for a token never change once computed, they can be cached — which is what makes generation linear rather than quadratic.