Context Windows and the KV Cache
A model has no memory between calls — only whatever fits in the window you send it. The KV cache is what stops it re-reading all of that from scratch for every single token it writes.
Overview
Quick Context
A language model holds no state between requests. Everything it appears to remember — your name, the file you pasted, what it said three turns ago — is re-sent on every call, inside the context window. The window is a hard limit on how much can be sent, and everything competes for it: the system prompt, retrieved documents, the conversation so far, and the space the reply needs to be written into.
That last one catches people out. The reply is generated into the same window, so a full prompt leaves nowhere to answer.
What You Send
The Serving Side
off means every new token re-reads the whole context from scratch
The Window, And The Work
—The Window
The Cache
Context and the KV Cache: A Practical Guide
Why a chatbot forgets, why long prompts cost so much, and where the memory goes.
What the KV cache is
Generation is one token at a time, and each new token attends to every token before it. Attention needs a key and a value vector for each earlier position — and those never change once computed, because a token's key and value depend only on the tokens up to it.
So the model keeps them. That store is the KV cache. With it, producing token n means computing one new key/value pair and attending over n cached ones. Without it, the model recomputes every key and value from the start for every token, and the total work over a reply of length g grows with the square of the sequence rather than linearly.
cache bytes = 2 × layers × kv_heads × head_dim × tokens × bytes_per_value
The 2 is for K and V. Every term is fixed by the model except the token count, which is why cache memory grows in a straight line with context length — and why long contexts are a memory problem before they are a compute problem.
What the context window actually is
The context window is the maximum number of tokens a model can attend to at once — prompt, retrieved documents, conversation history and generated output, all sharing one budget.
It is measured in tokens, not words. English averages roughly 1.3 tokens per word, so a 128,000-token window is about 96,000 words — and considerably less for code, names or non-Latin scripts.
Two properties surprise people.
It has no memory between calls. A chat interface re-sends the entire conversation with every message. The model does not remember the previous turn; it reads it again. That is why long conversations get slower and more expensive.
Filling it is not free, and not neutral. Attention cost grows with the square of the length, and models attend less reliably to material in the middle of a long context — the documented "lost in the middle" effect. A relevant document at position 40 of 60 may as well not be there.
The KV cache
During generation, each new token attends to every previous token. Recomputing the keys and values for all of them at every step would make generation quadratic in output length.
The keys and values of previous tokens do not change — so they are computed once and cached. Each new token computes only its own K and V, appends them, and attends against the stored ones.
| Without cache | With cache | |
|---|---|---|
| Cost of token n | Recompute all n | Compute 1, read n−1 |
| Total for N tokens | O(N²) work | O(N) work |
| Memory | None extra | 2 tensors per layer per token |
That trade — a large memory cost for linear generation — is what makes interactive generation possible at all, and it is the dominant memory consumer in LLM serving.
The size is calculable:
cache bytes = 2 × layers × heads × head_dim × tokens × bytes_per_value
For a 7B model (32 layers, 32 heads, head dim 128) at 16-bit with 8,000 tokens: 2 × 32 × 32 × 128 × 8000 × 2 bytes ≈ 4.2GB. For one request. Ten concurrent requests exceed most single GPUs before the model weights are counted.
Prefill and decode: two different workloads
Generation has two phases with entirely different characteristics, and understanding the split explains most LLM latency and pricing behaviour.
Prefill processes the prompt. All its tokens are known, so they are processed in parallel in one pass — compute-bound, efficient, fast per token.
Decode produces the output. Each token depends on the previous one, so it is strictly sequential — memory-bandwidth-bound, and it must read the entire cache and the model weights for every single token.
| Prefill | Decode | |
|---|---|---|
| Parallel | Yes, across all prompt tokens | No, one at a time |
| Bottleneck | Compute | Memory bandwidth |
| Cost per token | Low | High |
| Metric | Time to first token | Tokens per second |
That is why input tokens are priced well below output tokens, why a long prompt adds latency before the first word appears, and why generating 500 tokens takes far longer than reading 5,000.
Counting the cache, token by token
The sections above explain what the cache holds and why prefill and decode behave differently. Here is the arithmetic -- how much memory the cache costs at each sequence length, why it and not the weights decides your batch size, and exactly how much work caching saves.
Things to try
- Fill the window. Push Retrieved Documents up until the bar turns red. The overflow readout tells you how much does not fit; in a real system that is where a request either errors or silently drops the oldest turns.
- Watch the reply get squeezed. The reply needs room too. Fill the prompt to within 200 tokens of the window and ask for a 600-token answer: the last part of the answer has nowhere to go.
- Turn the cache off. The count of key/value vectors computed jumps by orders of magnitude, because every reply token rebuilds them for the entire context instead of adding one. This is the difference between a chatbot that streams and one that stalls.
- Grow the context with the cache on. Cache size climbs in a straight line with tokens — at 32 KV heads and fp16, half a megabyte per token, so a full 8k window is about 4 GB of memory per concurrent request.
- Switch to the GQA model. Same size, 8 KV heads instead of 32, and the cache drops to a quarter. This is the entire reason grouped-query attention exists, and why almost every model released since 2023 uses it.
- Try the 70B. More layers multiply the cache again. Context length is cheap to advertise and expensive to serve.
What this explains
- Why long chats get slower and pricier. Every turn re-sends the whole history, so cost grows with the square of the conversation unless the history is trimmed or summarised.
- Why prompt caching exists. If the first few thousand tokens are identical between requests — a system prompt, a document — their keys and values can be computed once and reused, which is what providers sell as prompt or context caching.
- Why batching is awkward. Each concurrent request needs its own cache, so a server's ceiling is usually KV memory rather than compute. Paged attention exists to stop that memory being wasted on fragmentation.
- Why the middle gets ignored. A long context is not uniformly attended: models reliably use the beginning and the end better than the middle. Filling the window is not the same as being understood.
- Why quantised caches are a thing. Storing K and V in 8 bits instead of 16 halves the biggest memory consumer in serving, at some cost in quality.
Worth remembering
The context window is the model's entire working memory, re-sent on every request, and it is shared between the system prompt, retrieved context, the conversation and the reply being written into it. The KV cache turns generation from quadratic work into linear by storing each token's key and value once, and it is paid for in memory that scales with layers, KV heads and sequence length — half a megabyte per token on a 7B model with full multi-head attention, a quarter of that with grouped-query attention. Long contexts are therefore a serving-memory problem first, a cost problem second, and an attention-quality problem third.
Making the cache affordable
Every serious serving optimisation targets this memory.
Grouped-query attention (GQA). Share key and value projections across groups of query heads. With 32 query heads and 8 KV groups the cache is a quarter the size, with quality close to full multi-head. Standard in current models, and adopted specifically for this reason.
Multi-query attention (MQA). One shared KV pair for all heads — the cache shrinks by the head count, with a larger quality cost.
PagedAttention. Allocate the cache in fixed-size blocks like operating-system memory pages rather than one contiguous slab per request. It removes the fragmentation from over-allocating for the maximum possible length, and it allows prefix sharing between requests. This is what vLLM is built around, and it typically multiplies throughput several-fold.
Prefix caching. When many requests share a prefix — the same system prompt, the same retrieved documents — cache its KV once and reuse it. For a RAG system with a long fixed instruction block, this is a substantial saving.
Quantising the cache to 8-bit halves it, with a small accuracy cost.
Sliding-window attention. Attend only to the last N tokens, bounding the cache regardless of conversation length. Used by some models to trade distant context for constant memory.
Managing a long context in practice
Whatever the window size, filling it is usually the wrong instinct. Four practices:
Retrieve rather than stuff. Five relevant chunks beat fifty chunks containing five relevant ones — cheaper, faster, and more accurate because irrelevant context degrades answers.
Put the important material at the ends. Given the middle-of-context weakness, place the most relevant documents first and the question last.
Summarise old conversation turns rather than carrying them verbatim, and keep a running summary plus the last few exchanges.
Measure before assuming a bigger window helps. Recall on a needle-in-a-haystack test degrades well before the stated limit for many models. The advertised number is a capacity, not a guarantee of uniform attention across it.
Questions people ask
Do long context windows make RAG obsolete? No. Retrieval is cheaper, more precise, provides citations, and avoids the attention dilution of a very long prompt.
Why is my second message in a conversation slower? The whole history is re-sent and re-prefilled, so prompt length grows with the conversation.
Why are output tokens more expensive than input? Prefill is parallel and compute-bound; decode is sequential and bandwidth-bound, costing far more per token.
What is time to first token? Prefill latency — how long before generation starts. Dominated by prompt length.
Can I reuse the cache across requests? Yes for a shared prefix, which is exactly what prefix caching does. Not for divergent content.
Does the cache affect output quality? No — it is an exact optimisation. Quantising it can, slightly.
Recap in one screen
- The context window is one shared token budget for prompt, history, documents and output.
- There is no memory between calls; the conversation is re-sent every time.
- The KV cache stores previous tokens' keys and values, turning quadratic generation into linear at a large memory cost.
- Prefill is parallel and cheap per token; decode is sequential and expensive — hence the pricing asymmetry.
- GQA, PagedAttention and prefix caching all exist to shrink or share that cache.