Query, embedding and prompt caching
Three different caches with the same name. A query cache stores the finished answer, keyed on the question. An embedding cache stores a vector, keyed on the text. A prompt cache stores the model's internal KV state for a shared prefix. They save different things and go stale in different ways.
Overview
Embedding cache: the easy one
Keyed on a hash of the text and the model name. Embedding is deterministic, so the same text always gives the same vector — which makes this cache both trivially correct and permanently valid, until you change models.
Include the model identifier in the key. Vectors from different models are not comparable, and a cache that silently mixes them produces retrieval results that look plausible and are meaningless. The biggest win is at indexing time: re-indexing a corpus after a chunking tweak re-embeds only the chunks that actually changed.
Parameters
Visualisation
—Readout
What to watch
- Each cache is keyed on something different, so each has a different hit rate.
- An exact repeat hits the query cache and skips everything.
- A reworded question misses both exact caches and can still hit the prompt cache.
Query, embedding and prompt caching: A Practical Guide
What are the different caches in a RAG system, and what does each one actually save?
Query cache: the highest saving, and the highest risk
Keyed on the question, storing the final answer. A hit skips retrieval, reranking and generation — often seconds and most of the cost. Real traffic is heavily repetitive, so hit rates can be high.
Two problems. Staleness: the answer was correct for the corpus as it was, so any document update can invalidate it, and there is no cheap way to know which entries. Most systems use a short TTL and flush on re-index. Exact matching: "what is the refund policy" and "how do refunds work" are one question and two keys. Semantic caching — keying on the embedding and accepting a near match — raises the hit rate and introduces the risk of returning the answer to a subtly different question.
Prompt caching: inside the model
Provider-side, and a different mechanism entirely: the model keeps the attention state (the KV cache) for a prefix it has already processed. Send the same long system prompt and the prefill for that portion is skipped.
It is prefix-based, so it only helps if the shared part comes first. Put the system prompt and any fixed instructions at the front and the retrieved chunks and the question after them; putting the variable part first defeats it entirely. Typical savings are large on time-to-first-token and on input cost, and nothing about correctness changes — the model computes the same thing.
Things to try
- Push exact repeats to 70%. The query cache dominates — and it is the cache that goes stale the moment a document changes.
- Set exact repeats to 0 and reworded to 60%. The exact caches stop helping entirely; only the prompt cache, which keys on the shared prefix, still does.
- Turn prompt caching off. Reworded queries lose their only remaining saving, which is why prompt order matters.
What to remember
Three different caches share the name. The query cache stores a finished answer keyed on the question and saves the most per hit, at the risk of serving a stale one. The embedding cache stores a vector keyed on the text and is permanently valid until the model changes. Prompt caching lives in the provider and only helps when the shared text comes first in the prompt.