Multi-Query Retriever
A single phrasing is one sample of what the reader meant. Ask the same question three more ways and retrieve for all of them — the union catches what any one wording would have missed.
Overview
The vocabulary mismatch problem
Embedding-based retrieval finds documents whose vectors are near the query’s vector. That works well when the question and the document use similar language, and fails when they do not.
Ask “how do I speed up my model?” of a corpus that discusses “reducing inference latency”, “quantisation” and “batch throughput” and the embedding may simply not land near any of them. The information is present and the phrasing is wrong, and a single query gets a single shot at matching it.
The failure is silent. The retriever returns its nearest neighbours regardless, so you get plausible-looking, unhelpful chunks with no signal that anything was missed.
Which queries run
Recall
Each phrasing, and its own top 2
—The union, against what was actually relevant
Where the variants come from
In a real multi-query retriever an LLM writes the rephrasings. The three below are written by hand as a labelled example of what it would produce — no model runs in this page. Everything after that point, the retrieval and the union, is really computed here.
Multi-Query Retriever: A Practical Guide
One question can be asked many ways, and vector search only finds what is phrased like the query it was given. A multi-query retriever generates several rewordings, searches with each, and merges the results.
Generating several queries and merging
A multi-query retriever asks a language model to produce N alternative phrasings of the question — typically 3 to 5 — each approaching it from a different angle. The original question above might become:
- “What techniques reduce model inference latency?”
- “How can I improve throughput at serving time?”
- “What makes a neural network run faster in production?”
Each is embedded and searched independently, giving N result sets which are then combined and deduplicated. Because the rewordings cover different vocabulary, their nearest neighbours differ, and the union covers substantially more of the relevant material than any one query would.
Merging is usually done with reciprocal rank fusion rather than raw scores. RRF assigns each document a score of Σ 1/(60 + rank) across the lists it appears in, which rewards documents that several rewordings agree on and avoids the problem that similarity scores from different queries are not directly comparable.
Guided experiments
- Compare a single query with the set. Look at what one phrasing retrieves against the union of all of them. The union is broader, and usually includes at least one document no single query found.
- Find the documents only one variant retrieves. These are exactly the recall the technique is buying — material the original phrasing would have missed entirely.
- Look for agreement. Documents retrieved by several variants are the strongest candidates, and that consensus is what rank fusion promotes to the top.
- Watch the precision cost. The merged list is longer and contains more marginal results. Multi-query improves recall and dilutes precision, which is why a reranker usually follows it.
What it costs, and what to use instead
Multi-query is not free. Generating the variants is an extra LLM call on the critical path, adding latency and cost to every request, and it runs N searches instead of one. For an interactive application that overhead is real.
Related approaches make different trades. HyDE generates a hypothetical answer and embeds that, on the argument that an answer is textually closer to the passage containing it than a question is. Query decomposition splits a multi-part question into separate sub-questions, which multi-query does not do — it rephrases rather than divides. And hybrid search, combining dense retrieval with BM25 keyword matching, addresses much of the same vocabulary problem for a fraction of the cost, because exact term matching catches precisely the rare words embeddings handle worst.
Common mistakes
- Generating variants that all say the same thing. If the rewordings are near-identical, they retrieve the same documents and you have paid for nothing. The prompt must push for genuinely different angles and vocabulary.
- Merging by raw similarity score. Scores from different query embeddings are not comparable. Use rank-based fusion.
- Skipping deduplication. The same chunk retrieved by four variants will otherwise occupy four slots of context.
- Not reranking afterwards. The merged list is broader and noisier; a cross-encoder reranker recovers the precision.
- Using it where the problem is chunking. If the relevant text is split badly across chunks, no amount of query rewriting will retrieve it intact.
What to remember
A multi-query retriever compensates for the fact that a single embedding gets one attempt at matching the corpus vocabulary: it generates several rewordings, retrieves with each, and fuses the ranked lists. It buys recall at the cost of an extra LLM call, N searches, and some precision — so it pairs naturally with a reranker. Where the mismatch is about rare or exact terms, hybrid search with BM25 gets much of the same benefit far more cheaply.