Multi-Query Retriever

The vocabulary mismatch problem

Embedding-based retrieval finds documents whose vectors are near the query’s vector. That works well when the question and the document use close language, and fails when they do not.

Ask “how do I speed up my model?” of a corpus that discusses “reducing inference latency”, “quantisation” and “batch throughput” and the embedding may simply not land near any of them. The data is present and the phrasing is wrong. And a single query gets a single shot at matching it.

The failure is silent. The retriever returns its nearest neighbours regardless, so you get plausible-looking, unhelpful chunks with no sign that a thing was missed. A multi-query retriever fixes this by making a few other phrasings of the question. Often 3 to 5. Using a language model. Then searching with each phrasing on its own and merging the results. "How do I speed up my model?" turns "reduce inference latency," "optimise batch throughput," and "quantise model weights". Three queries that together cover vocabulary the first missed. The merge step deduplicates by document id and can either union the results (boost recall) or intersect them (boost precision). The cost is N extra embedding calls and N extra vector searches per user query. This is small compared to the generation step that follows. This page shows the vocabulary mismatch on a real corpus, shows the recall improvement from multiple queries, and covers the tradeoff between union and intersection merging.

Making a few queries and merging

A multi-query retriever asks a language model to give N other phrasings of the question. Often 3 to 5. Each approaching it from a different angle. The first question above might turn:

• “What methods reduce model inference latency?”
• “How can I improve throughput at serving time?”
• “What makes a neural network run faster in production?”

Each is embedded and searched on its own. This gives N result sets which are then combined and deduplicated. Because the rewordings cover different vocabulary, their nearest neighbours differ. And the union covers much more of the apt material than any one query would.

Merging is often done with reciprocal rank fusion rather than raw scores. RRF assigns each document a score of Σ 1/(60 + rank) across the lists it appears in. This rewards documents that a few rewordings agree on and avoids the problem that likeness scores from different queries are not right comparable.

One question, a few searches

A single embedding of a question is one point in the space. If the question is phrased differently from the documents that answer it, that point sits in the wrong place. And no amount of index tuning will fix it.

Multi-query retrieval hedges by making a few rewrites of the question, searching with each, and combining the results.

First: "How do I stop getting so many alerts?"

1. "How to reduce alert volume"

2. "Configuring notification thresholds"

3. "Suppressing copy alerts"

Each rewrite lands in a different region of the embedding space. And their union covers far more of the apt material than any one of them. The third might be the only one that fetches the deduplication documentation.

The method is straightforward:

prompt = """Generate 3 alternative phrasings of this question that would
retrieve relevant documents. One per line, no numbering.

Question: {question}"""

queries = [question] + model.generate(prompt).strip().split("\n")
ranked_lists = [index.search(q, k=20) for q in queries]
results = rrf(ranked_lists)          # fuse by rank

Note that the first question is included. The rewrites are additions, not replacements — a poor rewrite then dilutes the results slightly rather than destroying them.

Why fusion rather than joining

The results from a few searches must be combined, and how matters.

Concatenating and deduplicating loses the ranking data: a document ranked first by two rewrites is treated the same as one ranked twentieth by one.

Averaging scores needs the scores to be comparable across searches, and they are not reliably so.

Reciprocal rank fusion uses only ranks:

score(d) = Σ_queries 1 / (60 + rank(d))

A document fetched highly by a few rewrites builds up from each. So agreement across rewrites turns evidence. That is just the property you want: a document that only one rewrite found is plausible; one that all four found is probably right.

RRF also needs no normalisation, which is what makes it real when combining an arbitrary number of retrievers.

The costs

The searches parallelise, so the added latency is roughly one model call plus one search — not N searches. That makes the technique much cheaper in practice than it looks.

Three or four rewrites is the usual range. Beyond that, the extra rewrites are paraphrases of paraphrases and the returns flatten while the reranking load grows.

Cache aggressively: at temperature 0 the rewrites are deterministic, and repeated or close questions are common in real traffic.

What a few searches recover that one does not

The sections above explain why the pattern makes variants and fuses them rather than concatenating. Here it is run -- three phrasings of one question against one corpus, the union measured against each variant alone. And the case where extra queries make the result worse.

import numpy as np

DOCS = {"D1 delivery": [0.15, 0.05, 0.92, 0.35, 0.02], "D2 shipping": [0.15, 0.92, 0.05, 0.35, 0.02],
        "D3 postage": [0.92, 0.05, 0.05, 0.35, 0.02], "D6 late-fee": [0.12, 0.05, 0.94, 0.02, 0.30],
        "D7 price-list": [0.12, 0.94, 0.05, 0.02, 0.30], "D8 cost-notes": [0.94, 0.05, 0.05, 0.02, 0.30]}
RELEVANT = {"D1 delivery", "D2 shipping", "D3 postage"}

def unit(v):
    v = np.asarray(v, float)
    return v / np.linalg.norm(v)

def rank(q):
    sc = {d: float(unit(q) @ unit(v)) for d, v in DOCS.items()}
    return sorted(sc, key=lambda x: -sc[x]), sc

VARIANTS = {"V1 what-shipping-cost": [0.90, 0.05, 0.05, 0.30, 0.00],
            "V2 delivery-fee": [0.10, 0.05, 0.90, 0.30, 0.00],
            "V3 postage-price": [0.10, 0.90, 0.05, 0.30, 0.00]}
K = 3
print("ONE QUESTION, THREE PHRASINGS -- the corpus uses 'cost', 'fee'")
print("and 'price' in different documents. EACH VARIANT ALONE, top %d:" % K)
for name, q in VARIANTS.items():
    order, sc = rank(q)
    top = order[:K]
    r = len(set(top) & RELEVANT) / float(len(RELEVANT))
    print("%-24s %-32s %10.4f"
          % (name, " | ".join(top), r))
print("   each phrasing retrieves by its own WORD, not by the subject --")
print("   'delivery-fee' finds D1 and D6, and misses D2 entirely. which")
print("   word the user happens to type decides what they see.")

Fusing those three result lists with reciprocal rank fusion recovers what no single phrasing found on its own.

import numpy as np

DOCS = {"D1 delivery": [0.15, 0.05, 0.92, 0.35, 0.02], "D2 shipping": [0.15, 0.92, 0.05, 0.35, 0.02],
        "D3 postage": [0.92, 0.05, 0.05, 0.35, 0.02], "D6 late-fee": [0.12, 0.05, 0.94, 0.02, 0.30],
        "D7 price-list": [0.12, 0.94, 0.05, 0.02, 0.30], "D8 cost-notes": [0.94, 0.05, 0.05, 0.02, 0.30]}
RELEVANT = {"D1 delivery", "D2 shipping", "D3 postage"}

def unit(v):
    v = np.asarray(v, float)
    return v / np.linalg.norm(v)

def rank(q):
    sc = {d: float(unit(q) @ unit(v)) for d, v in DOCS.items()}
    return sorted(sc, key=lambda x: -sc[x]), sc

VARIANTS = {"V1 what-shipping-cost": [0.90, 0.05, 0.05, 0.30, 0.00],
            "V2 delivery-fee": [0.10, 0.05, 0.90, 0.30, 0.00],
            "V3 postage-price": [0.10, 0.90, 0.05, 0.30, 0.00]}
K = 3

def rrf(lists, k=60):
    out = {}
    for lst in lists:
        for i, d in enumerate(lst):
            out[d] = out.get(d, 0.0) + 1.0 / (k + i + 1)
    return out

lists = [rank(q)[0] for q in VARIANTS.values()]
fused = rrf(lists)
order = sorted(fused, key=lambda x: -fused[x])
fused_top = order[:K]
print("NOW FUSE THEM. reciprocal rank fusion over the three result lists:")
print("%-14s %14s %14s" % ("document", "RRF score", "relevant?"))
for d in order:
    print("%-14s %14.6f %14s" % (d, fused[d], "YES" if d in RELEVANT else ""))
print("   top %d after fusion: %s" % (K, " | ".join(fused_top)))
print("   recall of 3 = %.4f"
      % (len(set(fused_top) & RELEVANT) / float(len(RELEVANT))))
print("   every single variant above was still missing at least one --")
print("   that gap is the entire product.")

That still leaves the question of why fusion beats simply concatenating the phrasings into one long query. Averaging vectors turns out to be a worse fix than combining rankings.

import numpy as np

VARIANTS = {"V1 what-shipping-cost": [0.90, 0.05, 0.05, 0.30, 0.00],
            "V2 delivery-fee": [0.10, 0.05, 0.90, 0.30, 0.00],
            "V3 postage-price": [0.10, 0.90, 0.05, 0.30, 0.00]}
DOCS = {"D1 delivery": [0.15, 0.05, 0.92, 0.35, 0.02], "D2 shipping": [0.15, 0.92, 0.05, 0.35, 0.02],
        "D3 postage": [0.92, 0.05, 0.05, 0.35, 0.02], "D6 late-fee": [0.12, 0.05, 0.94, 0.02, 0.30],
        "D7 price-list": [0.12, 0.94, 0.05, 0.02, 0.30], "D8 cost-notes": [0.94, 0.05, 0.05, 0.02, 0.30]}
RELEVANT = {"D1 delivery", "D2 shipping", "D3 postage"}

def unit(v):
    v = np.asarray(v, float)
    return v / np.linalg.norm(v)

def rank(q):
    sc = {d: float(unit(q) @ unit(v)) for d, v in DOCS.items()}
    return sorted(sc, key=lambda x: -sc[x]), sc

K = 3
print("WHY FUSION AND NOT CONCATENATION. suppose you glue the three")
print("phrasings into one long query instead, by averaging their vectors:")
CONCAT = np.mean([np.asarray(v, float) for v in VARIANTS.values()], axis=0)
o_cat, sc_cat = rank(CONCAT)
print("%-16s %-30s %10.4f"
      % ("concatenated", " | ".join(o_cat[:K]),
         len(set(o_cat[:K]) & RELEVANT) / 3.0))
print("   on THIS query it still finds all three -- averaging did not")
print("   break it here. what it DID do is spend the query's whole")
print("   budget pointing weakly at three vocabularies at once:")
for w, i in (("fee-word", 2), ("price-word", 1), ("cost", 0)):
    print("      axis %d highest single variant %.2f, averaged %.2f"
          % (i, max(v[i] for v in VARIANTS.values()), CONCAT[i]))
print("   it survived only because the shipping axis is shared by all")
print("   three phrasings. remove that shared signal and the averaged")
print("   vector has nothing left to discriminate with, while each")
print("   separate variant still matches its own vocabulary. fusion")
print("   combines RANKINGS, so no variant has to compromise;")
print("   concatenation combines VECTORS, and a vector points one way.")

The pattern is not free, and it is not foolproof either. One more call on the critical path. And a bad variant can drag a good result set down with it.

DOCS = {"D1 delivery": [0.15, 0.05, 0.92, 0.35, 0.02], "D2 shipping": [0.15, 0.92, 0.05, 0.35, 0.02],
        "D3 postage": [0.92, 0.05, 0.05, 0.35, 0.02], "D6 late-fee": [0.12, 0.05, 0.94, 0.02, 0.30],
        "D7 price-list": [0.12, 0.94, 0.05, 0.02, 0.30], "D8 cost-notes": [0.94, 0.05, 0.05, 0.02, 0.30]}
RELEVANT = {"D1 delivery", "D2 shipping", "D3 postage"}
def rank(q):
    import numpy as np
    def unit(v):
        v = np.asarray(v, float)
        return v / np.linalg.norm(v)
    sc = {d: float(unit(q) @ unit(v)) for d, v in DOCS.items()}
    return sorted(sc, key=lambda x: -sc[x])
VARIANTS = {"V1 what-shipping-cost": [0.90, 0.05, 0.05, 0.30, 0.00],
            "V2 delivery-fee": [0.10, 0.05, 0.90, 0.30, 0.00],
            "V3 postage-price": [0.10, 0.90, 0.05, 0.30, 0.00]}
def rrf(lists, k=60):
    out = {}
    for lst in lists:
        for i, d in enumerate(lst):
            out[d] = out.get(d, 0.0) + 1.0 / (k + i + 1)
    return out
lists = [rank(q) for q in VARIANTS.values()]
order = sorted(rrf(lists), key=lambda x: -rrf(lists)[x])
print("THE COST: plain retrieval is 0 LLM calls, 1 search. multi-query")
print("with 3 variants is 1 LLM call plus 3 parallel searches -- the")
print("searches run in parallel, but the generation IS on the critical path.")
print()
print("AND WHEN IT MAKES THINGS WORSE. add a variant that drifts:")
lists4 = lists + [rank([0.10, 0.05, 0.10, 0.30, 0.90])]
o4 = sorted(rrf(lists4), key=lambda x: -rrf(lists4)[x])
lost = [d for d in order[:3] if d not in o4[:3]]
gained = [d for d in o4[:3] if d not in order[:3]]
print("   'return shipping' is a plausible model rewrite about a")
print("   different subject. %s dropped out of the top 3 and %s took"
      % (", ".join(lost) or "nothing", ", ".join(gained) or "nothing"))
print("   its place. RRF cannot tell that variant should not have had a")
print("   vote -- every list it is handed counts equally. generate few")
print("   variants, and check them: the failure is silent and looks")
print("   like a retrieval problem rather than a prompt problem.")

Guided tests

1. Compare a single query with the set. Look at what one phrasing fetches against the union of all of them. The union is broader, and often includes at least one document no single query found.
2. Find the documents only one variant fetches. These are just the recall the technique is buying. Material the first phrasing would have missed fully.
3. Look for agreement. Documents fetched by a few variants are the strongest candidates. And that consensus is what rank fusion promotes to the top.
4. Watch the precision cost. The merged list is longer and holds more marginal results. Multi-query improves recall and dilutes precision, which is why a reranker often follows it.

What it costs, and what to use instead

Multi-query is not free. Making the variants is an extra LLM call on the critical path. This adds latency and cost to each request. And it runs N searches instead of one. For an interactive use that cost is real.

Tied approaches make different trades. HyDE makes a hypothetical answer and embeds that, on the input that an answer is textually closer to the passage holding it than a question is. Query decomposition splits a multi-part question into separate sub-questions, which multi-query does not do. It rephrases rather than divides. And hybrid search, combining dense retrieval with BM25 keyword matching, fixes much of the same vocabulary problem for a fraction of the cost,. This is because exact term matching catches just the rare words embeddings handle worst.

Common mistakes

• Making variants that all say the same thing. If the rewordings are near-the same, they retrieve the same documents and you have paid for nothing. The prompt must push for truly different angles and vocabulary.
• Merging by raw likeness score. Scores from different query embeddings are not comparable. Use rank-based fusion.
• Skipping deduplication. The same chunk fetched by four variants will if not take four slots of context.
• Not reranking afterwards. The merged list is broader and noisier; a cross-encoder reranker recovers the precision.
• Using it where the problem is chunking. If the apt text is split badly across chunks, no amount of query rewriting will retrieve it intact.

What to recall

A multi-query retriever compensates for the fact that a single embedding gets one try at matching the corpus vocabulary. It makes a few rewordings, fetches with each, and fuses the ranked lists. It buys recall at the cost of an extra LLM call, N searches, and some precision. So it pairs naturally with a reranker. Where the mismatch is about rare or exact terms, hybrid search with BM25 gets much of the same gain far more cheaply.

Where it helps, and where it does not

It helps with vague or short queries. "It's broken" gives an embedding with almost no sign; a few set rewrites give a few usable ones.

It helps when documents use different vocabulary from users. Jargon, product names, formal register.

It helps with multi-faceted questions, where different aspects live in different documents. Decomposition is a tied and sometimes better approach. Split the question into its parts by name and retrieve for each.

It does not help much when the query now holds exact identifiers. Rewriting "error E4471" risks losing the code. This was the most retrievable part. Including the first query protects against this.

It does not help when recall is now high. If recall@20 is 0.97, there is almost nothing left to find and the extra latency buys nothing. Measure before adding it.

That last point extends: multi-query fixes a recall problem. If your failures are sorting failures. The right document is fetched at rank 15. A reranker is the correct tool, not more queries.

Tied methods, and how they combine

Step-back prompting is worth knowing: for a set question, first ask a broader one ("what are the general principles of X?"), retrieve for both, and fuse. It surfaces background material that a narrow query misses.

Decomposition is the better choice for truly compound questions. "How do refunds differ from exchanges, and what are the deadlines?" is three questions; rewriting it three ways is less effective than splitting it into its actual parts.

These compose, and each adds latency. The sorting to add them in is by measured gain on your own question set. And in most systems hybrid search and reranking come first.

Questions people ask

How many rewrites? Three or four including the first. More flattens out.

Should I include the first query? Yes, always — it protects against a bad rewrite and preserves exact identifiers.

Which model for the rewrites? A small fast one. Paraphrasing a question is not a demanding task.

Does it increase hallucination? No — the rewrites are only search keys and are never shown to the user or to the answering model.

How do I combine the result lists? Reciprocal rank fusion, which needs only ranks.

Is it better than HyDE? They address the same gap differently and can be used together. HyDE matches the documents' register; multi-query covers a few phrasings.

Recap in one screen

• One embedding of a badly-phrased question searches from the wrong point; a few rewrites cover more ground.
• Generate 3–4 phrasings, search with each, and fuse with reciprocal rank fusion.
• Always include the first query, so a poor rewrite cannot destroy the results.
• Agreement across rewrites is evidence, and RRF rewards it by itself.
• It fixes recall problems; sorting problems need a reranker instead.