Multi-Query Retriever

Generating several queries and merging

A multi-query retriever asks a language model to produce N alternative phrasings of the question — typically 3 to 5 — each approaching it from a different angle. The original question above might become:

• “What techniques reduce model inference latency?”
• “How can I improve throughput at serving time?”
• “What makes a neural network run faster in production?”

Each is embedded and searched independently, giving N result sets which are then combined and deduplicated. Because the rewordings cover different vocabulary, their nearest neighbours differ, and the union covers substantially more of the relevant material than any one query would.

Merging is usually done with reciprocal rank fusion rather than raw scores. RRF assigns each document a score of Σ 1/(60 + rank) across the lists it appears in, which rewards documents that several rewordings agree on and avoids the problem that similarity scores from different queries are not directly comparable.

One question, several searches

A single embedding of a question is one point in the space. If the question is phrased differently from the documents that answer it, that point sits in the wrong place, and no amount of index tuning will fix it.

Multi-query retrieval hedges by generating several rewrites of the question, searching with each, and combining the results.

Original: "How do I stop getting so many alerts?"

1. "How to reduce alert volume"

2. "Configuring notification thresholds"

3. "Suppressing duplicate alerts"

Each rewrite lands in a different region of the embedding space, and their union covers far more of the relevant material than any one of them. The third might be the only one that retrieves the deduplication documentation.

The mechanism is straightforward:

prompt = """Generate 3 alternative phrasings of this question that would
retrieve relevant documents. One per line, no numbering.

Question: {question}"""

queries = [question] + model.generate(prompt).strip().split("\n")
ranked_lists = [index.search(q, k=20) for q in queries]
results = rrf(ranked_lists)          # fuse by rank

Note that the original question is included. The rewrites are additions, not replacements — a poor rewrite then dilutes the results slightly rather than destroying them.

Why fusion rather than concatenation

The results from several searches must be combined, and how matters.

Concatenating and deduplicating loses the ranking information: a document ranked first by two rewrites is treated the same as one ranked twentieth by one.

Averaging scores requires the scores to be comparable across searches, and they are not reliably so.

Reciprocal rank fusion uses only ranks:

score(d) = Σ_queries 1 / (60 + rank(d))

A document retrieved highly by several rewrites accumulates from each, so agreement across rewrites becomes evidence. That is exactly the property you want: a document that only one rewrite found is plausible; one that all four found is probably right.

RRF also needs no normalisation, which is what makes it practical when combining an arbitrary number of retrievers.

The costs

The searches parallelise, so the added latency is roughly one model call plus one search — not N searches. That makes the technique considerably cheaper in practice than it looks.

Three or four rewrites is the usual range. Beyond that, the additional rewrites are paraphrases of paraphrases and the returns flatten while the reranking load grows.

Cache aggressively: at temperature 0 the rewrites are deterministic, and repeated or similar questions are common in real traffic.

What several searches recover that one does not

The sections above explain why the pattern generates variants and fuses them rather than concatenating. Here it is run -- three phrasings of one question against one corpus, the union measured against each variant alone, and the case where extra queries make the result worse.

import numpy as np

# 5 axes of phrasing and subject
# 'cost', 'price' and 'fee' are three WORDS for one idea, and the
# shipping axis is the subject. the distractors D6-D8 use the query's
# vocabulary about something else entirely -- which is the thing a
# single phrasing cannot tell apart.
AXES = ["cost-word", "price-word", "fee-word", "shipping", "other"]

DOCS = {
    "D1 delivery charges page": [0.15, 0.05, 0.92, 0.35, 0.02],
    "D2 shipping prices table": [0.15, 0.92, 0.05, 0.35, 0.02],
    "D3 postage cost FAQ": [0.92, 0.05, 0.05, 0.35, 0.02],
    "D6 late-payment fee terms": [0.12, 0.05, 0.94, 0.02, 0.30],
    "D7 software price list": [0.12, 0.94, 0.05, 0.02, 0.30],
    "D8 cost-of-living notes": [0.94, 0.05, 0.05, 0.02, 0.30],
}
RELEVANT = {"D1 delivery charges page", "D2 shipping prices table",
            "D3 postage cost FAQ"}

def unit(v):
    v = np.asarray(v, float)
    return v / np.linalg.norm(v)

def rank(q):
    sc = {d: float(unit(q) @ unit(v)) for d, v in DOCS.items()}
    return sorted(sc, key=lambda x: -sc[x]), sc

VARIANTS = {
    "V1 'what does shipping cost'": [0.90, 0.05, 0.05, 0.30, 0.00],
    "V2 'delivery fee'": [0.10, 0.05, 0.90, 0.30, 0.00],
    "V3 'postage price'": [0.10, 0.90, 0.05, 0.30, 0.00],
}

print("ONE QUESTION, THREE PHRASINGS. the corpus uses all three words --")
print("'cost', 'fee' and 'price' -- in different documents:")
print("%-30s %s" % ("", "  ".join("%11s" % a for a in AXES)))
for name, v in DOCS.items():
    print("%-30s %s" % (name, "  ".join("%11.2f" % x for x in v)))
print()

K = 3
print("EACH VARIANT ALONE, top %d:" % K)
print("%-30s %-38s %12s" % ("query variant", "top %d" % K, "recall of 3"))
per = {}
for name, q in VARIANTS.items():
    order, sc = rank(q)
    top = order[:K]
    per[name] = top
    r = len(set(top) & RELEVANT) / float(len(RELEVANT))
    print("%-30s %-38s %12.4f"
          % (name, " | ".join(t.split()[0] for t in top), r))
print("   each phrasing retrieves by its own WORD, not by the subject.")
print("   'delivery fee' finds the delivery-charges page and the")
print("   late-payment fee terms, and misses the shipping prices table")
print("   entirely -- three of these documents are about shipping and")
print("   three are not, and no single variant separates them.")
print("   which word the user happens to type decides what they see.")
print()

print("NOW FUSE THEM. reciprocal rank fusion over the three result lists:")
def rrf(lists, k=60):
    out = {}
    for lst in lists:
        for i, d in enumerate(lst):
            out[d] = out.get(d, 0.0) + 1.0 / (k + i + 1)
    return out

lists = [rank(q)[0] for q in VARIANTS.values()]
fused = rrf(lists)
order = sorted(fused, key=lambda x: -fused[x])
print("%-32s %14s %14s" % ("document", "RRF score", "relevant?"))
for d in order:
    print("%-32s %14.6f %14s"
          % (d, fused[d], "YES" if d in RELEVANT else ""))
fused_top = order[:K]
print("   top %d after fusion: %s" % (K, " | ".join(t.split()[0] for t in fused_top)))
print("   recall of 3 = %.4f"
      % (len(set(fused_top) & RELEVANT) / float(len(RELEVANT))))
print()
print("   and at a larger k, where the point is clearer:")
print("%-30s %10s %10s %10s" % ("", "k=1", "k=3", "k=5"))
for name, q in VARIANTS.items():
    o, _ = rank(q)
    print("%-30s %10.2f %10.2f %10.2f"
          % (name, *[len(set(o[:kk]) & RELEVANT) / 3.0 for kk in (1, 3, 5)]))
print("%-30s %10.2f %10.2f %10.2f"
      % ("FUSED", *[len(set(order[:kk]) & RELEVANT) / 3.0 for kk in (1, 3, 5)]))
print("   at k=3 the fusion has all three and every single variant is")
print("   still missing at least one. that gap is the entire product.")
print()

print("WHY FUSION AND NOT CONCATENATION. suppose you glue the three")
print("phrasings into one long query instead:")
CONCAT = np.mean([np.asarray(v, float) for v in VARIANTS.values()], axis=0)
o_cat, sc_cat = rank(CONCAT)
print("%-30s %s" % ("the averaged query", "  ".join("%11.2f" % x for x in CONCAT)))
print("%-30s %-38s %12.4f"
      % ("concatenated", " | ".join(t.split()[0] for t in o_cat[:K]),
         len(set(o_cat[:K]) & RELEVANT) / 3.0))
print("%-30s %-38s %12.4f"
      % ("fused", " | ".join(t.split()[0] for t in fused_top),
         len(set(fused_top) & RELEVANT) / 3.0))
print("   concatenating AVERAGES the phrasings into one vector, and the")
print("   average of 'fee', 'price' and 'cost' is a vector that leans")
print("   slightly toward all three and strongly toward none:")
for w, i in (("fee-word", 2), ("price-word", 1), ("cost", 0)):
    print("      %-14s highest single variant %.2f, averaged %.2f"
          % (w, max(v[i] for v in VARIANTS.values()), CONCAT[i]))
print("   on THIS query both approaches find all three, and that is worth")
print("   saying rather than hiding: averaging did not break it here.")
print("   what the averaging DID do is spend the query's whole budget on")
print("   pointing weakly at three vocabularies at once -- 0.90 down to")
print("   0.33 on each -- and it survived only because the shipping axis")
print("   is shared by all three phrasings and is what separates the")
print("   relevant documents from the distractors.")
print("   remove that shared subject signal and the averaged vector has")
print("   nothing left to discriminate with, while each separate variant")
print("   still matches its own vocabulary exactly. that is the case")
print("   fusion is insurance against.")
print("   the general principle: fusion combines RANKINGS, so no variant")
print("   ever has to compromise. concatenation combines VECTORS, and a")
print("   vector can only point one way.")
print()

print("THE COST, which is the reason not to switch it on everywhere:")
print("%-34s %14s %16s %16s"
      % ("", "LLM calls", "retrievals", "latency shape"))
print("%-34s %14d %16d %16s" % ("plain retrieval", 0, 1, "1 search"))
print("%-34s %14d %16d %16s"
      % ("multi-query, 3 variants", 1, len(VARIANTS), "1 gen + 3 parallel"))
print("   the searches are independent, so they run in parallel and cost")
print("   little wall-clock time. the GENERATION is on the critical path")
print("   and cannot be -- so the latency is one LLM call, not three")
print("   searches.")
print()

print("AND WHEN IT MAKES THINGS WORSE. add a variant that drifts:")
DRIFT = {"V4 'return shipping'": [0.10, 0.05, 0.10, 0.30, 0.90]}
lists4 = lists + [rank(list(DRIFT.values())[0])[0]]
f4 = rrf(lists4)
o4 = sorted(f4, key=lambda x: -f4[x])
print("%-34s %-30s %12s" % ("", "top 3", "recall of 3"))
print("%-34s %-30s %12.4f"
      % ("3 good variants", " | ".join(t.split()[0] for t in order[:3]),
         len(set(order[:3]) & RELEVANT) / 3.0))
print("%-34s %-30s %12.4f"
      % ("+ 1 drifted variant", " | ".join(t.split()[0] for t in o4[:3]),
         len(set(o4[:3]) & RELEVANT) / 3.0))
lost = [d for d in order[:3] if d not in o4[:3]]
gained = [d for d in o4[:3] if d not in order[:3]]
print("   'return shipping' is a plausible thing for a model to generate")
print("   from this question, and it is about a different subject.")
print("   %s dropped out of the top 3 and %s took its place."
      % (", ".join(d.split()[0] for d in lost) or "nothing",
         ", ".join(d.split()[0] for d in gained) or "nothing"))
print("   RRF cannot tell that this variant should not have had a vote --")
print("   every list it is handed counts equally.")
print("   so the quality of the pattern is the quality of the generated")
print("   variants, and the generation is unsupervised. generate few, and")
print("   check them: the failure is silent and looks like a retrieval")
print("   problem rather than a prompt problem.")

Guided experiments

1. Compare a single query with the set. Look at what one phrasing retrieves against the union of all of them. The union is broader, and usually includes at least one document no single query found.
2. Find the documents only one variant retrieves. These are exactly the recall the technique is buying — material the original phrasing would have missed entirely.
3. Look for agreement. Documents retrieved by several variants are the strongest candidates, and that consensus is what rank fusion promotes to the top.
4. Watch the precision cost. The merged list is longer and contains more marginal results. Multi-query improves recall and dilutes precision, which is why a reranker usually follows it.

What it costs, and what to use instead

Multi-query is not free. Generating the variants is an extra LLM call on the critical path, adding latency and cost to every request, and it runs N searches instead of one. For an interactive application that overhead is real.

Related approaches make different trades. HyDE generates a hypothetical answer and embeds that, on the argument that an answer is textually closer to the passage containing it than a question is. Query decomposition splits a multi-part question into separate sub-questions, which multi-query does not do — it rephrases rather than divides. And hybrid search, combining dense retrieval with BM25 keyword matching, addresses much of the same vocabulary problem for a fraction of the cost, because exact term matching catches precisely the rare words embeddings handle worst.

Common mistakes

• Generating variants that all say the same thing. If the rewordings are near-identical, they retrieve the same documents and you have paid for nothing. The prompt must push for genuinely different angles and vocabulary.
• Merging by raw similarity score. Scores from different query embeddings are not comparable. Use rank-based fusion.
• Skipping deduplication. The same chunk retrieved by four variants will otherwise occupy four slots of context.
• Not reranking afterwards. The merged list is broader and noisier; a cross-encoder reranker recovers the precision.
• Using it where the problem is chunking. If the relevant text is split badly across chunks, no amount of query rewriting will retrieve it intact.

What to remember

A multi-query retriever compensates for the fact that a single embedding gets one attempt at matching the corpus vocabulary: it generates several rewordings, retrieves with each, and fuses the ranked lists. It buys recall at the cost of an extra LLM call, N searches, and some precision — so it pairs naturally with a reranker. Where the mismatch is about rare or exact terms, hybrid search with BM25 gets much of the same benefit far more cheaply.

Where it helps, and where it does not

It helps with vague or short queries. "It's broken" gives an embedding with almost no signal; several specific rewrites give several usable ones.

It helps when documents use different vocabulary from users — jargon, product names, formal register.

It helps with multi-faceted questions, where different aspects live in different documents. Decomposition is a related and sometimes better approach: split the question into its parts explicitly and retrieve for each.

It does not help much when the query already contains exact identifiers. Rewriting "error E4471" risks losing the code, which was the most retrievable part. Including the original query protects against this.

It does not help when recall is already high. If recall@20 is 0.97, there is almost nothing left to find and the extra latency buys nothing. Measure before adding it.

That last point generalises: multi-query addresses a recall problem. If your failures are ordering failures — the right document is retrieved at rank 15 — a reranker is the correct tool, not more queries.

Related techniques, and how they combine

Step-back prompting is worth knowing: for a specific question, first ask a broader one ("what are the general principles of X?"), retrieve for both, and fuse. It surfaces background material that a narrow query misses.

Decomposition is the better choice for genuinely compound questions. "How do refunds differ from exchanges, and what are the deadlines?" is three questions; rewriting it three ways is less effective than splitting it into its actual parts.

These compose, and each adds latency. The ordering to add them in is by measured benefit on your own question set — and in most systems hybrid search and reranking come first.

Questions people ask

How many rewrites? Three or four including the original. More flattens out.

Should I include the original query? Yes, always — it protects against a bad rewrite and preserves exact identifiers.

Which model for the rewrites? A small fast one. Paraphrasing a question is not a demanding task.

Does it increase hallucination? No — the rewrites are only search keys and are never shown to the user or to the answering model.

How do I combine the result lists? Reciprocal rank fusion, which needs only ranks.

Is it better than HyDE? They address the same gap differently and can be used together. HyDE matches the documents' register; multi-query covers several phrasings.

Recap in one screen

• One embedding of a badly-phrased question searches from the wrong point; several rewrites cover more ground.
• Generate 3–4 phrasings, search with each, and fuse with reciprocal rank fusion.
• Always include the original query, so a poor rewrite cannot destroy the results.
• Agreement across rewrites is evidence, and RRF rewards it automatically.
• It fixes recall problems; ordering problems need a reranker instead.