Query Rewriting and HyDE

Two ways to close the gap

Query expansion adds related terms to the original query — synonyms, likely technical vocabulary — widening what it can match against. HyDE (Hypothetical Document Embeddings) goes further: ask a language model to draft a plausible answer to the question, without ever checking whether that answer is true, and embed that hypothetical answer instead of the question. A fabricated but stylistically realistic answer sits much closer in embedding space to a real answer than a terse question ever does — the retrieval step never sees or uses the hypothetical text's truth, only its vocabulary and register.

The vocabulary gap

Users do not phrase questions the way documents phrase answers.

User: "why is my thing not working"

Document: "Troubleshooting connection failures in the client library"

Nothing overlaps. The embedding similarity is low, keyword overlap is zero, and the right document is not retrieved — even though it is the answer.

That mismatch has a name, the vocabulary gap, and it is one of the largest sources of retrieval failure in real systems. Two families of technique address it, both by changing the query rather than the index.

Query rewriting

Use a model to reformulate the question before searching. Several distinct operations hide under the same name:

Expansion. Add synonyms and related terms. "car" becomes "car automobile vehicle".

Clarification. Turn a vague question into a specific one. "why is my thing not working" becomes "why is the client library failing to connect".

Decomposition. Split a multi-part question into parts, retrieve for each, and combine. "How do refunds and exchanges differ?" becomes two searches.

Contextualisation. Resolve references against conversation history. "What about for premium users?" becomes "What is the refund policy for premium users?" — and this one is essential in any chat interface, because a follow-up question is meaningless as a standalone search.

Multi-query. Generate several rewrites, search with each, and fuse the results. Broader recall at the cost of several searches.

prompt = """Rewrite this question as 3 different search queries that would
find relevant documents. Return one per line, no numbering.

Question: {question}"""

queries = model.generate(prompt).strip().split("\n")
results = rrf([index.search(q, k=20) for q in queries])

Contextualisation deserves emphasis because it is the one that is always needed and often forgotten. Without it, the second turn of every conversation retrieves badly.

HyDE: search with a fabricated answer

HyDE — hypothetical document embeddings — takes an unintuitive approach that works surprisingly well.

Instead of embedding the question, ask a model to write a plausible answer, then embed that and search with it.

Question: "why is my thing not working"

Hypothetical answer: "Connection failures in the client library are usually caused by an expired API token, an incorrect endpoint URL, or a firewall blocking outbound requests on port 443. Check the token expiry first..."

That fabricated passage is written in the same register as the documents you are searching — technical, declarative, using the vocabulary answers use. Its embedding sits much closer to real answer passages than the question's embedding did.

The generated answer may be factually wrong, and that does not matter. It is never shown to the user; it is used only as a search key, and what makes it useful is its style and vocabulary, not its accuracy.

Two practical variants: generate several hypothetical answers and average their embeddings, which is more robust; and combine the hypothetical embedding with the original question's rather than replacing it.

Measuring the vocabulary gap, and closing it

Query rewriting and HyDE both attack the same problem -- a short question does not look like the long answer that would satisfy it. This measures that gap on real vectors, applies both fixes, and finds the case where each one makes retrieval worse.

import numpy as np

# 6 concept axes a passage or query can be about
# the first two axes are REGISTER -- how the text is written -- and the
# rest are subject matter. the vocabulary gap lives in the register axes.
AXES = ["colloquial", "formal", "refund", "shipping", "time", "legalese"]

DOCS = {
    "D1 refund policy text": [0.05, 0.75, 0.55, 0.05, 0.35, 0.45],
    "D2 shipping policy text": [0.05, 0.70, 0.05, 0.75, 0.35, 0.40],
    "D3 refund FAQ entry": [0.35, 0.45, 0.65, 0.05, 0.35, 0.10],
    "D4 terms and conditions": [0.02, 0.65, 0.20, 0.15, 0.15, 0.85],
    "D5 chatty forum post": [0.90, 0.05, 0.10, 0.15, 0.10, 0.00],
}
RELEVANT = {"D1 refund policy text", "D3 refund FAQ entry"}

def unit(v):
    v = np.asarray(v, float)
    return v / np.linalg.norm(v)

def sim(a, b):
    return float(unit(a) @ unit(b))

QUERY = [0.85, 0.05, 0.45, 0.00, 0.15, 0.00]     # "can I get my money back"
print("THE QUERY: 'can I get my money back' -- short, informal, and almost")
print("entirely about one axis:")
print("%-26s %s" % ("", "  ".join("%9s" % a for a in AXES)))
print("%-26s %s" % ("the query", "  ".join("%9.2f" % v for v in QUERY)))
print()
print("THE DOCUMENTS, which are long, formal and cover several axes each:")
for name, v in DOCS.items():
    print("%-26s %s" % (name, "  ".join("%9.2f" % x for x in v)))
print()
print("THAT MISMATCH IS THE VOCABULARY GAP. it is not that the query and")
print("the document disagree -- it is that a question is SHORT and")
print("SPECIFIC and an answer is LONG and CONTEXTUAL, so their vectors")
print("point in measurably different directions:")
print("%-30s %14s %s" % ("document", "similarity", "relevant?"))
base = {d: sim(QUERY, v) for d, v in DOCS.items()}
for d in sorted(base, key=lambda x: -base[x]):
    print("%-30s %14.4f %s"
          % (d, base[d], "YES" if d in RELEVANT else ""))
def prec_at(scores, k=2):
    order = sorted(scores, key=lambda x: -scores[x])
    return sum(1 for d in order[:k] if d in RELEVANT) / float(k)
print("   precision@2 = %.4f" % prec_at(base))
print()

print("FIX 1 -- QUERY REWRITING. an LLM rewrites the question into the")
print("vocabulary the corpus uses, before anything is retrieved:")
REWRITES = {
    "original": QUERY,
    "'refund policy'": [0.05, 0.70, 0.70, 0.00, 0.15, 0.15],
    "'refund policy processing time'": [0.05, 0.70, 0.70, 0.00, 0.45, 0.15],
    "'shipping and returns terms'": [0.05, 0.70, 0.25, 0.55, 0.30, 0.35],
}
keys = list(DOCS)
print("%-34s %s %12s"
      % ("query variant", " ".join("%9s" % d.split()[0] for d in keys),
         "precision@2"))
for label, q in REWRITES.items():
    sc = {d: sim(q, v) for d, v in DOCS.items()}
    print("%-34s %s %12.4f"
          % (label, " ".join("%9.4f" % sc[d] for d in keys), prec_at(sc)))
print("   the rewrite drops the colloquial register and adds the words the")
print("   DOCUMENTS use -- 'policy', 'processing time' -- which the user")
print("   had no reason to type. precision@2 goes from %.4f to %.4f, and"
      % (prec_at(base),
         prec_at({d: sim(REWRITES["'refund policy processing time'"], v)
                  for d, v in DOCS.items()})))
print("   D5 drops out of the top 2 entirely.")
last = "'shipping and returns terms'"
sc_last = {d: sim(REWRITES[last], v) for d, v in DOCS.items()}
print("   now the last row. it is a perfectly plausible rewrite -- a model")
print("   asked to formalise 'can I get my money back' might well produce")
print("   it -- and it drifts onto shipping, scoring %.4f on D2 and taking"
      % sc_last["D2 shipping policy text"])
print("   precision back down to %.4f." % prec_at(sc_last))
print("   a rewrite is a GUESS made without seeing the corpus. a wrong")
print("   guess does not fail loudly; it retrieves the wrong thing")
print("   confidently, and nothing downstream can tell.")
print()

print("FIX 2 -- HyDE. instead of rewriting the question, have the model")
print("write a fake ANSWER and search with that. the fake answer does not")
print("have to be true -- it has to look like the document you want:")
HYDE = {
    "answer A: a plausible refund paragraph":
        [0.05, 0.70, 0.60, 0.05, 0.40, 0.30],
    "answer B: the same, but says 90 days (wrong)":
        [0.05, 0.68, 0.62, 0.05, 0.45, 0.28],
    "answer C: drifted onto shipping":
        [0.05, 0.65, 0.15, 0.60, 0.35, 0.30],
}
print("%-46s %10s %12s" % ("search vector", "top hit", "precision@2"))
for label, q in HYDE.items():
    sc = {d: sim(q, v) for d, v in DOCS.items()}
    top = max(sc, key=lambda x: sc[x])
    print("%-46s %10s %12.4f" % (label, top.split()[0], prec_at(sc)))
HY = {lab: {d: sim(q, v) for d, v in DOCS.items()} for lab, q in HYDE.items()}
print("   answer A works because it has the SHAPE of a policy document --")
print("   the same axes, in the same proportions -- even though nobody")
print("   checked whether its content is right.")
print("   answer B contains a WRONG fact (it says 90 days) and retrieves")
print("   just as well, which is the counterintuitive part: HyDE is not")
print("   using the answer's correctness, only its vocabulary and")
print("   register. the hallucination is harmless because the fake answer")
print("   is thrown away after the search.")
print("   answer C is the failure mode. the model drifted onto shipping,")
print("   and the search followed it. HyDE inherits every error the")
print("   generation step makes, and unlike a rewrite -- which is short")
print("   and easy to eyeball -- a fabricated paragraph can go wrong in")
print("   ways nobody will look at.")
print()

print("THE COMPARISON, on this query:")
print("%-30s %14s %14s %s"
      % ("method", "precision@2", "extra LLM calls", "extra latency"))
rows = [
    ("plain retrieval", prec_at(base), 0, "none"),
    ("query rewriting", prec_at({d: sim(REWRITES["'refund policy processing time'"], v)
                                 for d, v in DOCS.items()}), 1, "one short generation"),
    ("HyDE", prec_at(HY["answer A: a plausible refund paragraph"]), 1,
     "one long generation"),
]
for name, p, calls, lat in rows:
    print("%-30s %14.4f %14d %s" % (name, p, calls, lat))
print("   both cost an LLM call BEFORE retrieval, on the critical path of")
print("   every query. HyDE's is longer because it generates a paragraph")
print("   rather than a phrase, and that is the practical difference")
print("   between them more than any quality gap.")
print()

print("WHEN NOT TO USE EITHER:")
print("%-38s %s" % ("the query is already a full sentence", "no gap to close"))
print("%-38s %s" % ("the query is an exact identifier", "rewriting destroys it"))
print("%-38s %s" % ("latency budget is tight", "an LLM call before search"))
print("%-38s %s" % ("the corpus matches user language", "nothing to translate"))
print("   the identifier row is the one that bites. a user searching for")
print("   an order number or an error code wants EXACT matching, and both")
print("   of these methods will happily paraphrase it into something that")
print("   no longer matches anything:")
EXACT = [0.0, 0.0, 0.0, 0.0, 0.0, 1.0]           # a literal code
PARA = [0.10, 0.35, 0.05, 0.05, 0.25, 0.60]      # 'rewritten'
print("      searching for the literal string: similarity to D4 = %.4f"
      % sim(EXACT, DOCS["D4 terms and conditions"]))
print("      after a helpful rewrite:          similarity to D4 = %.4f"
      % sim(PARA, DOCS["D4 terms and conditions"]))
print("   the rewrite looks like an improvement by this measure and is a")
print("   disaster in practice, because the user did not want something")
print("   SIMILAR to their order number.")
print("   the standard answer is to route: detect identifier-like queries")
print("   and send them straight to keyword search, and reserve rewriting")
print("   and HyDE for the natural-language ones where a gap actually")
print("   exists.")

Experiments to try

1. Read the raw query's ranking. A passage about data drift in production beats the overfitting passage — it happens to share more surface words ("model", "worse", "new", "data") with the terse question, even though the overfitting passage is the intended answer.
2. Switch to the expanded query. A handful of added technical terms — "overfitting", "generalization" — appear in the target passage but not the drift one, and that is enough to move the target back to the top.
3. Switch to HyDE. The hypothetical answer is written in the same register as the target passage — full sentences, technical vocabulary, no question words — and it pulls ahead by the widest margin of the three, precisely because it does not resemble the drift passage's phrasing at all.
4. Compare all three similarity numbers. Same underlying question, same target document, three very different embedded-text choices — and three different outcomes for whether the right document is even reachable.

What can go wrong

HyDE embeds a fabrication. If the hypothetical answer is confidently wrong about the topic rather than just stylistically answer-shaped, it can retrieve confidently wrong documents just as easily as right ones — it improves vocabulary match, not correctness. It also costs an extra model call before retrieval even starts, which query expansion usually does not.

The short of it

Retrieval quality is not only a property of the index and the similarity metric — it depends on how close the embedded query text sits to the embedded document text in vocabulary and register. Query expansion narrows that gap cheaply by adding terms; HyDE narrows it further by replacing the question with a fabricated answer written in the target register, at the cost of an extra generation step and the risk of chasing a plausible-sounding but wrong hypothesis.

What each costs

Every one of these techniques adds a model call before retrieval starts.

HyDE is the most expensive because it generates substantially more text. A small fast model is usually adequate for the generation, which brings the cost down considerably.

Three mitigations worth having:

Cache by query text. Repeated and similar questions are common, and the rewrite is deterministic at temperature 0.

Run rewrites in parallel with the original query's search, so the latency is the maximum rather than the sum.

Apply selectively. Short vague queries benefit; long specific ones usually do not. A length or specificity heuristic avoids paying the cost when there is nothing to gain.

When these help, and when they hurt

They help most with: short vague queries, conversational follow-ups, domain jargon the user does not know, and multi-part questions.

They hurt when: the query already contains exact identifiers — rewriting "error E4471" can lose the code, which was the most useful part. Guard by keeping the original query in the fused search alongside any rewrites.

HyDE specifically hurts when the model has no knowledge of the domain and generates a hypothetical answer that is not merely wrong but off-topic, pulling retrieval in an unhelpful direction.

The safe general pattern is additive rather than substitutive: search with the original query and the rewrites, and fuse. That way a bad rewrite dilutes the results slightly rather than replacing them.

Where they sit in a pipeline

A realistic ordering, with each stage added only if it measures better:

1. Contextualise against conversation history — almost always worth it in chat.
2. Self-query to extract metadata filters, if the corpus has metadata.
3. Rewrite or HyDE to close the vocabulary gap.
4. Hybrid retrieval — dense plus BM25 — with the original and rewritten queries.
5. Fuse with reciprocal rank fusion.
6. Rerank with a cross-encoder.
7. Deduplicate or apply MMR.

Measure each addition on a question set with known answers. In most systems hybrid search and reranking give the largest gains, and query transformation matters most where users phrase things very differently from the documents.

Questions people ask

Does HyDE need a large model? No — the hypothetical answer only needs the right register and vocabulary. A small fast model works well.

Does it matter that the hypothetical answer is wrong? Not for retrieval, since it is used only as a search key. It would matter if you showed it to the user, which you should not.

Should I replace the original query? No — search with both and fuse. That way a poor rewrite cannot destroy the results.

How many rewrites for multi-query? Three to five. Beyond that, returns flatten and latency grows.

Is query rewriting better than a better embedding model? A better embedding model helps every query; rewriting helps the badly-phrased ones. Try the model first.

Can I cache these? Yes, and you should — keyed on the query text, at temperature 0.

Recap in one screen

• Users phrase questions differently from how documents phrase answers, and that gap causes retrieval failures.
• Query rewriting expands, clarifies, decomposes or contextualises the question before searching.
• Contextualising against conversation history is essential in chat, and frequently forgotten.
• HyDE generates a plausible answer and searches with its embedding, matching the documents' register.
• Search with the original query as well as the rewrites, and fuse — never substitute blindly.