Re-ranking: Bi-Encoders vs Cross-Encoders
Retrieve fast and approximately with a bi-encoder, then spend more compute reading only the survivors closely with a cross-encoder. Two stages, two very different costs.
Overview
The idea in brief
A bi-encoder embeds the query and every document separately, ahead of time — that is what makes the embeddings precomputable and search fast, but it also means the model never looks at a query and a document together. A cross-encoder takes both as one input and lets the model attend across them jointly, at the cost of running a full forward pass per query-document pair, at query time, with nothing precomputable.
Pipeline
query: "python memory management"
the cross-encoder's judgments below are shown as a labelled example of what jointly reading query+document produces — a real one runs a trained model, which cannot run in this page
Cost
Stage 1 — Bi-Encoder Retrieval (cosine)
—Stage 2 — Final Order
Reading It
Re-ranking: A Practical Guide
Spend the expensive model only on the candidates that survived the cheap one.
The two-stage pattern
Cross-encoders are far more accurate but cost too much to run over an entire corpus — a million documents means a million forward passes per query. The standard fix is two stages: a cheap bi-encoder (or BM25, or both) retrieves a shortlist of maybe 20-100 candidates, and only that shortlist is re-scored by the expensive cross-encoder, which then decides the final order. You pay the cross-encoder's cost dozens of times per query instead of millions of times.
Two ways to score a query against a document
A bi-encoder embeds the query and the document separately, then compares the two vectors. Because documents can be embedded ahead of time, search is a fast vector comparison over a precomputed index.
A cross-encoder takes the query and one document together as a single input and outputs a relevance score. The model attends across both, so every query term can interact with every document term — and nothing can be precomputed.
bi-encoder: score = cos(embed(q), embed(d))
cross-encoder: score = model(q, d)
That structural difference produces the entire trade-off:
| Bi-encoder | Cross-encoder | |
|---|---|---|
| Documents embedded in advance | Yes | No |
| Query cost | One embedding, then vector search | One model pass per document |
| Scales to millions | Yes | No |
| Accuracy | Good | Substantially better |
| Interaction between q and d | None until the comparison | Full attention across both |
Why the two-stage pipeline exists
A cross-encoder over a million documents means a million model passes per query. Impossible.
A bi-encoder over a million documents is one embedding plus an indexed vector search. Milliseconds.
So the standard arrangement uses each for what it is good at:
- Retrieve 50–100 candidates with the bi-encoder (and BM25, fused). Fast, broad, some noise.
- Rerank those candidates with the cross-encoder. Expensive per item, affordable on 50.
- Pass the top 3–10 to the model.
The insight is that recall and precision can be separated. Stage one needs high recall — the right document must be somewhere in the 50. Stage two provides precision, ordering them accurately.
Measured on standard benchmarks, adding a cross-encoder reranker typically improves nDCG by 10–20% over dense retrieval alone. It is one of the highest-value additions to a RAG pipeline, and it is often skipped.
In code
from sentence_transformers import SentenceTransformer, CrossEncoder
# stage 1: bi-encoder, over the whole index
bi = SentenceTransformer("all-MiniLM-L6-v2")
q_vec = bi.encode([query], normalize_embeddings=True)[0]
candidates = top_k_by_dot_product(q_vec, index, k=50)
# stage 2: cross-encoder, over the 50
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, c.text) for c in candidates]
scores = ce.predict(pairs)
ranked = [c for _, c in sorted(zip(scores, candidates), reverse=True)][:5]Note the shapes: the bi-encoder embeds once and compares against a precomputed matrix; the cross-encoder receives 50 (query, document) pairs and returns 50 scores. The second call is where the latency is, and it is batched.
Rough latency for 50 candidates with a small cross-encoder on a GPU: 20–50ms. On a CPU, several hundred milliseconds — which is why reranker size is a real deployment decision.
Why one vector per document is not enough
The code above shows the two-stage API. What it cannot show is the reason the second stage helps at all -- that a bi-encoder commits to one vector per document before it knows the question. This builds both scorers from scratch on a case where that commitment is exactly the problem.
Try it yourself
- Read Stage 1. Pure cosine similarity over word overlap. One candidate about python the snake ranks deceptively high — it shares the words "python" and "memory" with the query, and a bi-encoder scoring query and document independently has no way to notice they mean something different in context.
- Turn on the reranker. The final order changes: the snake document drops sharply, and a genuinely on-topic document that Stage 1 under-ranked moves up. This is what "reading query and document together" catches that comparing two independent vectors cannot.
- Turn it back off. The order reverts to pure Stage 1 — the reranker did not change what was retrieved, only the order the survivors are returned in.
- Read the cost panel. The bi-encoder cost is already paid, before the query ever arrived. The cross-encoder cost is exactly the number of candidates it re-scores — six forward passes, not a million.
Worth remembering
A bi-encoder scores query and document independently, which is what makes it fast enough to search a whole corpus but blind to interactions between them — a lexical coincidence can outrank a genuine match. A cross-encoder reads both together and catches that, at a cost that only scales with the shortlist, not the corpus. Two-stage retrieval — cheap and broad, then expensive and narrow — is how production search gets both speed and accuracy instead of choosing one.
Choosing a reranker
| Option | Character |
|---|---|
| MiniLM cross-encoder | Small, fast, strong baseline |
| Larger cross-encoders | Better, several times slower |
| Multilingual variants | When the corpus is not English |
| ColBERT (late interaction) | Between the two — token-level vectors, precomputable |
| LLM-as-reranker | Prompt a model to score relevance; flexible, expensive |
ColBERT deserves a mention as the genuine middle ground. It stores a vector per token rather than per document, and scores by matching query tokens against document tokens at query time. That gives much of a cross-encoder's fine-grained interaction while keeping the document side precomputable — at the cost of a far larger index.
LLM reranking — asking a language model to rate each candidate's relevance, or to order a list — is flexible and adapts to instructions, and it is slow and expensive enough that it is usually reserved for small candidate sets or offline evaluation.
How many candidates to rerank
The parameter to tune, and it is a straight latency-versus-quality trade.
| Candidates | Effect |
|---|---|
| 10 | Fast, and the reranker can only reorder what stage one already ranked highly |
| 50 | The usual choice — enough room to rescue a mid-ranked document |
| 100–200 | Better recall, noticeably slower |
The number to measure is stage one's recall@n: if recall@50 is 0.95 and recall@100 is 0.96, reranking 100 buys almost nothing. If recall@50 is 0.7, the bottleneck is retrieval, not ranking, and fixing chunking or adding hybrid search matters more than a bigger reranker.
That diagnostic ordering is worth stating plainly: reranking cannot recover a document that was never retrieved.
Questions people ask
Do I need a reranker? If retrieval quality limits your system and latency allows 20–50ms, it is one of the best available improvements.
Can a cross-encoder replace the bi-encoder? Only on a very small corpus. It cannot be indexed.
How many candidates should I rerank? 50 is the standard starting point; tune from measured recall.
Is an LLM a good reranker? Effective and expensive. Use a purpose-trained cross-encoder for production paths.
Does reranking help with the "lost in the middle" problem? Yes — it puts the most relevant chunk first, where the model attends to it most.
Can I fine-tune a reranker? Yes, on (query, relevant, irrelevant) triples from your own data. It often beats a general model by a wide margin on a specific domain.
Recap in one screen
- A bi-encoder embeds query and document separately, so documents can be indexed and search is fast.
- A cross-encoder reads both together, which is far more accurate and cannot be precomputed.
- Use both: broad cheap retrieval for recall, then reranking for precision.
- Rerank around 50 candidates; measure stage-one recall to know whether that is the bottleneck.
- Reranking cannot rescue a document that retrieval never returned.