Re-ranking: Bi-Encoders vs Cross-Encoders
Retrieve fast and approximately with a bi-encoder, then spend more compute reading only the survivors closely with a cross-encoder. Two stages, two very different costs.
Pipeline
query: "python memory management"
the cross-encoder's judgments below are shown as a labelled example of what jointly reading query+document produces — a real one runs a trained model, which cannot run in this page
Cost
Stage 1 — Bi-Encoder Retrieval (cosine)
—Stage 2 — Final Order
Reading It
Re-ranking: A Practical Guide
Spend the expensive model only on the candidates that survived the cheap one.
Quick Context
A bi-encoder embeds the query and every document separately, ahead of time — that is what makes the embeddings precomputable and search fast, but it also means the model never looks at a query and a document together. A cross-encoder takes both as one input and lets the model attend across them jointly, at the cost of running a full forward pass per query-document pair, at query time, with nothing precomputable.
The two-stage pattern
Cross-encoders are far more accurate but cost too much to run over an entire corpus — a million documents means a million forward passes per query. The standard fix is two stages: a cheap bi-encoder (or BM25, or both) retrieves a shortlist of maybe 20-100 candidates, and only that shortlist is re-scored by the expensive cross-encoder, which then decides the final order. You pay the cross-encoder's cost dozens of times per query instead of millions of times.
Interactive Exploration Guide
- Read Stage 1. Pure cosine similarity over word overlap. One candidate about python the snake ranks deceptively high — it shares the words "python" and "memory" with the query, and a bi-encoder scoring query and document independently has no way to notice they mean something different in context.
- Turn on the reranker. The final order changes: the snake document drops sharply, and a genuinely on-topic document that Stage 1 under-ranked moves up. This is what "reading query and document together" catches that comparing two independent vectors cannot.
- Turn it back off. The order reverts to pure Stage 1 — the reranker did not change what was retrieved, only the order the survivors are returned in.
- Read the cost panel. The bi-encoder cost is already paid, before the query ever arrived. The cross-encoder cost is exactly the number of candidates it re-scores — six forward passes, not a million.
Key Takeaway
A bi-encoder scores query and document independently, which is what makes it fast enough to search a whole corpus but blind to interactions between them — a lexical coincidence can outrank a genuine match. A cross-encoder reads both together and catches that, at a cost that only scales with the shortlist, not the corpus. Two-stage retrieval — cheap and broad, then expensive and narrow — is how production search gets both speed and accuracy instead of choosing one.