Re-ranking: Bi-Encoders vs Cross-Encoders

The idea in brief

A bi-encoder embeds the query and each document apart, ahead of time — that is what makes the embeddings precomputable and search fast. But it too means the net never looks at a query and a document together. A cross-encoder takes both as one input and lets the net attend across them jointly, at the cost of running a full forward pass per query-document pair, at query time, with nothing precomputable. The precision gap between these two approaches is real and well-measured. On MS MARCO and close benchmarks, a cross-encoder always and much beats a bi-encoder by 5-15 points of nDCG,. This is since joint attention can net fine-grained query-document links (negation, qualification, partial match) that free embeddings compress away. The usual pattern is a two-stage pipeline: the bi-encoder (or BM25) fetches a shortlist of 20-100 candidates cheaply. And the cross-encoder rescores just that shortlist to find the final ranking. This page shows both nets scoring the same query-document pairs side by side, shows where the bi-encoder's ranking disagrees with the cross-encoder's, and covers the latency-precision tradeoff as the shortlist size changes.

The two-stage pattern

Cross-encoders are far more right but cost too much to run over an whole corpus — a million documents means a million forward passes per query. The usual fix is two stages: a cheap bi-encoder (or BM25, or both) fetches a shortlist of maybe 20-100 candidates. And just that shortlist is re-scored by the costly cross-encoder. This then decides the final rank. You pay the cross-encoder's cost dozens of times per query instead of millions of times.

Two ways to score a query against a document

A bi-encoder embeds the query and the document apart. Then checks the two vectors. Since documents can be embedded ahead of time, search is a fast vector check over a precomputed index.

A cross-encoder takes the query and one document together as a single input and outputs a fit score. The net attends across both. So each query term can interact with each document term. And nothing can be precomputed.

bi-encoder:   score = cos(embed(q), embed(d))

cross-encoder: score = net(q, d)

That structural gap gives the whole trade-off:

Why the two-stage pipeline exists

A cross-encoder over a million documents means a million net passes per query. Not doable.

A bi-encoder over a million documents is one embedding plus an indexed vector search. Milliseconds.

So the usual setup uses each for what it is good at:

1. Retrieve 50–100 candidates with the bi-encoder (and BM25, fused). Fast, broad, some noise.
2. Rerank those candidates with the cross-encoder. Costly per thing, affordable on 50.
3. Pass the top 3–10 to the net.

The insight is that recall and precision can be separated. Stage one needs high recall — the right document must be somewhere in the 50. Stage two gives precision, sorting them well.

Measured on common benchmarks, adding a cross-encoder reranker much improves nDCG by 10–20% over dense retrieval alone. It is one of the highest-value additions to a RAG pipeline. And it is much skipped.

In code

from sentence_transformers import SentenceTransformer, CrossEncoder

# stage 1: bi-encoder, over the whole index
bi = SentenceTransformer("all-MiniLM-L6-v2")
q_vec = bi.encode([query], normalize_embeddings=True)[0]
candidates = top_k_by_dot_product(q_vec, index, k=50)

# stage 2: cross-encoder, over the 50
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, c.text) for c in candidates]
scores = ce.predict(pairs)

ranked = [c for _, c in sorted(zip(scores, candidates), reverse=True)][:5]

Note the shapes: the bi-encoder embeds once and checks against a precomputed matrix. The cross-encoder receives 50 (query, document) pairs and returns 50 scores. The second call is where the latency is, and it is batched.

Rough latency for 50 candidates with a small cross-encoder on a GPU: 20–50ms. On a CPU, a few hundred milliseconds — which is why reranker size is a real deployment decision.

Why one vector per document is not enough

The code up shows the two-stage API. What it cannot show is the reason the second stage helps at all -- that a bi-encoder commits to one vector per document before it knows the question. This builds both scorers from scratch on a case where that commitment is just the problem.

import numpy as np

DOCS = {
    "D1 full review": np.array([0.90, 0.90, 0.90, 0.90]),
    "D2 battery deep-dive": np.array([0.05, 1.00, 0.05, 0.00]),
    "D3 camera deep-dive": np.array([0.05, 0.00, 1.00, 0.00]),
    "D4 pricing page": np.array([1.00, 0.00, 0.00, 0.00]),
    "D5 one battery sentence": np.array([0.00, 0.35, 0.00, 0.00]),
}
QUERIES = {
    "how long does the battery last": np.array([0.0, 1.0, 0.0, 0.0]),
    "how good is the camera": np.array([0.0, 0.0, 1.0, 0.0]),
}

def unit(v):
    return v / (np.linalg.norm(v) + 1e-12)

print("A bi-encoder stores one UNIT VECTOR per document, no matter how")
print("much the document actually says:")
for name, v in DOCS.items():
    print("%-26s raw sum=%.2f  stored=%s"
          % (name, v.sum(), "  ".join("%.2f" % x for x in unit(v))))

bi = {n: [float(unit(v) @ unit(q)) for q in QUERIES.values()] for n, v in DOCS.items()}
print()
print("Battery-query score, sorted:")
for name in sorted(DOCS, key=lambda d: -bi[d][0]):
    print("   %-26s %.4f" % (name, bi[name][0]))
print("D5 -- a single sentence mentioning the battery -- scores highest,")
print("ahead of the deep-dive and the full review. Normalising gives a")
print("thin document nothing else to spread its budget across, so it")
print("looks MORE on-topic than a thorough document that covers four")
print("things: direction is all cosine similarity sees.")

A cross-encoder never builds that per-document vector at all. So it never has this normalisation problem in the first place.

import numpy as np

DOCS = {
    "D1 full review": np.array([0.90, 0.90, 0.90, 0.90]),
    "D2 battery deep-dive": np.array([0.05, 1.00, 0.05, 0.00]),
    "D5 one battery sentence": np.array([0.00, 0.35, 0.00, 0.00]),
}
query = np.array([0.0, 1.0, 0.0, 0.0])  # "how long does the battery last"

def cross(q, d):
    # attend to the dimensions the query cares about, then read the
    # document's content there -- something no fixed vector can do
    attn = q / (q.sum() + 1e-12)
    return float((attn * d).sum())

print("A cross-encoder scores the PAIR jointly, reading only the part")
print("of the document the query asked about:")
ce = {name: cross(query, v) for name, v in DOCS.items()}
for name in sorted(DOCS, key=lambda d: -ce[d]):
    print("   %-26s %.4f" % (name, ce[name]))
print()
print("Now the order is by CONTENT: D5 drops to what it actually says")
print("about batteries, and the full review rises. The cross-encoder")
print("never normalised anything, because it never built a document")
print("vector to normalise -- it read the query, picked the dimensions")
print("asked about, and read the document there.")

Put both scorers over both queries and the disagreement shows up right at the top of the ranking, not just more down.

import numpy as np

DOCS = {
    "D1 full review": np.array([0.90, 0.90, 0.90, 0.90]),
    "D2 battery deep-dive": np.array([0.05, 1.00, 0.05, 0.00]),
    "D3 camera deep-dive": np.array([0.05, 0.00, 1.00, 0.00]),
    "D4 pricing page": np.array([1.00, 0.00, 0.00, 0.00]),
    "D5 one battery sentence": np.array([0.00, 0.35, 0.00, 0.00]),
}
QUERIES = {
    "how long does the battery last": np.array([0.0, 1.0, 0.0, 0.0]),
    "how good is the camera": np.array([0.0, 0.0, 1.0, 0.0]),
}

def unit(v):
    return v / (np.linalg.norm(v) + 1e-12)

def cross(q, d):
    attn = q / (q.sum() + 1e-12)
    return float((attn * d).sum())

bi = {n: [float(unit(v) @ unit(q)) for q in QUERIES.values()] for n, v in DOCS.items()}
ce = {n: [cross(q, v) for q in QUERIES.values()] for n, v in DOCS.items()}

print("THE RANKINGS THAT PRODUCES:")
for i, q in enumerate(QUERIES):
    print("   query: %s" % q)
    b = sorted(DOCS, key=lambda d: -bi[d][i])
    c = sorted(DOCS, key=lambda d: -ce[d][i])
    print("      bi-encoder:    %s" % "  >  ".join(x.split()[0] for x in b))
    print("      cross-encoder: %s" % "  >  ".join(x.split()[0] for x in c))
print("They disagree at the TOP, not just below it: short, narrowly")
print("worded chunks punch above their weight in a vector index, and")
print("reranking is what puts them back in their place.")

That precision is just why cross-encoders are not used alone. The next block shows what running one over a whole corpus would cost, and why the two-stage split exists.

N = 1_000_000

print("SO WHY NOT USE THE CROSS-ENCODER FOR EVERYTHING?")
print("%-28s %26s %26s" % ("", "bi-encoder", "cross-encoder"))
for label, b, c in (
        ("document vectors stored", "{:,}".format(N), "0"),
        ("work done offline", "{:,} encodes".format(N), "none possible"),
        ("work done per query", "1 encode + a dot product",
         "{:,} forward passes".format(N)),
        ("can use an ANN index", "yes", "no -- no vectors to index")):
    print("%-28s %26s %26s" % (label, b, c))
print("An ANN index compares PRECOMPUTED vectors, and a cross-encoder")
print("has none -- its score only exists once a query and a document")
print("are put together, so it must look at every document, every time.")
print()

CE_MS_PER_DOC = 1.0
print("HENCE THE TWO STAGES, for a %s-document corpus:" % "{:,}".format(N))
print("%-40s %16s %14s" % ("design", "cross-encoder calls", "latency"))
print("%-40s %16s %14s"
      % ("cross-encoder on everything", "{:,}".format(N),
         "%.0f minutes" % (N * CE_MS_PER_DOC / 1000 / 60)))
for k in (10, 50, 200):
    print("%-40s %16d %14s"
          % ("bi-encoder, then rerank top %d" % k, k,
             "%.0f ms" % (k * CE_MS_PER_DOC + 5)))
print("Rescoring 50 candidates instead of a million is %s times cheaper"
      % "{:,}".format(N // 50))
print("and still sees every document with a serious chance. The bi-encoder's")
print("job is to be CHEAP and lose nothing, which is why it is judged on")
print("recall@k, never precision@1 -- and why it sets the ceiling: anything")
print("stage one misses is invisible to the reranker, however good the")
print("reranker is. Tune k by watching recall@k of stage one.")

Try it yourself

1. Read Stage 1. Pure cosine likeness over word overlap. One candidate about python the snake ranks deceptively high — it shares the words "python" and "memory" with the query. And a bi-encoder scoring query and document on its own has no way to notice they mean a thing else in context.
2. Turn on the reranker. The final rank changes: the snake document drops sharply. And a truly on-topic document that Stage 1 under-ranked moves up. This is what "reading query and document together" catches that checking two free vectors cannot.
3. Turn it back off. The rank reverts to pure Stage 1 — the reranker did not change what was fetched, just the rank the survivors are gave in.
4. Read the cost panel. The bi-encoder cost is now paid, before the query ever arrived. The cross-encoder cost is just the count of candidates it re-scores — six forward passes, not a million.

Worth recalling

A bi-encoder scores query and document on its own, which is what makes it fast enough to search a whole corpus but blind to links between them — a lexical coincidence can outrank a genuine match. A cross-encoder reads both together and catches that, at a cost that just scales with the shortlist, not the corpus. Two-stage retrieval — cheap and broad, then costly and narrow — is how live search gets both speed and precision instead of choosing one.

Choosing a reranker

ColBERT deserves a mention as the genuine middle ground. It stores a vector per token rather than per document, and scores by matching query tokens against document tokens at query time. That gives much of a cross-encoder's fine-grained link while keeping the document side precomputable. At the cost of a far more index.

LLM reranking — asking a language net to rate each candidate's fit, or to rank a list — is flexible and adapts to cues. And it is slow and costly enough that it is much reserved for small candidate sets or offline evaluation.

How many candidates to rerank

The parameter to tune, and it is a straight latency-versus-grade trade.

The count to measure is stage one's recall@n: if recall@50 is 0.95 and recall@100 is 0.96, reranking 100 buys almost nothing. If recall@50 is 0.7, the bottleneck is retrieval, not ranking, and fixing chunking or adding hybrid search matters more than a bigger reranker.

That diagnostic sorting is worth stating plainly: reranking cannot recover a document that was never fetched.

Questions people ask

Do I need a reranker? If retrieval grade limits your system and latency lets 20–50ms, it is one of the best available improvements.

Can a cross-encoder replace the bi-encoder? Just on a quite small corpus. It cannot be indexed.

How many candidates should I rerank? 50 is the usual starting point. Tune from measured recall.

Is an LLM a good reranker? Effective and costly. Use a purpose-trained cross-encoder for live paths.

Does reranking help with the "lost in the middle" problem? Yes. It puts the most apt chunk first, where the net attends to it most.

Can I fine-tune a reranker? Yes, on (query, apt, irrelevant) triples from your own data. It much beats a general net by a wide margin on a set domain.

Recap in one screen

• A bi-encoder embeds query and document apart, so documents can be indexed and search is fast.
• A cross-encoder reads both together, which is far more right and cannot be precomputed.
• Use both: broad cheap retrieval for recall, then reranking for precision.
• Rerank around 50 candidates. Measure stage-one recall to know whether that is the bottleneck.
• Reranking cannot rescue a document that retrieval never gave.