What is MRR (Mean Reciprocal Rank)?

When position is the whole question

MRR is the right metric when the consumer stops at the first good result. Question answering with a single correct answer, "I'm feeling lucky" search, entity lookup, a code assistant jumping to a definition — in all of these the second correct result is worth nothing.

It also matters inside RAG more than it first appears. Attention over a long context is uneven, and evidence placed first is more likely to be used than evidence placed eighth. Two retrievers with identical recall@5 can produce measurably different answers if one puts the key chunk at the top and the other buries it. Recall cannot see that difference; MRR can.

The practical use: track recall@k to know whether the evidence arrives, and MRR to know whether it arrives somewhere the model will actually look. A reranker that leaves recall unchanged while lifting MRR is doing real work.

What it ignores, and when that is wrong

Every relevant result after the first. A query with one relevant document at rank 1 and a query with ten relevant documents at ranks 1 to 10 both score 1.0. If your questions need multiple sources — comparisons, summaries, anything aggregative — MRR is close to blind to what you care about, and Recall@k is the metric to use.

Degrees of relevance. A marginally useful document at rank 1 outscores a perfect one at rank 2. MAP averages precision over every relevant position and nDCG additionally uses graded judgements; both are strictly more informative and both cost more to label.

The zero. Queries with no relevant result contribute 0, which is correct and worth watching: a system with excellent MRR on the queries it answers and a large silent tail of zeros has a coverage problem that the average partially conceals. Report the proportion of zero-score queries next to it.

Reading MRR against the alternatives

The family is easiest to keep straight by what each one uses:

Hit Rate@k — is there anything relevant in the top k? Binary, ignores position entirely.

MRR — where is the first relevant result? Uses position, ignores everything after it.

MAP — where is every relevant result? Uses all positions, still binary relevance.

nDCG — where is every relevant result, and how relevant is each? Uses positions and grades, and is the most informative and most expensive.

They are a ladder of increasing information and increasing labelling cost. Climb it only as far as your judgements can honestly support: nDCG computed from binary labels guessed by a weak model is not better than a hit rate from careful human ones.

How high was the first right answer?

MRR measures where the first relevant result appeared, averaged over queries.

MRR = (1/|Q|) Σ 1 / rank of the first relevant result

The reciprocal is what gives the metric its character:

Worked example over four queries, with the first relevant result at ranks 1, 3, 2 and not-found:

MRR = (1.00 + 0.33 + 0.50 + 0) / 4 = 0.46

Note how harsh the drop from rank 1 to rank 2 is — half the credit for one position. That is deliberate, and it matches two realities: users scan from the top, and language models attend most strongly to the earliest context.

What it is and is not good for

MRR suits questions with essentially one right answer. Factoid question answering, entity lookup, "which document covers X" — cases where finding the answer once is what matters and there is nothing to gain from finding it twice.

MRR ignores everything after the first hit. A system that puts one relevant document at rank 1 and nine irrelevant ones below scores the same as one that puts ten relevant documents in the top ten. If coverage matters, MRR is the wrong metric.

That makes it complementary rather than competing:

The pair worth reporting together for RAG is recall@k and MRR. Recall says whether the ceiling is high enough; MRR says whether the ordering is putting the answer where the model will attend to it.

Reading the two together

The second row is the common and fixable one. High recall with low MRR means the candidate set is good and the ordering is not, which is exactly what a cross-encoder reranker addresses — and it typically moves MRR substantially while leaving recall unchanged.

The third row points the other way: no reranker can help, and the work is in chunking, the embedding model, or adding keyword search.

Scoring by where the first good answer landed

The sections above describe what MRR measures and what it ignores. Here it is as arithmetic on a concrete result set, along with the two numeric traps the description cannot show you: what the zeros do to the average, and why the mean of reciprocal ranks behaves nothing like the reciprocal of the mean rank.

import numpy as np

QUERIES = {
    "how do I reset my password": [False, True, False, True],
    "what are your opening hours": [True, True, False, False],
    "do you ship to Canada": [False, False, False, True],
    "refund policy": [False, False, False, False],
    "contact support": [True, False, False, False],
}

print("FIVE QUERIES, each showing which of the top 4 results were relevant:")
print("%-32s %-12s %10s %14s"
      % ("query", "results", "first hit", "reciprocal rank"))
rrs = []
for q, res in QUERIES.items():
    first = next((i + 1 for i, r in enumerate(res) if r), None)
    rr = 1.0 / first if first else 0.0
    rrs.append(rr)
    print("%-32s %-12s %10s %14.4f"
          % (q[:30], "".join("#" if r else "." for r in res),
             str(first) if first else "none", rr))
print("%-32s %-12s %10s %14.4f"
      % ("MEAN RECIPROCAL RANK", "", "", float(np.mean(rrs))))
print()

print("WHAT MRR CANNOT SEE. compare two result lists for the same query:")
LISTS = {
    "one good result, at rank 1": [True, False, False, False],
    "four good results, from 1": [True, True, True, True],
    "one good result, at rank 3": [False, False, True, False],
    "three good, starting at 3": [False, False, True, True],
}
print("%-32s %-10s %10s %14s %16s"
      % ("list", "pattern", "RR", "precision@4", "relevant found"))
for name, res in LISTS.items():
    first = next((i + 1 for i, r in enumerate(res) if r), None)
    print("%-32s %-10s %10.4f %14.4f %16d"
          % (name, "".join("#" if r else "." for r in res),
             1.0 / first if first else 0.0, sum(res) / 4.0, sum(res)))
print("   the first two lists score IDENTICALLY under MRR -- 1.0000 each")
print("   -- and one of them found four times as much. MRR stops looking")
print("   after the first hit, by construction.")
print("   that is a feature when one answer is all anyone wants, and a")
print("   serious flaw when the user needs several.")
print()

print("THE ZERO PROBLEM. a query with no relevant result in the list gets")
print("RR = 0, and zeros pull an average hard:")
print("%-30s %14s %14s" % ("", "MRR", "MRR, excl. zeros"))
nonzero = [r for r in rrs if r > 0]
print("%-30s %14.4f %14.4f"
      % ("all %d queries" % len(rrs), float(np.mean(rrs)),
         float(np.mean(nonzero))))
print("   %.4f against %.4f, from the same system. the second number"
      % (np.mean(rrs), np.mean(nonzero)))
print("   answers 'when it works, how well does it work' and the first")
print("   answers 'how well does it work'. both are legitimate; quoting")
print("   one and meaning the other is not.")
print("   report the failure rate separately: %d of %d queries (%.0f%%)"
      % (len(rrs) - len(nonzero), len(rrs),
         100.0 * (len(rrs) - len(nonzero)) / len(rrs)))
print("   returned nothing relevant at all, and that number is invisible")
print("   inside an MRR of %.4f." % np.mean(rrs))
print()

print("AND A SUBTLETY IN THE AVERAGE ITSELF. reciprocal rank is a")
print("nonlinear function of rank, so the MEAN of reciprocal ranks is not")
print("the reciprocal of the mean rank:")
ranks = [1, 2, 10]
print("      first hits at ranks %s" % str(ranks))
print("      mean rank            = %.4f  -> 1/mean = %.4f"
      % (np.mean(ranks), 1.0 / np.mean(ranks)))
print("      mean reciprocal rank = %.4f" % np.mean([1.0 / r for r in ranks]))
print("   %.4f against %.4f. MRR is dominated by the good queries; the"
      % (np.mean([1.0 / r for r in ranks]), 1.0 / np.mean(ranks)))
print("   mean rank is dominated by the bad ones. a single result at rank")
print("   1000 barely moves MRR and wrecks the mean rank:")
for tail in (10, 100, 1000):
    rs = [1, 2, tail]
    print("      worst rank %5d ->  MRR %.4f   mean rank %8.1f"
          % (tail, np.mean([1.0 / r for r in rs]), np.mean(rs)))
print("   neither is wrong. they answer different questions, and MRR's")
print("   answer -- 'how quickly does a user usually succeed' -- is the")
print("   one that matches how search is actually used.")

Things to try

1. Click result #1 on and off. MRR swings between 1.0 and 0.33 on a single change, while recall moves a fraction — MRR is by far the most position-sensitive of the four.
2. Mark only the last result relevant. RR is 0.1, and no amount of additional relevant documents further down would improve it.
3. Mark results 1 and 2 both relevant, then mark only result 1. MRR is identical either way — everything after the first hit is invisible to it.

What to remember

MRR averages 1/rank of the first relevant result. It is the metric for systems where the consumer stops at the first good answer, and it is far more sensitive to the top of the ranking than recall or precision. It ignores every relevant result after the first, so it is the wrong choice for questions needing several sources. Inside RAG it is a useful companion to recall: recall says the evidence arrived, MRR says whether it arrived where the model will actually attend to it.

All four of these read the same object: an ordered list of retrieved results, with each one labelled relevant or not by a human or a strong model. They differ only in what they choose to notice about it — which is why quoting one without saying which is close to meaningless, and why the visualisation above shows all four at once.

The labels are the expensive part. A relevance judgement per query-document pair is human work, and an evaluation set of fifty queries with judged results is worth more than any amount of metric sophistication on top of unjudged data. Build the set first.

Measuring it

def reciprocal_rank(retrieved_ids, relevant_ids):
    relevant = set(relevant_ids)
    for i, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in relevant:
            return 1.0 / i
    return 0.0

def mrr(eval_set, retriever, k=10):
    return sum(
        reciprocal_rank(retriever(q["question"], k=k), q["relevant_ids"])
        for q in eval_set
    ) / len(eval_set)

Two details that affect the number:

The cutoff matters. MRR@10 treats a relevant document at rank 15 as not found, scoring 0. Report the cutoff you used, and keep it consistent when comparing.

Zeros dominate. A query where nothing relevant was retrieved contributes 0, and with 20% such queries the maximum achievable MRR is 0.8. Low MRR can therefore be a recall problem in disguise — which is why the two are read together.

For per-query debugging, the reciprocal ranks are more useful than the mean. Sort queries by reciprocal rank and look at the zeros first: those are the outright failures, and they usually cluster around a recognisable cause.

MRR and RRF are not the same thing

The similarity in name and formula causes confusion, so it is worth separating them.

MRR is an evaluation metric — 1/rank, averaged over queries, measuring how well a system ranks.

Reciprocal rank fusion is a combination method — Σ 1/(k + rank) summed across several ranked lists, producing a merged ranking.

They share the reciprocal-of-rank idea and do entirely different jobs: one scores a system, the other builds one. The +k in RRF (usually 60) is a smoothing constant that flattens the difference between top ranks; MRR has no such constant, which is why it is so harsh on rank 2.

Related rank-sensitive metrics

MAP (mean average precision) averages the precision at each rank where a relevant document appears, then averages over queries. Unlike MRR it accounts for all relevant documents, which makes it the better choice when several exist.

nDCG@k discounts gains logarithmically by position and supports graded relevance — highly relevant, somewhat relevant, irrelevant. The most complete measure, and it needs graded judgements, which are more expensive to collect.

Hit rate@k is the crude version: did any relevant document appear in the top k, yes or no. Useful as a headline, and blind to position.

For most RAG work, recall@k plus MRR is enough. Move to nDCG when comparing closely-matched configurations and you have graded labels.

Questions people ask

What is a good MRR? Corpus-dependent. Track it against your own baseline over time rather than an absolute target.

Should I use MRR or MAP? MRR when there is one right answer; MAP when several relevant documents exist and all of them matter.

Why is MRR so harsh on rank 2? Because 1/2 is half of 1/1. It reflects that being first matters disproportionately, for users and for model attention.

Does MRR need graded relevance? No — binary judgements are sufficient, which is part of its appeal.

How does a reranker affect it? Substantially, and that is precisely what a reranker is for: reordering a fixed candidate set.

Is MRR the same as reciprocal rank fusion? No — one evaluates a ranking, the other combines several rankings.

Recap in one screen

• MRR averages 1/rank of the first relevant result, so being first is worth twice being second.
• It suits single-answer questions and ignores everything after the first hit.
• Report it with recall@k: high recall and low MRR means add a reranker; low recall means fix retrieval first.
• Queries with nothing relevant contribute 0, so low MRR can be a recall problem in disguise.
• MRR evaluates a ranking; reciprocal rank fusion combines rankings — similar formula, different job.