Retrieval Evaluation Metrics

Why classification metrics are not enough

A retriever does not return a yes or no; it returns an ordered list of candidates. Two systems can retrieve exactly the same documents and be very different in quality if one puts the apt ones first and the other buries them at position 10.

So retrieval metrics are evaluated at a cutoff k. The number of results a user or a downstream model will really see. And the better ones are sensitive to position within that cutoff.

Precision@k asks: of the k results returned, how many are apt? Recall@k asks: of all the apt documents in the corpus, how many appeared in the top k? Both are simple ratios, and both ignore position within the list — a apt document at rank 1 and one at rank k count equally.

MRR (mean reciprocal rank) cares about position. For each query, it finds the rank of the first apt result and takes the reciprocal: 1 for rank 1, 0.5 for rank 2, 0.1 for rank 10. Averaged over queries, it tells you how far down the list a user often has to look to find the first useful result. It is the right metric when the user needs exactly one answer and will stop as soon as they find it.

NDCG (normalised discounted cumulative gain) is the most general metric. It assigns a graded relevance score to each result (not just apt/irrelevant), discounts it by position (a logarithmic discount, so rank 1 matters most), sums the discounted scores, and normalises by the ideal ranking. The result is a number between 0 and 1 where 1 means the system returned the best doable ranking. NDCG is the standard metric for search engines and recommendation systems. And it is the metric most RAG pipelines should optimise if they can afford graded relevance labels.

Precision@k and Recall@k

precision@k = (apt in top k) / k

recall@k    = (apt in top k) / (total apt)

Precision@k asks how much of what you returned was useful. Recall@k asks how much of what exists you managed to find.

With 3 apt documents in the collection and a top-5 list holding 2 of them. Precision@5 = 2/5 = 0.4, recall@5 = 2/3 = 0.67.

Both ignore order fully — a apt document at position 1 and at position 5 count identically. For a RAG pipeline feeding a fixed number of chunks to a model that may be the right simplification; for a search results page it is not.

MRR and NDCG

MRR (mean reciprocal rank) uses only the position of the first apt result, averaged over queries:

MRR = mean(1 / rank of first apt)

First position scores 1.0, second 0.5, third 0.33. It is the right metric when the user needs one good answer and will stop reading once they have it. Question answering, navigational search. And it on purpose ignores all of it after the first hit.

Use NDCG when relevance comes in degrees and the whole ordering matters; use MRR when only the first hit does.

Measuring the stage that often fails

Almost all RAG failures are retrieval failures: the right chunk never reached the model. Debugging the prompt while recall is 0.4 is wasted effort. So retrieval needs its own metrics, measured apart.

What you need is a small evaluation set — 50 to 200 real questions, each with the id of the chunk (or chunks) that really answers it. Building it is the unglamorous part and it is what makes each later decision measurable rather than anecdotal.

The metrics divide into two families:

Did we find it? Recall@k, hit rate@k. Binary, per query.

Where did we find it? MRR, nDCG, precision@k. Rank-sensitive.

Recall@k is the one that matters most for RAG

Because the model only sees the top k, recall@k is a ceiling on the whole system. If the answer is not in the retrieved set, no prompt engineering recovers it.

recall@k = (apt documents in the top k) / (all apt documents)

With one apt document per question, recall@k equals hit rate@k. And that is the common case for factoid evaluation sets.

Measure it at a few k and read the curve:

That table diagnoses exactly. Recall@20 of 0.94 with recall@5 of 0.71 means the retrieval finds the document but does not rank it highly —. This is a reranking problem, not a chunking or embedding problem. If recall@50 were 0.6, the document is not being found at all, and reranking cannot help.

That distinction — find versus rank — is the single most useful thing these metrics tell you.

Rank-sensitive metrics

MRR (mean reciprocal rank) averages 1/rank of the first apt result:

MRR is right when there is in effect one right answer and its position matters — which describes most question answering. It is harsh on rank 2 versus rank 1, which is arguably correct given how much more attention a model pays to the first chunk.

nDCG@k handles graded relevance: a document can be highly apt, somewhat apt or irrelevant. And the metric discounts gains logarithmically by position. It is the standard in data retrieval research and needs graded judgements. This are more costly to collect than binary ones.

Precision@k matters less for RAG than the others, because passing a few irrelevant chunks alongside the right one is often tolerable. It matters when context budget is tight or when irrelevant context measurably degrades answers — which it does.

Interactive Exploration Guide

1. Move the cutoff. Slide k from 1 to 10 and watch precision@k and recall@k move in opposite directions. A larger k almost always raises recall and lowers precision.
2. Change the ranking. Switch Ranking so the apt documents sit lower. Precision@k and recall@k at a large k barely move. While MRR and NDCG drop sharply — that difference is exactly what order-aware metrics measure.
3. Put one apt result first. Choose a ranking with a apt document at position 1. MRR jumps to 1.0 no matter what follows, which shows how much it ignores.
4. Set k = 1. Precision@1 and recall@1 become very blunt. At small k the choice of metric matters far more than at large k.

Choosing one for a RAG pipeline

For retrieval feeding a language model, recall@k is often the metric that matters most. The model can ignore an irrelevant chunk among the k it receives. But it cannot use a apt chunk that was never retrieved. A missed document is unrecoverable, whereas a spurious one is merely noise.

Set k to the number of chunks you really pass to the model and measure recall at that value. Then use precision or NDCG as a secondary check. Since packing the context with irrelevant material does cost tokens and can distract the model.

What trips people up

• Reporting a metric without its k. “Precision 0.4” is meaningless; precision@5 is a number.
• Using accuracy. It has no meaning over a ranked list of a large corpus. Where almost all of it is irrelevant.
• Ignoring order when order matters. Precision@k cannot distinguish a system that ranks well from one that merely retrieves the same set in a worse order.
• Assuming complete relevance labels. Most evaluation sets label only judged documents. So an unjudged-but-apt result counts against you and recall is step by step understated.
• Averaging over too few queries. These metrics are noisy per query; differences need many queries to be meaningful.

Key takeaway

Retrieval returns a ranking, so metrics are computed at a cutoff k and differ mainly in whether they care about position. Precision@k and recall@k ignore order within the cutoff; MRR looks only at the first apt result; NDCG discounts by position and handles graded relevance. For RAG, recall@k at your actual context size is the number to optimise,. This is because the model can survive a bad chunk but not a missing one.

Building the evaluation set

This is the work that makes all of it else doable, and there are three routes.

Hand-written. Domain experts write questions and find the answering chunk. Highest quality, slowest. 50 questions is enough to start and enough to catch large regressions.

Made from documents. For each chunk, ask a model to write a question it answers. Fast, and it gives questions phrased like the document, which flatters retrieval. The vocabulary gap that causes real failures is absent by construction.

Mined from logs. Real user queries paired with the documents that were clicked, cited or rated helpful. The most realistic, and it needs a system already in production.

The practical compromise: generate a first set to get moving. Then replace it with mined real queries as they become available, and keep a hand-written set of known-hard cases as a regression suite.

Two habits that matter. Include negative cases — questions the corpus truly cannot answer — so you can measure whether the system declines rather than inventing. And version the set, so a change in the score means a change in the system rather than a change in the questions.

Measuring end to end as well

Retrieval metrics are needed and not enough. The generation stage has its own failure modes, and the standard measures are:

Groundedness and correctness can diverge informatively. An answer that is correct but ungrounded means the model answered from its own knowledge. Rather than the documents — which looks like success and will fail silently on anything the model did not memorise.

Frameworks such as RAGAS and TruLens implement these with a model as judge. Model-graded metrics are noisy and directionally useful; validate them against human judgement on a sample before trusting the absolute numbers.

What to do with the numbers

Work top to bottom. Fixing retrieval before generation, and finding before ranking, is the ordering that avoids wasted effort.

Five metrics, one result list, five different answers

The sections above define each metric. Putting them side by side on the SAME ranking is what shows why they disagree. And which disagreements are telling you something you need to act on.

import numpy as np

RESULTS = [False, True, False, True, True, False, False, False, True, False]
TOTAL_RELEVANT = 6
K = 5

print("ONE QUERY, %d results, %d relevant documents exist in the corpus:"
      % (len(RESULTS), TOTAL_RELEVANT))
print("   rank:     " + " ".join("%2d" % (i + 1) for i in range(len(RESULTS))))
print("   relevant: " + " ".join(" #" if r else " ." for r in RESULTS))

def prec(k):
    return sum(RESULTS[:k]) / float(k)
def rec(k):
    return sum(RESULTS[:k]) / float(TOTAL_RELEVANT)
def hit(k):
    return 1.0 if any(RESULTS[:k]) else 0.0
def rr():
    for i, r in enumerate(RESULTS):
        if r:
            return 1.0 / (i + 1)
    return 0.0
def dcg(rels):
    return sum((1.0 if r else 0.0) / np.log2(i + 2) for i, r in enumerate(rels))
def ndcg(k):
    ideal = dcg(sorted(RESULTS[:k], reverse=True))
    return dcg(RESULTS[:k]) / ideal if ideal else 0.0

print()
print("ALL FIVE AT k=%d:" % K)
rows = [("hit rate@%d" % K, hit(K), "was ANY of it useful"),
        ("precision@%d" % K, prec(K), "how much of what I sent is noise"),
        ("recall@%d" % K, rec(K), "how much of the evidence I found"),
        ("reciprocal rank", rr(), "how far down the first good one was"),
        ("NDCG@%d" % K, ndcg(K), "how well the whole list is ordered")]
for n, v, q in rows:
    print("%-22s %12.4f %s" % (n, v, q))
print("Five numbers from ONE ranking. None of them is wrong -- they are")
print("answers to five different questions, and quoting one as")
print("'retrieval quality' picks a question without saying so.")

The same five metrics also move differently as k changes, which is where most disagreements between reports come from.

import numpy as np

RESULTS = [False, True, False, True, True, False, False, False, True, False]
TOTAL_RELEVANT = 6

def prec(k):
    return sum(RESULTS[:k]) / float(k)
def rec(k):
    return sum(RESULTS[:k]) / float(TOTAL_RELEVANT)
def hit(k):
    return 1.0 if any(RESULTS[:k]) else 0.0
def dcg(rels):
    return sum((1.0 if r else 0.0) / np.log2(i + 2) for i, r in enumerate(rels))
def ndcg(k):
    ideal = dcg(sorted(RESULTS[:k], reverse=True))
    return dcg(RESULTS[:k]) / ideal if ideal else 0.0

print("HOW EACH ONE MOVES WITH k, which is where the disagreements are:")
print("%-8s %12s %12s %12s %12s"
      % ("k", "hit rate", "precision", "recall", "NDCG"))
for k in (1, 2, 3, 5, 10):
    print("%-8d %12.4f %12.4f %12.4f %12.4f"
          % (k, hit(k), prec(k), rec(k), ndcg(k)))
print("Precision saw-tooths downward, recall only rises, hit rate")
print("saturates at 1 and stops, and NDCG creeps. That is the")
print("definitions, not noise: recall's denominator is FIXED so it can")
print("only rise; precision's denominator is k so junk lowers it; hit")
print("rate is yes/no so it stops moving once it hits 1. Any claim that")
print("retrieval 'got better' has to name both the metric and k.")

The disagreement that matters most shows up when two different retrievers are compared on the same query.

import numpy as np

TOTAL_RELEVANT = 6
A = [True, True, False, False, False, False, False, False, False, False]
B = [False, False, True, True, True, True, False, False, False, False]

def dcg(rels):
    return sum((1.0 if r else 0.0) / np.log2(i + 2) for i, r in enumerate(rels))
def P(lst, k):
    return sum(lst[:k]) / float(k)
def R(lst, k):
    return sum(lst[:k]) / float(TOTAL_RELEVANT)
def RRv(lst):
    for i, r in enumerate(lst):
        if r:
            return 1.0 / (i + 1)
    return 0.0
def ND(lst, k):
    ideal = dcg(sorted(lst[:k], reverse=True))
    return dcg(lst[:k]) / ideal if ideal else 0.0

print("TWO RETRIEVERS ON THE SAME QUERY:")
print("%-10s %-24s %10s %10s %10s %10s"
      % ("retriever", "pattern", "prec@5", "rec@5", "RR", "NDCG@5"))
for name, lst in (("A", A), ("B", B)):
    print("%-10s %-24s %10.4f %10.4f %10.4f %10.4f"
          % (name, "".join("#" if r else "." for r in lst[:10]),
             P(lst, 5), R(lst, 5), RRv(lst), ND(lst, 5)))

print("A wins on reciprocal rank and NDCG@5: it puts its two relevant")
print("documents at ranks 1 and 2 then stops. B wins on precision@5 and")
print("recall@5: it starts at rank 3 and keeps going. If this feeds a")
print("chatbot that cites ONE source, A is better -- its first result")
print("is already the answer. If it feeds a RAG prompt with 5 chunks, B")
print("is better -- it supplies more supporting passages and the model")
print("reads all five either way. Report the metric your SYSTEM cares")
print("about: ask how many retrieved documents the next stage reads.")

Each one of those metrics rests on something that is not itself a metric. How the relevance labels were made.

ROWS = [
    ("human annotators", "high", "small, if guidelines are good"),
    ("click logs", "free", "you only see what you ranked"),
    ("an LLM judge", "moderate", "agrees with itself, not you"),
    ("the pool of retrieved docs", "free", "cannot see what nobody found"),
]

print("EVERY METRIC ABOVE DEPENDS ON ONE THING THAT IS NOT A METRIC:")
print("the labels -- someone decided which documents are relevant.")
print("%-30s %-22s %s" % ("how the labels are made", "cost", "the bias it adds"))
for row in ROWS:
    print("%-30s %-22s %s" % row)

print()
print("The last row quietly caps everything. If your relevant set is")
print("'whatever the current system retrieved and a human then")
print("approved', recall is measured against a denominator your own")
print("retriever chose -- a genuinely better retriever that finds")
print("something new gets no credit, and may score WORSE for returning")
print("an unlabelled document. That is why a serious evaluation set is")
print("built by pooling several retrievers, and why building it is")
print("most of the effort of evaluating retrieval at all.")

Questions people ask

How many evaluation questions do I need? 50 catches large regressions; 200 gives fairly stable checks between similar configurations.

Which metric should I optimise? Recall@k where k is the number of chunks you pass to the model. That is the system's ceiling.

Should I use made questions? To get started, yes. While knowing they understate the vocabulary gap and so overstate your recall.

What is a good recall@5? Corpus-dependent. Compare against your own baseline rather than an absolute target, and track it over time.

Do I need graded relevance? Only if you are using nDCG. Binary judgements suffice for recall, hit rate and MRR.

How do I evaluate when there is no single right chunk? Mark all acceptable chunks as apt and use recall, or move to graded judgements and nDCG.

Recap in one screen

• Measure retrieval apart: it is where nearly all RAG failures are.
• Recall@k is the ceiling on the whole system, because the model only sees the top k.
• Comparing recall at a few k separates "not found" from "found but badly ranked" — different fixes.
• MRR suits single-answer questions; nDCG needs graded judgements.
• Build 50–200 questions with known answers, include unanswerable ones, and version the set.