Retrieval Evaluation Metrics

Precision@k and Recall@k

precision@k = (relevant in top k) / k

recall@k    = (relevant in top k) / (total relevant)

Precision@k asks how much of what you returned was useful. Recall@k asks how much of what exists you managed to find.

With 3 relevant documents in the collection and a top-5 list containing 2 of them: precision@5 = 2/5 = 0.4, recall@5 = 2/3 = 0.67.

Both ignore order entirely — a relevant document at position 1 and at position 5 count identically. For a RAG pipeline feeding a fixed number of chunks to a model that may be the right simplification; for a search results page it is not.

MRR and NDCG

MRR (mean reciprocal rank) uses only the position of the first relevant result, averaged over queries:

MRR = mean(1 / rank of first relevant)

First position scores 1.0, second 0.5, third 0.33. It is the right metric when the user needs one good answer and will stop reading once they have it — question answering, navigational search — and it deliberately ignores everything after the first hit.

Use NDCG when relevance comes in degrees and the whole ordering matters; use MRR when only the first hit does.

Measuring the stage that usually fails

Almost all RAG failures are retrieval failures: the right chunk never reached the model. Debugging the prompt while recall is 0.4 is wasted effort, so retrieval needs its own metrics, measured separately.

What you need is a small evaluation set — 50 to 200 real questions, each with the id of the chunk (or chunks) that actually answers it. Building it is the unglamorous part and it is what makes every subsequent decision measurable rather than anecdotal.

The metrics divide into two families:

Did we find it? Recall@k, hit rate@k. Binary, per query.

Where did we find it? MRR, nDCG, precision@k. Rank-sensitive.

Recall@k is the one that matters most for RAG

Because the model only sees the top k, recall@k is a ceiling on the whole system. If the answer is not in the retrieved set, no prompt engineering recovers it.

recall@k = (relevant documents in the top k) / (all relevant documents)

With one relevant document per question, recall@k equals hit rate@k, and that is the common case for factoid evaluation sets.

Measure it at several k and read the curve:

That table diagnoses precisely. Recall@20 of 0.94 with recall@5 of 0.71 means the retrieval finds the document but does not rank it highly — which is a reranking problem, not a chunking or embedding problem. If recall@50 were 0.6, the document is not being found at all, and reranking cannot help.

That distinction — find versus rank — is the single most useful thing these metrics tell you.

Rank-sensitive metrics

MRR (mean reciprocal rank) averages 1/rank of the first relevant result:

MRR is appropriate when there is essentially one right answer and its position matters — which describes most question answering. It is harsh on rank 2 versus rank 1, which is arguably correct given how much more attention a model pays to the first chunk.

nDCG@k handles graded relevance: a document can be highly relevant, somewhat relevant or irrelevant, and the metric discounts gains logarithmically by position. It is the standard in information retrieval research and requires graded judgements, which are more expensive to collect than binary ones.

Precision@k matters less for RAG than the others, because passing a few irrelevant chunks alongside the right one is usually tolerable. It matters when context budget is tight or when irrelevant context measurably degrades answers — which it does.

Interactive Exploration Guide

1. Move the cutoff. Slide k from 1 to 10 and watch precision@k and recall@k move in opposite directions. A larger k almost always raises recall and lowers precision.
2. Change the ranking. Switch Ranking so the relevant documents sit lower. Precision@k and recall@k at a large k barely move, while MRR and NDCG drop sharply — that difference is exactly what order-aware metrics measure.
3. Put one relevant result first. Choose a ranking with a relevant document at position 1. MRR jumps to 1.0 regardless of what follows, which shows how much it ignores.
4. Set k = 1. Precision@1 and recall@1 become extremely blunt. At small k the choice of metric matters far more than at large k.

Choosing one for a RAG pipeline

For retrieval feeding a language model, recall@k is usually the metric that matters most. The model can ignore an irrelevant chunk among the k it receives, but it cannot use a relevant chunk that was never retrieved — a missed document is unrecoverable, whereas a spurious one is merely noise.

Set k to the number of chunks you actually pass to the model and measure recall at that value. Then use precision or NDCG as a secondary check, since packing the context with irrelevant material does cost tokens and can distract the model.

What trips people up

• Reporting a metric without its k. “Precision 0.4” is meaningless; precision@5 is a number.
• Using accuracy. It has no meaning over a ranked list of a large corpus, where almost everything is irrelevant.
• Ignoring order when order matters. Precision@k cannot distinguish a system that ranks well from one that merely retrieves the same set in a worse order.
• Assuming complete relevance labels. Most evaluation sets label only judged documents, so an unjudged-but-relevant result counts against you and recall is systematically understated.
• Averaging over too few queries. These metrics are noisy per query; differences need many queries to be meaningful.

Key takeaway

Retrieval returns a ranking, so metrics are computed at a cutoff k and differ mainly in whether they care about position. Precision@k and recall@k ignore order within the cutoff; MRR looks only at the first relevant result; NDCG discounts by position and handles graded relevance. For RAG, recall@k at your actual context size is the number to optimise, because the model can survive a bad chunk but not a missing one.

Building the evaluation set

This is the work that makes everything else possible, and there are three routes.

Hand-written. Domain experts write questions and identify the answering chunk. Highest quality, slowest. 50 questions is enough to start and enough to catch large regressions.

Generated from documents. For each chunk, ask a model to write a question it answers. Fast, and it produces questions phrased like the document, which flatters retrieval — the vocabulary gap that causes real failures is absent by construction.

Mined from logs. Real user queries paired with the documents that were clicked, cited or rated helpful. The most realistic, and it requires a system already in production.

The practical compromise: generate a first set to get moving, then replace it with mined real queries as they become available, and keep a hand-written set of known-hard cases as a regression suite.

Two habits that matter. Include negative cases — questions the corpus genuinely cannot answer — so you can measure whether the system declines rather than inventing. And version the set, so a change in the score means a change in the system rather than a change in the questions.

Measuring end to end as well

Retrieval metrics are necessary and not sufficient. The generation stage has its own failure modes, and the standard measures are:

Groundedness and correctness can diverge informatively. An answer that is correct but ungrounded means the model answered from its own knowledge rather than the documents — which looks like success and will fail silently on anything the model did not memorise.

Frameworks such as RAGAS and TruLens implement these with a model as judge. Model-graded metrics are noisy and directionally useful; validate them against human judgement on a sample before trusting the absolute numbers.

What to do with the numbers

Work top to bottom. Fixing retrieval before generation, and finding before ranking, is the ordering that avoids wasted effort.

Five metrics, one result list, five different answers

The sections above define each metric. Putting them side by side on the SAME ranking is what shows why they disagree -- and which disagreements are telling you something you need to act on.

import numpy as np

# one query. True where the retrieved document is relevant.
RESULTS = [False, True, False, True, True, False, False, False, True, False]
TOTAL_RELEVANT = 6
K = 5

print("ONE QUERY, %d results, %d relevant documents exist in the corpus:"
      % (len(RESULTS), TOTAL_RELEVANT))
print("   rank:     " + " ".join("%2d" % (i + 1) for i in range(len(RESULTS))))
print("   relevant: " + " ".join(" #" if r else " ." for r in RESULTS))
print()

def prec(k):
    return sum(RESULTS[:k]) / float(k)
def rec(k):
    return sum(RESULTS[:k]) / float(TOTAL_RELEVANT)
def hit(k):
    return 1.0 if any(RESULTS[:k]) else 0.0
def rr():
    for i, r in enumerate(RESULTS):
        if r:
            return 1.0 / (i + 1)
    return 0.0
def dcg(rels):
    return sum((1.0 if r else 0.0) / np.log2(i + 2) for i, r in enumerate(rels))
def ndcg(k):
    ideal = dcg(sorted(RESULTS[:k], reverse=True))
    return dcg(RESULTS[:k]) / ideal if ideal else 0.0

print("ALL FIVE AT k=%d:" % K)
print("%-22s %12s %s" % ("metric", "value", "the question it answers"))
rows = [("hit rate@%d" % K, hit(K), "was ANY of it useful"),
        ("precision@%d" % K, prec(K), "how much of what I sent is noise"),
        ("recall@%d" % K, rec(K), "how much of the evidence I found"),
        ("reciprocal rank", rr(), "how far down the first good one was"),
        ("NDCG@%d" % K, ndcg(K), "how well the whole list is ordered")]
for n, v, q in rows:
    print("%-22s %12.4f %s" % (n, v, q))
print("   five numbers between %.2f and %.2f from ONE ranking. none of"
      % (min(v for _, v, _ in rows), max(v for _, v, _ in rows)))
print("   them is wrong -- they are answers to five different questions,")
print("   and quoting one as 'retrieval quality' picks a question without")
print("   saying so.")
print()

print("HOW EACH ONE MOVES WITH k, which is where the disagreements are:")
print("%-8s %12s %12s %12s %12s"
      % ("k", "hit rate", "precision", "recall", "NDCG"))
for k in (1, 2, 3, 5, 10):
    print("%-8d %12.4f %12.4f %12.4f %12.4f"
          % (k, hit(k), prec(k), rec(k), ndcg(k)))
print("   precision saw-tooths downward, recall only rises, hit rate")
print("   saturates at 1 and stops, and NDCG creeps. that is not noise,")
print("   it is the definitions:")
print("      recall's denominator is FIXED, so it can only rise.")
print("      precision's denominator is k, so adding junk lowers it.")
print("      hit rate is a yes/no, so it stops moving once it is 1.")
print("   any argument about whether retrieval 'got better' has to name")
print("   both the metric and the k, or the two sides are not discussing")
print("   the same thing.")
print()

print("NOW THE DISAGREEMENT THAT MATTERS. compare two retrievers on the")
print("same query:")
A = [True, True, False, False, False, False, False, False, False, False]
B = [False, False, True, True, True, True, False, False, False, False]
def m(lst, k, f):
    return f(lst, k)
def P(lst, k):
    return sum(lst[:k]) / float(k)
def R(lst, k):
    return sum(lst[:k]) / float(TOTAL_RELEVANT)
def RRv(lst):
    for i, r in enumerate(lst):
        if r:
            return 1.0 / (i + 1)
    return 0.0
def ND(lst, k):
    ideal = dcg(sorted(lst[:k], reverse=True))
    return dcg(lst[:k]) / ideal if ideal else 0.0
print("%-16s %-24s %10s %10s %10s %10s"
      % ("retriever", "pattern", "prec@5", "rec@5", "RR", "NDCG@5"))
for name, lst in (("A", A), ("B", B)):
    print("%-16s %-24s %10.4f %10.4f %10.4f %10.4f"
          % (name, "".join("#" if r else "." for r in lst[:10]),
             P(lst, 5), R(lst, 5), RRv(lst), ND(lst, 5)))
print("   A wins on reciprocal rank (%.4f against %.4f) and on NDCG@5."
      % (RRv(A), RRv(B)))
print("   B wins on precision@5 and recall@5 (%.4f against %.4f)."
      % (P(B, 5), P(A, 5)))
print("   they disagree because they are good at different things. A puts")
print("   its two relevant documents at ranks 1 and 2 and then stops; B")
print("   starts at rank 3 and keeps going.")
print("      if this feeds a chatbot that cites ONE source, A is better --")
print("         its first result is already the answer.")
print("      if this feeds a RAG prompt with 5 chunks, B is better -- it")
print("         supplies %d supporting passages against A's %d, and the"
      % (sum(B[:5]), sum(A[:5])))
print("         model reads all five either way.")
print("   the metric you report should be the one your SYSTEM cares about,")
print("   and the way to find out which that is, is to ask how many of the")
print("   retrieved documents the next stage actually reads.")
print()

print("AND THE ONE THING ALL OF THEM DEPEND ON, which is not a metric at")
print("all: the labels. every number above needs someone to have decided")
print("which documents are relevant.")
print("%-30s %-22s %s" % ("how the labels are made", "cost", "the bias it adds"))
for row in (("human annotators", "high", "small, if guidelines are good"),
            ("click logs", "free", "you only see what you ranked"),
            ("an LLM judge", "moderate", "agrees with itself, not you"),
            ("the pool of retrieved docs", "free", "cannot see what nobody found")):
    print("%-30s %-22s %s" % row)
print("   the last row is the one that quietly caps everything. if your")
print("   relevant set is 'whatever the current system retrieved and a")
print("   human then approved', then recall is measured against a")
print("   denominator your own retriever chose -- and a genuinely better")
print("   retriever that finds something new gets no credit for it, and")
print("   may score WORSE for returning a document nobody labelled.")
print("   that is why a serious evaluation set is built by pooling several")
print("   retrievers, and why the effort of building one is most of the")
print("   effort of evaluating retrieval at all.")

Questions people ask

How many evaluation questions do I need? 50 catches large regressions; 200 gives reasonably stable comparisons between similar configurations.

Which metric should I optimise? Recall@k where k is the number of chunks you pass to the model. That is the system's ceiling.

Should I use generated questions? To get started, yes — while knowing they understate the vocabulary gap and therefore overstate your recall.

What is a good recall@5? Corpus-dependent. Compare against your own baseline rather than an absolute target, and track it over time.

Do I need graded relevance? Only if you are using nDCG. Binary judgements suffice for recall, hit rate and MRR.

How do I evaluate when there is no single right chunk? Mark all acceptable chunks as relevant and use recall, or move to graded judgements and nDCG.

Recap in one screen

• Measure retrieval separately: it is where nearly all RAG failures are.
• Recall@k is the ceiling on the whole system, because the model only sees the top k.
• Comparing recall at several k separates "not found" from "found but badly ranked" — different fixes.
• MRR suits single-answer questions; nDCG needs graded judgements.
• Build 50–200 questions with known answers, include unanswerable ones, and version the set.