What is Precision@k?

Why noise is not free in a RAG pipeline

The old intuition — "the model can just ignore irrelevant chunks" — is not quite true, and the ways it fails are worth naming.

Cost and latency. Every retrieved chunk is input tokens on every request. Halving the number of chunks halves that bill and shortens time to first token.

Distraction. Irrelevant context measurably reduces answer quality. A plausible-but-wrong passage is worse than no passage — the model has been handed a reason to be confidently incorrect, and it is grounded in a real retrieved document, which makes the error harder to spot.

Position effects. Attention over a long context is uneven, with the middle attended least. Padding the prompt with irrelevant chunks pushes good evidence into that dead zone.

So precision is not merely an efficiency metric here. Past a point, improving precision improves the answers.

The trade with recall, and where each belongs

Precision and recall pull against each other as k moves. Raising k can only help recall and usually hurts precision, because the highest-scoring results were already at the top and what follows is progressively worse.

The standard combination is F1, the harmonic mean, which punishes an imbalance — a system with precision 1.0 and recall 0.1 scores 0.18, not 0.55. The harmonic mean is used precisely because it refuses to let one strong number hide a weak one.

In a modern RAG stack the two are usually optimised at different stages. The retriever runs at high k for recall; the reranker cuts that to a handful for precision. Measuring recall@50 and precision@5 in the same evaluation tells you which of the two stages is failing, and they are fixed by completely different work.

What precision cannot see

Order within k. Relevant documents at ranks 1 and 2 score the same as relevant documents at ranks 4 and 5. For a generator, that difference is real; precision is blind to it and MRR or nDCG is not.

What was missed. A retriever that returns three perfect documents and misses seven more scores precision 1.0. The metric is silent about the corpus it did not touch, which is exactly why it is never reported alone.

Degrees of usefulness. Binary relevance forces a yes/no on documents that are genuinely partial. If your judgements are graded, nDCG uses that information and precision discards it.

What share of what you returned is useful

precision@k = (relevant documents in the top k) / k

If 3 of the top 5 results are relevant, precision@5 = 0.6. The denominator is always k, whether or not that many relevant documents exist.

That last detail matters and is a known limitation: if only 2 relevant documents exist in the whole corpus, precision@5 cannot exceed 0.4 however perfect the retrieval. For that reason precision@k is best read alongside recall, not alone.

Worked through. Top 5 results for "parental leave entitlement", with relevance marked:

precision@5 = 3/5 = 0.6. precision@3 = 2/3 = 0.67. precision@1 = 1.0.

Note that precision@k does not care where the relevant results sit. Ranks 1, 2, 4 and ranks 3, 4, 5 give the same precision@5, which is why rank-sensitive metrics exist.

Why it matters less than recall for RAG — but still matters

For a retrieval system feeding a language model, the two errors are not symmetric. A missing relevant chunk means the answer cannot be produced. An extra irrelevant chunk costs context budget and some attention.

So the retrieval stage should be tuned for recall, and precision is recovered downstream by reranking and by passing only a few chunks to the model.

That does not make precision irrelevant, for three reasons:

Context budget. Passing 20 chunks when 3 are relevant wastes tokens that cost money and latency.

Irrelevant context degrades answers. Models do use what they are given, and contradictory or off-topic passages measurably reduce correctness.

Attention dilution. More context means each relevant chunk gets proportionally less attention, and material in the middle of a long context is attended to less reliably.

So the target is high recall in stage one and high precision in what finally reaches the prompt — which is precisely what a reranker delivers.

Where precision is the primary metric

User-facing search results. Someone scanning a result list judges the system by the first few entries. Precision@1, @3 and @5 are what they experience; recall@50 is invisible to them.

Recommendations. A carousel of five items is judged on whether those five are good.

Alerting and triage. Every false positive costs human review time, so precision is the operational constraint.

Anything with a tight context budget — a small model, a long prompt, or a per-request cost limit.

In those cases the trade runs the other way: favour precision, accept lower recall, and give the user a way to ask for more.

Of what you showed, how much was right

Precision@k is the simplest retrieval metric and the easiest to misread. This computes it on a small result list, shows the three things it deliberately ignores, and puts it beside the metrics that cover for each one.

import numpy as np

# 8 retrieved documents, in rank order. True = actually relevant.
RESULTS = [True, True, False, True, False, False, True, False]
TOTAL_RELEVANT = 6          # how many relevant documents exist in the corpus

print("A RESULT LIST, in the order the retriever returned it:")
print("%-8s %12s %s" % ("rank", "relevant?", ""))
for i, r in enumerate(RESULTS):
    print("%-8d %12s %s" % (i + 1, "YES" if r else "no", "#" if r else "."))
print("   %d of the %d returned are relevant, and the corpus contains %d"
      % (sum(RESULTS), len(RESULTS), TOTAL_RELEVANT))
print("   relevant documents in total.")
print()

print("PRECISION@k = (relevant in the top k) / k")
print("%-8s %16s %14s %14s" % ("k", "relevant in top k", "precision@k",
                               "recall@k"))
for k in range(1, len(RESULTS) + 1):
    hits = sum(RESULTS[:k])
    print("%-8d %16d %14.4f %14.4f"
          % (k, hits, hits / float(k), hits / float(TOTAL_RELEVANT)))
print("   precision does not fall smoothly -- it saw-tooths, rising at")
print("   every relevant result and dropping at every irrelevant one. the")
print("   TREND is downward, and that is the normal shape: a good")
print("   retriever puts its best guesses first, so the deeper you go the")
print("   more junk you accumulate.")
print("   recall only ever rises, because the numerator can only grow.")
print("   the two always trade off, which is why a bare 'precision' with")
print("   no k attached is not a number you can act on.")
print()

print("BLIND SPOT ONE -- ORDER WITHIN THE TOP k. these three lists all")
print("have precision@4 = %.2f:" % 0.5)
variants = [("best first", [True, True, False, False]),
            ("alternating", [True, False, True, False]),
            ("worst first", [False, False, True, True])]
print("%-16s %-14s %14s" % ("list", "pattern", "precision@4"))
for name, v in variants:
    print("%-16s %-14s %14.4f"
          % (name, "".join("#" if x else "." for x in v),
             sum(v) / 4.0))
print("   identical scores. but a user reading top-down meets a relevant")
print("   result at rank 1 in the first list and rank 3 in the last, and")
print("   those are very different experiences.")
print("   precision@k is a SET metric: it asks what is in the top k, never")
print("   in what order. MRR and NDCG exist precisely to fix this.")
def dcg(v):
    return sum((1.0 if x else 0.0) / np.log2(i + 2) for i, x in enumerate(v))

print("      but not every order-aware metric separates all three:")
print("%-18s %14s %14s" % ("list", "recip. rank", "NDCG@4"))
for name, v in variants:
    first = next((i + 1 for i, x in enumerate(v) if x), None)
    ideal = dcg(sorted(v, reverse=True))
    print("%-18s %14.4f %14.4f"
          % (name, 1.0 / first if first else 0.0,
             dcg(v) / ideal if ideal else 0.0))
print("      reciprocal rank gives the first two lists the IDENTICAL score")
print("      of 1.0000, because it only looks at where the first relevant")
print("      result landed and stops. it catches 'worst first' and is")
print("      blind to everything after rank 1.")
print("      NDCG separates all three, because it discounts every position")
print("      by log2(rank+1) and so keeps counting all the way down.")
print("      three metrics, three different questions: what is IN the top")
print("      k, where the FIRST good one is, and how well the whole list")
print("      is ordered.")
print()

print("BLIND SPOT TWO -- IT IGNORES WHAT IT DID NOT RETURN. precision@5")
print("cannot see past rank 5:")
k = 5
print("      precision@%d = %.4f" % (k, sum(RESULTS[:k]) / float(k)))
print("      recall@%d    = %.4f" % (k, sum(RESULTS[:k]) / float(TOTAL_RELEVANT)))
print("   the same precision is achievable whether the corpus holds %d"
      % TOTAL_RELEVANT)
print("   relevant documents or 500:")
print("%-24s %14s %14s" % ("relevant in corpus", "precision@5", "recall@5"))
for total in (3, 6, 50, 500):
    print("%-24d %14.4f %14.4f"
          % (total, sum(RESULTS[:k]) / float(k),
             min(1.0, sum(RESULTS[:k]) / float(total))))
print("   precision does not move at all. recall collapses. if your job is")
print("   'find me everything about X' -- legal discovery, a literature")
print("   review, a compliance audit -- precision is nearly useless on its")
print("   own and recall is the number that matters.")
print()

print("BLIND SPOT THREE -- SHORT LISTS FLATTER YOU. precision@1 is the")
print("easiest number in retrieval to make look good:")
print("%-10s %14s %s" % ("k", "precision@k", "what it really tells you"))
notes = {1: "did rank 1 happen to be right",
         3: "is the top of the list usable",
         5: "typical for a RAG context window",
         8: "everything this list has"}
for k in (1, 3, 5, 8):
    print("%-10d %14.4f %s" % (k, sum(RESULTS[:k]) / float(k),
                               notes.get(k, "")))
print("   reporting precision@1 alone is the retrieval equivalent of")
print("   quoting your best day. pick k from what the SYSTEM does: if you")
print("   stuff 5 chunks into a prompt, precision@5 is the number that")
print("   predicts how much of that prompt is noise.")
print()

print("THE ONE THAT ACTUALLY MATTERS FOR RAG. every irrelevant chunk in")
print("the context window is not merely wasted -- it is a distraction the")
print("model has to ignore:")
print("%-16s %12s %14s %16s"
      % ("k chunks", "precision@k", "useful chunks", "distractors"))
for k in (1, 3, 5, 8):
    kk = min(k, len(RESULTS))
    hits = sum(RESULTS[:kk])
    print("%-16d %12.4f %14d %16d" % (k, hits / float(kk), hits, kk - hits))
print("   going from 3 chunks to 8 adds %d useful ones and %d distractors."
      % (sum(RESULTS[:8]) - sum(RESULTS[:3]),
         (8 - sum(RESULTS[:8])) - (3 - sum(RESULTS[:3]))))
print("   'just retrieve more' trades a little recall for a lot of noise,")
print("   and past some k it makes answers worse rather than better. that")
print("   is the whole argument for reranking: retrieve widely, then throw")
print("   most of it away before the model ever sees it.")
print()

print("AND A WARNING ABOUT AVERAGING. precision@k is defined per query, so")
print("a system-level number is a mean over queries, and means hide things:")
QUERIES = {
    "easy query": [True, True, True, True, True],
    "typical query": [True, False, True, False, False],
    "hard query": [False, False, False, False, False],
    "another typical": [True, True, False, False, False],
}
print("%-22s %14s" % ("query", "precision@5"))
vals = []
for name, r in QUERIES.items():
    v = sum(r) / 5.0
    vals.append(v)
    print("%-22s %14.4f" % (name, v))
print("%-22s %14.4f" % ("MEAN", float(np.mean(vals))))
print("%-22s %14.4f" % ("MEDIAN", float(np.median(vals))))
print("%-22s %14.4f" % ("worst query", float(np.min(vals))))
print("   the mean of %.2f describes no query in the set. one query"
      % np.mean(vals))
print("   returned NOTHING useful, and the average hides it completely.")
print("   report the distribution, or at least the fraction of queries")
print("   below some floor: %.0f%% of these scored under 0.5, which is the"
      % (100 * np.mean([v < 0.5 for v in vals])))
print("   number a user would recognise as 'it often does not work'.")

Things to try

1. Drag k up from 1. Precision generally falls while recall rises — that opposition is the whole reason both numbers are quoted together.
2. Set k to 10 with four relevant documents. Precision cannot exceed 0.4 however good the ranking is; the ceiling is set by the query, not the retriever.
3. Mark the two lowest results relevant and watch precision@5 stay put. Improving what sits below k does nothing for it — which is why k must match what you actually send to the model.

What to remember

Precision@k is the fraction of the returned k that was relevant, with k always as the denominator. It is the cheap metric to label, because it needs judgements only for what you returned. In a RAG pipeline it is not just about efficiency: irrelevant context costs tokens, pushes good evidence into the least-attended part of the prompt, and gives the model plausible material to be wrong with. Optimise recall at the retriever and precision at the reranker, and report both with their k.

All four of these read the same object: an ordered list of retrieved results, with each one labelled relevant or not by a human or a strong model. They differ only in what they choose to notice about it — which is why quoting one without saying which is close to meaningless, and why the visualisation above shows all four at once.

The labels are the expensive part. A relevance judgement per query-document pair is human work, and an evaluation set of fifty queries with judged results is worth more than any amount of metric sophistication on top of unjudged data. Build the set first.

Combining the two

Neither metric alone describes a retrieval system, and there are standard ways to combine them.

F1@k is the harmonic mean of precision@k and recall@k. One number, and it punishes imbalance: precision 1.0 with recall 0.0 gives an F1 of 0, which is the correct verdict.

Precision-recall curve. Sweep k and plot the pair. The shape shows the whole trade-off rather than one point on it.

Average precision (AP) averages the precision at each rank where a relevant document appears — so it rewards putting relevant documents early, unlike plain precision@k. Averaged across queries it gives MAP, a standard information-retrieval measure.

nDCG@k handles graded relevance and discounts by position logarithmically. The most complete measure, and it needs graded judgements rather than binary ones.

For most RAG work, recall@k plus MRR is sufficient. Add nDCG if you have graded judgements and are comparing configurations closely.

Measuring it

def precision_at_k(retrieved, relevant, k):
    top_k = retrieved[:k]
    return sum(1 for d in top_k if d in relevant) / k

def f1_at_k(retrieved, relevant, k):
    p = precision_at_k(retrieved, relevant, k)
    r = len(set(retrieved[:k]) & set(relevant)) / len(relevant)
    return 2 * p * r / (p + r) if (p + r) else 0.0

Two practical notes on interpretation.

Precision@k is bounded by the number of relevant documents. With 2 relevant documents, precision@10 cannot exceed 0.2. Compare precision at a k close to the number of relevant items, or use MAP.

Judgement quality dominates. A document marked irrelevant that a user would have found useful lowers measured precision without lowering real quality. Binary judgements are cheap and blunt; graded ones are expensive and better.

What moves precision

Reranking is the direct instrument. A cross-encoder reads the query and each candidate together and scores them far more accurately than a comparison of precomputed vectors, which is exactly the operation that raises precision within a fixed candidate set.

Questions people ask

Precision or recall for RAG? Recall in the retrieval stage, precision in what reaches the prompt. The two-stage pipeline delivers both.

Why is my precision@10 low even though results look good? Probably because fewer than 10 relevant documents exist. Check the denominator problem.

Should I use F1? Useful as a single summary. Report precision and recall separately as well, since the trade-off is the interesting part.

How many chunks should I pass to the model? 3–10. Measure answer correctness against the number passed — it usually peaks and then declines.

Does high precision mean good answers? Necessary and not sufficient. Generation can still fail with perfect retrieval.

What is MAP? Mean average precision — average precision at each relevant rank, averaged over queries. Rank-sensitive, and a better single number than precision@k.

Recap in one screen

• Precision@k is the share of the top k that is relevant, with k always as the denominator.
• For RAG, a missing chunk is fatal and an extra chunk is wasteful — so favour recall first.
• Irrelevant context still costs money, latency and answer quality, so precision matters in what reaches the prompt.
• Precision@k ignores order and is capped by the number of relevant documents; MAP and nDCG fix both.
• Reranking is the direct way to raise precision within a fixed candidate set.