Retrieval Evaluation Metrics
Same five relevant documents, ten retrieved, three orders. Precision and recall see no difference at all. Two other metrics see everything.
Ranking
all three rankings retrieve exactly the same 5 relevant / 5 irrelevant documents
Ranked Results, 1 to 10
—nDCG Working
Metrics @ k
Evaluating Retrieval: A Practical Guide
Knowing whether retrieval actually worked, not just assuming it did.
Quick Context
Every retrieval method on this site — vector search, BM25, the two combined — needs a way to say whether it actually worked. That requires a labelled test set: queries with a known correct set of relevant documents, checked against what the system actually returned.
The four metrics
- Precision@k — of the k documents retrieved, what fraction are relevant. Punishes returning junk.
- Recall@k — of all the relevant documents that exist, what fraction did the top k catch. Punishes missing things entirely.
- MRR (Mean Reciprocal Rank) — 1 divided by the rank of the first relevant result. Cares only about how long a user waits to see one good answer.
- nDCG (normalized Discounted Cumulative Gain) — every relevant document contributes, but one at rank 1 contributes far more than one at rank 10, via a log discount. The only metric here that rewards a good rank at every position, not just the first.
Interactive Exploration Guide
- Read the Good ranking at k=5. High precision, decent recall, MRR of 1.0 (the first hit is right at position 1), nDCG close to 1.0.
- Switch to Bad. Precision@5 and Recall@5 collapse — reasonable, since the relevant documents are further down. But watch this: at k=10, Precision and Recall are identical to the Good ranking. Same 5 relevant documents, same denominator — position never entered the calculation.
- Now read MRR and nDCG for the same Bad ranking. Both are far below the Good ranking's, at every k. They are the metrics that actually notice the relevant documents were buried.
- Try Mixed. Between the two on every metric — a middle case, not an edge case.
- Read the nDCG working panel. Every rank's contribution to DCG is shown, divided by its log discount — you can see exactly why a hit at rank 1 is worth more than the same hit at rank 8.
Key Takeaway
Precision@k and Recall@k describe a set — how much of what you returned was right, and how much of what was right did you return — and are blind to the order within that set. MRR and nDCG describe a ranking, and reward relevant results for showing up earlier. Reporting only precision and recall on a retrieval system can look identical for a ranking a real user would find excellent and one they would find nearly useless; always check at least one order-sensitive metric alongside them.