Relevance in LLM evaluation
Does the answer address the question that was asked? Relevance measures how much of the answer is on point — and it is the dimension that catches the failure the other three are blind to: a response that is entirely true, fully grounded, and quietly about something else.
Overview
Two different things are called relevance
The word is used for two distinct measurements and conflating them makes evaluation results incomparable.
Context relevance scores the retrieved chunks against the query. It is a retrieval metric, closely related to precision@k, and it tells you whether your retriever is sending noise to the generator.
Answer relevance scores the generated answer against the query. It is a generation metric, and it is what this page is about.
They fail independently. Perfect context and an irrelevant answer means the generator wandered; irrelevant context and a relevant answer usually means the model answered from memory, which groundedness will catch. Always say which one you mean.
Parameters
Visualisation
—Readout
What to watch
- The 24/7 phone claim is true and grounded and still off topic.
- Padding is scored here and nowhere else.
- Click claims to see relevance move independently of the rest.
Relevance in LLM evaluation: A Practical Guide
What is answer relevance, and how does it differ from context relevance?
The failures it exists to catch
Padding. The answer contains the requested information plus three paragraphs of adjacent context nobody asked for. Every claim is true and grounded, and the user has to hunt for the answer. Models trained to be helpful pad heavily, and no other dimension penalises it.
Answering a nearby question. Asked how long refunds take, the answer explains how to request one. Fluent, grounded, correct, and not responsive.
Hedging. Several paragraphs of caveats and conditions with no actual answer inside them. Technically nothing is wrong; nothing is useful either.
Restating the question. The degenerate answer that echoes the query back. It scores oddly well on naive similarity-based relevance measures, which is a good reason not to use them.
How to measure it without measuring similarity
The tempting approach — embed the question and the answer and take cosine similarity — is bad. It rewards vocabulary overlap, so restating the question scores highly and a correct answer that shares no words with the question scores poorly. "When are refunds issued?" answered with "Within a fortnight of delivery" is perfect and lexically distant.
The approach that works is question generation: ask a model to write the questions this answer would be a good response to, then compare those against the real question. If the generated questions match, the answer is on point; if they are broader or different, it is padded or off target. It measures what relevance actually means rather than a proxy for it.
The alternative is claim-level judging, as in the visualisation above: split the answer into claims and mark each as responsive or not. Relevance is then the supported proportion, and you can see exactly which sentence was padding.
The tension with completeness, and how to resolve it
Relevance and completeness pull in opposite directions, and optimising either alone produces a bad system.
Maximise relevance alone and answers get terse to the point of being unhelpful — every qualification stripped out because qualifications are not strictly what was asked. Maximise completeness alone and answers sprawl, because adding material can only help.
The resolution is to score both and watch them together, the same way precision and recall are read as a pair. An answer that is 0.95 relevant and 0.6 complete is leaving things out; one that is 0.6 relevant and 1.0 complete is burying the answer in padding. Neither single number would tell you which problem you have.
Note also that relevance is the dimension most affected by your prompt rather than your retrieval. "Answer in at most three sentences, addressing only what was asked" moves it substantially, and costs nothing.
Things to try
- The 24/7 phone claim is marked correct but off topic. Relevance drops while correctness and groundedness stay put — this is the padding case, visible only here.
- Click that claim to flip its grounding. Groundedness moves and relevance does not: whether the context supports a claim has nothing to do with whether it answers the question.
- Compare relevance and completeness across the claim list. The off-topic claim hurts one and does nothing for the other, which is why they have to be read as a pair.
What to remember
Answer relevance measures how much of the answer addresses the question asked. Distinguish it from context relevance, which scores retrieved chunks and is a retrieval metric. It is the only dimension that penalises padding, hedging and confidently answering a nearby question — all of which score perfectly on correctness and groundedness. Do not measure it with question-answer embedding similarity, which rewards restating the question; use generated questions or per-claim judging. Read it alongside completeness, because optimising either alone makes answers worse.
These four are usually measured with an LLM as judge: a strong model is shown the question, the retrieved context, the answer and sometimes a reference, and asked to score one dimension at a time. Judging one dimension per call is not a stylistic choice — a single prompt asking for all four produces correlated, mushy scores, because the model settles on a general impression and applies it everywhere.
Two habits make judge scores trustworthy. Ask for a decision and a justification, so a human can audit disagreements. And calibrate against a small human-labelled set before believing any of it — a judge that agrees with your annotators 70% of the time is not measuring what you think it is measuring.