Relevance in LLM evaluation
Does the answer address the question that was asked? Relevance measures how much of the answer is on point — and it is the dimension that catches the failure the other three are blind to: a response that is entirely true, fully grounded, and quietly about something else.
Overview
Two different things are called fit
The word is used for two distinct measurements and conflating them makes evaluation results incomparable.
Context fit scores the fetched chunks against the query. It is a retrieval metric, closely tied to precision@k. And it tells you whether your retriever is sending noise to the generator.
Answer fit scores the made answer against the query. It is a generation metric, and it is what this page is about.
They fail on its own. Perfect context and an irrelevant answer means the generator wandered; irrelevant context and a apt answer often means the model answered from memory. This groundedness will catch. Always say which one you mean. Answer fit is often scored by an LLM judge that reads the question and the answer. Then rates how right the answer fixes what was asked. A common implementation makes synthetic questions from the answer and measures the semantic likeness between those questions and the first. If the answer is apt, the questions it implies should resemble the question that was asked. Scores are often normalised to 0–1. A low score means the answer drifted off topic, included unrelated data, or addressed a different question fully. The metric pairs with faithfulness (does the answer stick to the fetched context?) and context fit (did the retriever find the right passages?) to form the three-axis evaluation framework used by RAGAS, DeepEval, and close libraries. This page scores a few question-answer pairs, shows the synthetic-question method step by step, and covers the distinction between fit and faithfulness that trips up most evaluations.
Does the answer address the question that was asked ?
In this guide
- Two different things are called fit
- The failures it exists to catch
- How to measure it without gauging likeness
- The tension with completeness, and how to resolve it
- Did it answer the question that was asked?
- Gauging it
- Why answers drift off-topic
- Gauging "answers the question" without gauging likeness
- Things to try with Relevance in LLM evaluation
- What to recall
- What improves it
- Fit within the metric suite
- Real evaluation setup
- Questions people ask about Relevance in LLM evaluation
- Relevance in LLM evaluation: recap in one screen
Parameters
Visualisation
—Readout
What to watch
- The 24/7 phone claim is true and grounded and still off topic.
- Padding is scored here and nowhere else.
- Click claims to see relevance move independently of the rest.
Relevance in LLM evaluation: A Practical Guide
What is answer relevance, and how does it differ from context relevance?
Two different things are called fit
The word is used for two distinct measurements and conflating them makes evaluation results incomparable.
Context fit scores the fetched chunks against the query. It is a retrieval metric, closely tied to precision@k. And it tells you whether your retriever is sending noise to the generator.
Answer fit scores the made answer against the query. It is a generation metric, and it is what this page is about.
They fail on its own. Perfect context and an irrelevant answer means the generator wandered; irrelevant context and a apt answer often means the model answered from memory. This groundedness will catch. Always say which one you mean. Answer fit is often scored by an LLM judge that reads the question and the answer. Then rates how right the answer fixes what was asked. A common implementation makes synthetic questions from the answer and measures the semantic likeness between those questions and the first. If the answer is apt, the questions it implies should resemble the question that was asked. Scores are often normalised to 0–1. A low score means the answer drifted off topic, included unrelated data, or addressed a different question fully. The metric pairs with faithfulness (does the answer stick to the fetched context?) and context fit (did the retriever find the right passages?) to form the three-axis evaluation framework used by RAGAS, DeepEval, and close libraries. This page scores a few question-answer pairs, shows the synthetic-question method step by step, and covers the distinction between fit and faithfulness that trips up most evaluations.
The failures it exists to catch
Padding. The answer holds the requested data plus three paragraphs of adjacent context nobody asked for. Each claim is true and grounded, and the user has to hunt for the answer. Models trained to be helpful pad heavily, and no other dimension penalises it.
Answering a nearby question. Asked how long refunds take, the answer explains how to request one. Fluent, grounded, correct, and not responsive.
Hedging. A few paragraphs of caveats and conditions with no actual answer inside them. Technically nothing is wrong; nothing is useful either.
Restating the question. The degenerate answer that echoes the query back. It scores oddly well on naive likeness-based fit measures, which is a good reason not to use them.
How to measure it without gauging likeness
The tempting approach — embed the question and the answer and take cosine likeness — is bad. It rewards vocabulary overlap, so restating the question scores highly and a correct answer that shares no words with the question scores poorly. "When are refunds issued?" answered with "Within a fortnight of delivery" is perfect and lexically distant.
The approach that works is question generation: ask a model to write the questions this answer would be a good response to. Then compare those against the real question. If the made questions match, the answer is on point; if they are broader or different, it is padded or off target. It measures what fit really means rather than a proxy for it.
The other is claim-level judging, as in the visualisation above. Split the answer into claims and mark each as responsive or not. Fit is then the backed share, and you can see just which sentence was padding.
The tension with completeness, and how to resolve it
Fit and completeness pull in opposite ways, and optimising either alone gives a bad system.
Boost fit alone and answers get terse to the point of being unhelpful. Each qualification stripped out because qualifications are not strictly what was asked. Boost completeness alone and answers sprawl, because adding material can only help.
The resolution is to score both and watch them together, the same way precision and recall are read as a pair. An answer that is 0.95 apt and 0.6 complete is leaving things out; one that is 0.6 apt and 1.0 complete is burying the answer in padding. Neither single number would tell you which problem you have.
Note also that fit is the dimension most affected by your prompt rather than your retrieval. "Answer in at most three sentences, fixing only what was asked" moves it much, and costs nothing.
Did it answer the question that was asked?
Fit asks whether the response fixes the user's actual question. It is separate from correctness, and a system can fail on it while being fully factual.
Question: "How do I cancel my subscription?"
Answer A: "Go to Settings, then Billing, then Cancel." — apt
Answer B: "Our subscription plans include Basic at £9 and Pro at £29.". True, and not an answer
Answer C: "Cancellation is available. You may also downgrade, pause, transfer or gift your subscription. And our refund policy states..." — buried|
B is the classic retrieval-driven failure: the model was given documents about subscriptions and gave a thing about subscriptions. C is the padding failure, which is common in tuned models and truly reduces usefulness.
Three distinguishable problems hide under low fit:
| Problem | Symptom |
|---|---|
| Off-topic | Answers a different question |
| Partial | Addresses one part of a multi-part question |
| Padded | Correct answer buried in unrequested material |
Gauging it
Fit has no ground-truth answer to compare against — it is judged against the question, not against a guide. That makes it a natural fit for a model judge. And it means the judging prompt has to be precise about what is being asked.
Question: {question}
Answer: {answer}
Does the answer directly address what was asked? Ignore whether it is
factually correct - judge only whether it responds to the question.
Reply as JSON: {"verdict": "relevant" | "partially_relevant" | "irrelevant",
"reason": "<one sentence>"}The cue to ignore correctness matters. Without it, judges conflate the two and the metric stops being on its own useful.
An other measurement worth knowing: answer-to-question likeness via made questions. Ask a model what question the answer appears to be answering, embed both that and the first question, and compare. A large gap shows the answer drifted. It is cheaper than a judge and cruder.
Why answers drift off-topic
Diagnosing low fit means knowing the causes, and most of them are not the model's fault.
Retrieval returned topically-tied but unhelpful chunks. The model uses what it is given. So context about subscription pricing gives an answer about pricing. This is the most common cause, and it is a retrieval problem.
The question was ambiguous. "How do I change it?" needs the conversation history resolved before retrieval; without contextualisation, both retrieval and generation guess.
Multi-part questions get partially answered, because the model fixes the first part and the fetched context only covers that part.
The prompt encourages padding. Cues to be "helpful and thorough" reliably give answers with unrequested extras.
Cue tuning bias. Models tuned on choice data tend towards longer answers, because human raters mildly prefer them. That is a systematic pull towards padding.
Gauging "answers the question" without gauging likeness
The obvious way to score fit is to compare the answer with the question. And it is the wrong way. A restatement of the question is maximally close to it and answers nothing. This shows that failure right, then the two things the dimension is really for.
That naive check ranks the restatement highest and the real answer low, which is backwards. The next block shows the two failures fit really exists to catch, ones nothing else in a metric suite flags.
The fix inverts the check: instead of comparing the answer to the question, generate the question the answer implies and compare that.
Fit also trades off against completeness, and the last block lays that tension out right.
Things to try with Relevance in LLM evaluation
- The 24/7 phone claim is marked correct but off topic. Fit drops while correctness and groundedness stay put. This is the padding case, visible only here.
- Click that claim to flip its grounding. Groundedness moves and fit does not: whether the context backs a claim has nothing to do with whether it answers the question.
- Compare fit and completeness across the claim list. The off-topic claim hurts one and does nothing for the other. This is why they have to be read as a pair.
What to recall
Answer fit measures how much of the answer fixes the question asked. Distinguish it from context fit, which scores fetched chunks and is a retrieval metric. It is the only dimension that penalises padding, hedging and confidently answering a nearby question. All of which score perfectly on correctness and groundedness. Do not measure it with question-answer embedding likeness, which rewards restating the question; use made questions or per-claim judging. Read it alongside completeness, because optimising either alone makes answers worse.
These four are often measured with an LLM as judge. A strong model is shown the question, the fetched context, the answer and sometimes a guide, and asked to score one dimension at a time. Judging one dimension per call is not a stylistic choice. A single prompt asking for all four gives linked, mushy scores, because the model settles on a general impression and applies it all over.
Two habits make judge scores trustworthy. Ask for a decision and a justification, so a human can audit disagreements. And calibrate against a small human-labelled set before believing any of it. A judge that agrees with your annotators 70% of the time is not gauging what you think it is gauging.
What improves it
Fix retrieval first. If the fetched chunks are about the right topic and the wrong question, the answer will be too. Measure recall@k on the set failing queries before touching the prompt.
Contextualise the question against conversation history before fetching. A follow-up question is meaningless as a standalone search. And this single step fixes a large share of fit failures in chat interfaces.
Decompose multi-part questions, retrieve for each part, and instruct the model to address each by name.
Constrain the shape of the answer. "Answer in at most three sentences. Do not include data that was not asked for." Blunt, and effective against padding.
Ask for the direct answer first, then any caveats. This helps even when the extra material stays, because the useful part is no longer buried.
Reduce the number of chunks. Ten chunks invite the model to synthesise across all of them; three keep it focused.
Fit within the metric suite
| Metric | Question | Independent of |
|---|---|---|
| Recall@k | Was the answer retrieved? | Generation |
| Groundedness | Is each claim supported by the context? | Truth |
| Correctness | Is the answer right? | Style |
| Relevance | Does it address the question? | Truth and completeness |
| Completeness | Does it cover everything asked? | Concision |
Fit and completeness pull against each other, which is worth naming by name. A terse answer scores well on fit and may miss needed detail; an exhaustive one covers all of it and buries the answer.
Reporting both, and reading them as a pair, is how you find the balance. High fit with low completeness means answers are too narrow; the reverse means they are padded.
Real evaluation setup
Three things make a fit measurement trustworthy:
Use real user questions. Made questions tend to be well-formed and single-part. This is just the case fit failures do not occur in. Mine logs where doable.
Include multi-part and ambiguous questions on purpose. These are where fit breaks. And a set of clean factoid questions will show 0.95 and tell you nothing.
Validate the judge on a sample. Grade 50 answers yourself and measure agreement. Judges are more steady on fit than on correctness, because the question is narrower. And it is still worth checking.
Track it per question type and in aggregate. Fit is often high on single-part factoid questions and much lower on comparative or procedural ones. And the aggregate hides that.
Questions people ask about Relevance in LLM evaluation
Is fit the same as correctness? No. An answer can be fully true and not address the question.
How is it unlike precision@k? Precision measures fetched documents; fit measures the made answer.
Can I measure it without ground truth? Yes — it is judged against the question. This is what makes it cheaper to evaluate than correctness.
Why are my answers so long? Cue-tuned models are biased towards length, because choice raters mildly prefer it. Constrain it by name in the prompt.
Should a refusal count as apt? A well-targeted refusal — "the documents do not cover this" — is apt. Track refusal rate apart so over-refusal is visible.
What if the question itself is unclear? Asking a clarifying question is arguably the most apt response. Decide whether your evaluation set rewards that.
Relevance in LLM evaluation: recap in one screen
- Fit asks whether the answer fixes the question, on its own of whether it is true.
- Three failure shapes: off-topic, partial, and correct-but-buried.
- The most common cause is retrieval returning topically-tied but unhelpful chunks.
- Contextualise follow-up questions, decompose multi-part ones, and constrain answer length.
- Read it alongside completeness — the two pull in opposite ways.