Correctness in LLM evaluation
Is the answer right? Correctness compares the answer's claims against a reference answer or against verifiable fact. It is the dimension users care about most and the one that is hardest to automate, because unlike groundedness it cannot be checked against the context — it needs a ground truth that someone has to produce.
Overview
Correct against what, just
Correctness is only set next to a guide, and choosing that guide is most of the work:
A written gold answer. The usual approach. A human writes the ideal answer per evaluation query and a judge compares. Precise, and costly enough that evaluation sets stay small — which is why fifty carefully written guides beat five hundred sloppy ones.
A key-facts list. Rather than prose, the guide is a set of facts the answer must state. Easier to write, much easier to judge always, and it composes naturally with completeness.
Verifiable work. Where the answer is code, a number or a query, correctness can be executed rather than judged. This is the gold standard when it is available and it almost never is for open-ended questions.
The cost is the reason this dimension needs separate treatment. Writing a gold answer takes domain expertise, verifying a model's answer against it takes judgement, and scaling either one takes money or an LLM-as-judge. Which adds its own biases. Exact string match fails at once on natural language. This is because paraphrase is the norm, and n-gram metrics (BLEU, ROUGE) reward surface overlap rather than factual precision. Claim-level judging. Decomposing both the answer and the guide into atomic facts and comparing them — is the approach that handles paraphrase, gives partial credit, and finds. This set fact was wrong. This page walks through each guide format, shows claim-level scoring on a worked case, and covers the failure mode this dimension exists to catch: the grounded-but-wrong answer. Where the model faithfully reports what a fetched document says and that document is out of date.
Is the answer right ?
In this guide
- Correct against what, just
- Why exact match fails, and what replaces it
- The failure this dimension exists to catch
- Judging it without fooling yourself
- Is the answer right?
- Model-as-judge, done carefully
- Correctness against groundedness
- Why string matching fails, and what the judge costs
- Things to try with Correctness in LLM evaluation
- What to recall
- Building the evaluation set
- What correctness does not tell you
- Running it as a regression suite
- Questions people ask about Correctness in LLM evaluation
- Correctness in LLM evaluation: recap in one screen
Parameters
Visualisation
—Readout
What to watch
- Correctness is judged against a reference, not against the context.
- A grounded claim can still be wrong if the source document is wrong.
- Click claims to see correctness and groundedness diverge.
Correctness in LLM evaluation: A Practical Guide
What is correctness, and why is it the most expensive dimension to measure?
Correct against what, just
Correctness is only set next to a guide, and choosing that guide is most of the work:
A written gold answer. The usual approach. A human writes the ideal answer per evaluation query and a judge compares. Precise, and costly enough that evaluation sets stay small — which is why fifty carefully written guides beat five hundred sloppy ones.
A key-facts list. Rather than prose, the guide is a set of facts the answer must state. Easier to write, much easier to judge always, and it composes naturally with completeness.
Verifiable work. Where the answer is code, a number or a query, correctness can be executed rather than judged. This is the gold standard when it is available and it almost never is for open-ended questions.
The cost is the reason this dimension needs separate treatment. Writing a gold answer takes domain expertise, verifying a model's answer against it takes judgement, and scaling either one takes money or an LLM-as-judge. Which adds its own biases. Exact string match fails at once on natural language. This is because paraphrase is the norm, and n-gram metrics (BLEU, ROUGE) reward surface overlap rather than factual precision. Claim-level judging. Decomposing both the answer and the guide into atomic facts and comparing them — is the approach that handles paraphrase, gives partial credit, and finds. This set fact was wrong. This page walks through each guide format, shows claim-level scoring on a worked case, and covers the failure mode this dimension exists to catch: the grounded-but-wrong answer. Where the model faithfully reports what a fetched document says and that document is out of date.
Why exact match fails, and what replaces it
The obvious automation — string check against the guide — fails at once on natural language. "14 days", "fourteen days" and "two weeks" are the same answer and share no characters. Exact match step by step punishes fluent phrasing.
The n-gram metrics inherited from translation (BLEU, ROUGE) are only a partial improvement: they reward surface overlap. So a wrong answer phrased like the guide can outscore a right answer phrased differently. They remain in use because they are cheap and deterministic. And they should not be trusted as a primary sign for factual correctness.
What works is claim-level judging: decompose both the answer and the guide into atomic facts. Then check each answer claim against the guide. That handles paraphrase, gives partial credit, and tells you which fact was wrong — which a single likeness score never will.
The failure this dimension exists to catch
The key case is the grounded but wrong answer. The model faithfully reports what a fetched document says. And that document is out of date, contradicted by a newer one, or simply mistaken.
Groundedness scores this perfectly. Relevance scores it perfectly. The answer is fluent, cited, and misinforms the user. No amount of prompt engineering fixes it, because the model did just what it was asked; the corpus is the defect.
That makes correctness the dimension that audits your data rather than your model. When correctness is low while groundedness is high, stop looking at the pipeline and go and look at the documents. Stale versions, superseded policies, and copy chunks disagreeing with each other are the usual culprits.
Judging it without fooling yourself
Show the judge the guide. A model asked "is this answer correct?" with no ground truth is being asked to recall the fact itself, and will confidently score wrong answers as right in just the domains where you needed evaluation most.
Ask for partial credit. Binary correct/incorrect throws away the gap between an answer with one wrong detail and one that is fully wrong. Per-claim scoring gives a share.
Calibrate against humans. Have annotators label a sample and measure agreement with your judge. If the judge agrees 70% of the time, a 3-point movement in your correctness score is noise.
Watch for self-choice. A judge tends to favour answers written in its own style. And to prefer longer answers. Both biases are documented and both inflate scores in ways that have nothing to do with being right.
Is the answer right?
Correctness compares the model's answer against a known ground truth. It is the metric stakeholders care about. And the hardest of the RAG metrics to measure well —. This is because natural language has many correct forms.
Question: "How long is parental leave?"
Truth: "39 weeks, of which 6 are at full pay."
Answer A: "You get 39 weeks, with the first 6 paid in full." — correct
Answer B: "Thirty-nine weeks." — correct but incomplete
Answer C: "39 weeks at full pay." — wrong in a detail that matters
Exact string matching marks all three wrong. That is why correctness is often scored by a model or a human rather than by check.
The methods, in ascending order of cost and quality:
| Method | Suits |
|---|---|
| Exact match | Short factual answers, extracted values |
| Token overlap (F1, ROUGE) | Extractive answers |
| Embedding similarity | A loose signal, easily fooled |
| Model as judge | Free-form answers — the practical default |
| Human review | The reference, and the slowest |
Model-as-judge, done carefully
Asking a model to grade another model's answer works well enough to be the standard approach. And it needs care to be trustworthy.
Question: {question}
Reference answer: {truth}
Model answer: {answer}
Does the model answer convey the same information as the reference?
Ignore differences in wording, length and style. Note any factual
contradiction or missing important detail.
Reply with JSON: {"verdict": "correct" | "partially_correct" | "incorrect",
"reason": "<one sentence>"}Four practices separate a useful judge from a noisy one:
Give it the guide answer. Judging correctness without a ground truth is asking the model to know the answer. This reintroduces the problem you were gauging.
Use a discrete scale. Three or four labels are far more steady than a 1–10 score. Where models cluster around 7 and 8.
Need a reason. It improves consistency and makes disagreements auditable.
Validate against humans on a sample. Grade 50 answers manually and measure agreement with the judge. Below about 80% agreement, the judge's absolute numbers are not usable — though relative checks between configurations may still be.
Known biases to guard against: judges prefer longer answers, prefer answers from the same model family, and are sensitive to the order in which candidates are presented. Randomising order and controlling for length help.
Correctness against groundedness
These are different questions and the pair is diagnostic:
| Grounded | Correct | Diagnosis |
|---|---|---|
| Yes | Yes | Working as intended |
| Yes | No | Retrieval returned the wrong document, faithfully used |
| No | Yes | Answered from parametric knowledge — unreliable |
| No | No | Hallucination |
The third row is the one that looks like success and is not. A system that answers correctly from the model's own memory. Rather than from the fetched documents will fail silently as soon as a question moves outside what the model happened to memorise. And no amount of testing on well-known facts will reveal it.
Gauging both is so not redundant. Groundedness tells you whether the system is using its sources; correctness tells you whether the result is right.
Why string matching fails, and what the judge costs
Correctness needs a guide answer, and comparing free text to a guide is harder than it sounds. This scores the same answers three ways. Exact match, token overlap. And a judge -- and shows just which kinds of answer each one gets wrong.
With scores on the table, the next block checks how many answers exact match gets right — and why paraphrase defeats it.
Token overlap relaxes that, but it adds a failure mode of its own. It cannot tell which shared word really carries the answer.
Since no snippet can run an actual model call, the next block lists what a model-as-judge fixes and the practical problems it adds instead.
None of that, but, tells correctness apart from groundedness — the final block lays out the four cases where they diverge.
Things to try with Correctness in LLM evaluation
- Click the digital-goods claim to mark it backed by the context. Groundedness rises to 1.0 and correctness does not move. It is still contradicted by the guide.
- That single claim is the grounded-but-wrong case: cited, confident, and misinforming. It is why correctness cannot be inferred from the context.
- Compare correctness with completeness. An answer can state only true things and still omit half of what was asked.
What to recall
Correctness compares the answer against a guide or verifiable fact, which makes it the only one of the four dimensions that needs ground truth someone has to write. Exact match and n-gram overlap fail on paraphrase; claim-level judging against a key-facts guide is what works. Its distinctive value is catching the grounded-but-wrong answer. Where the model faithfully reports a document that is itself out of date. A defect in your corpus that each other dimension scores as a success.
These four are often measured with an LLM as judge. A strong model is shown the question, the fetched context, the answer and sometimes a guide, and asked to score one dimension at a time. Judging one dimension per call is not a stylistic choice. A single prompt asking for all four gives linked, mushy scores, because the model settles on a general impression and applies it all over.
Two habits make judge scores trustworthy. Ask for a decision and a justification, so a human can audit disagreements. And calibrate against a small human-labelled set before believing any of it. A judge that agrees with your annotators 70% of the time is not gauging what you think it is gauging.
Building the evaluation set
Correctness needs ground truth, and where it comes from decides what the number means.
Hand-written by domain experts. Highest quality. 50 questions with careful guide answers is enough to catch regressions; 200 gives stable checks.
Derived from existing content. FAQ pairs, resolved support tickets, before answered queries. Realistic and cheap where they exist.
Made then reviewed. A model drafts questions and answers from the documents, a human corrects them. The pragmatic route for a new system, and the review step is not optional. Unreviewed made ground truth encodes the generator's errors.
Three things to include on purpose:
Unanswerable questions, where the corpus truly does not hold the answer. The correct response is a refusal, and gauging whether the system refuses is as key as gauging whether it answers.
Multi-hop questions needing two documents. These fail differently and are where most systems are weakest.
Adversarial cases — questions with a false premise, ambiguous questions, questions whose answer changed between document versions.
What correctness does not tell you
Where the failure was. A wrong answer might be a retrieval failure, a prompt failure or a model failure. Correctness alone cannot say, which is why retrieval metrics are measured apart.
Whether the answer was useful. A technically correct answer can be unhelpful. Too terse, badly structured, not fixing what was really asked. Relevance and completeness cover that.
How it behaves on the questions you did not test. A 200-question evaluation set is a sample, and real traffic is broader and stranger.
Whether it is calibrated. A system that is 80% correct and confident each time is more dangerous than one that is 80% correct and flags its doubt.
So correctness sits in a suite alongside groundedness, relevance, completeness and refusal rate — not on its own.
Running it as a regression suite
The practical value comes from automation, not from a one-off measurement.
Run it on each change — chunk size, embedding model, prompt wording, model version. Each of those can move correctness, and without a suite the movement is invisible.
Fix the seed and temperature 0 so a gap between runs is a real gap.
Track per-question results, not just the aggregate. A change that fixes ten questions and breaks eight looks like a small improvement and is worth grasp.
Keep a hard subset of questions the system has failed before, as a regression guard.
Version the evaluation set and record which version gave each score, or checks across time become meaningless.
Questions people ask about Correctness in LLM evaluation
How many questions do I need? 50 to catch large regressions, 200 for stable check between close configurations.
Can I trust a model as judge? For relative checks, often yes. For absolute numbers, validate against human grading on a sample first.
Should I use exact match? Only for short extracted values. Dates, amounts, identifiers. It fails on anything free-form.
What about answers that are correct but incomplete? Use a three-level scale, or measure completeness as a separate metric.
How do I handle questions with a few valid answers? Provide a few guide answers and accept a match against any.
Should the judge be the same model being evaluated? Preferably not. Self-choice bias is documented. Use a different model, or a human sample.
Correctness in LLM evaluation: recap in one screen
- Correctness compares the answer against known ground truth, and exact matching fails on free-form text.
- Model-as-judge is the practical default: give it the guide, use a discrete scale, need a reason, validate against humans.
- Correct-but-ungrounded is a warning, not a success — the system answered from memory.
- Include unanswerable, multi-hop and adversarial questions in the set.
- Run it as a versioned regression suite at temperature 0, tracking per-question results.