Correctness in LLM evaluation
Is the answer right? Correctness compares the answer's claims against a reference answer or against verifiable fact. It is the dimension users care about most and the one that is hardest to automate, because unlike groundedness it cannot be checked against the context — it needs a ground truth that someone has to produce.
Overview
Correct against what, exactly
Correctness is only defined relative to a reference, and choosing that reference is most of the work:
A written gold answer. The usual approach. A human writes the ideal answer per evaluation query and a judge compares. Precise, and expensive enough that evaluation sets stay small — which is why fifty carefully written references beat five hundred sloppy ones.
A key-facts list. Rather than prose, the reference is a set of facts the answer must state. Easier to write, much easier to judge consistently, and it composes naturally with completeness.
Verifiable computation. Where the answer is code, a number or a query, correctness can be executed rather than judged. This is the gold standard when it is available and it almost never is for open-ended questions.
Parameters
Visualisation
—Readout
What to watch
- Correctness is judged against a reference, not against the context.
- A grounded claim can still be wrong if the source document is wrong.
- Click claims to see correctness and groundedness diverge.
Correctness in LLM evaluation: A Practical Guide
What is correctness, and why is it the most expensive dimension to measure?
Why exact match fails, and what replaces it
The obvious automation — string comparison against the reference — fails immediately on natural language. "14 days", "fourteen days" and "two weeks" are the same answer and share no characters. Exact match systematically punishes fluent phrasing.
The n-gram metrics inherited from translation (BLEU, ROUGE) are only a partial improvement: they reward surface overlap, so a wrong answer phrased like the reference can outscore a right answer phrased differently. They remain in use because they are cheap and deterministic, and they should not be trusted as a primary signal for factual correctness.
What works is claim-level judging: decompose both the answer and the reference into atomic facts, then check each answer claim against the reference. That handles paraphrase, gives partial credit, and tells you which fact was wrong — which a single similarity score never will.
The failure this dimension exists to catch
The important case is the grounded but wrong answer. The model faithfully reports what a retrieved document says, and that document is out of date, contradicted by a newer one, or simply mistaken.
Groundedness scores this perfectly. Relevance scores it perfectly. The answer is fluent, cited, and misinforms the user. No amount of prompt engineering fixes it, because the model did exactly what it was asked; the corpus is the defect.
That makes correctness the dimension that audits your data rather than your model. When correctness is low while groundedness is high, stop looking at the pipeline and go and look at the documents: stale versions, superseded policies, and duplicate chunks disagreeing with each other are the usual culprits.
Judging it without fooling yourself
Show the judge the reference. A model asked "is this answer correct?" with no ground truth is being asked to recall the fact itself, and will confidently score wrong answers as right in exactly the domains where you needed evaluation most.
Ask for partial credit. Binary correct/incorrect throws away the difference between an answer with one wrong detail and one that is entirely wrong. Per-claim scoring gives a proportion.
Calibrate against humans. Have annotators label a sample and measure agreement with your judge. If the judge agrees 70% of the time, a 3-point movement in your correctness score is noise.
Watch for self-preference. A judge tends to favour answers written in its own style, and to prefer longer answers. Both biases are documented and both inflate scores in ways that have nothing to do with being right.
Things to try
- Click the digital-goods claim to mark it supported by the context. Groundedness rises to 1.0 and correctness does not move — it is still contradicted by the reference.
- That single claim is the grounded-but-wrong case: cited, confident, and misinforming. It is why correctness cannot be inferred from the context.
- Compare correctness with completeness. An answer can state only true things and still omit half of what was asked.
What to remember
Correctness compares the answer against a reference or verifiable fact, which makes it the only one of the four dimensions that requires ground truth someone has to write. Exact match and n-gram overlap fail on paraphrase; claim-level judging against a key-facts reference is what works. Its distinctive value is catching the grounded-but-wrong answer, where the model faithfully reports a document that is itself out of date — a defect in your corpus that every other dimension scores as a success.
These four are usually measured with an LLM as judge: a strong model is shown the question, the retrieved context, the answer and sometimes a reference, and asked to score one dimension at a time. Judging one dimension per call is not a stylistic choice — a single prompt asking for all four produces correlated, mushy scores, because the model settles on a general impression and applies it everywhere.
Two habits make judge scores trustworthy. Ask for a decision and a justification, so a human can audit disagreements. And calibrate against a small human-labelled set before believing any of it — a judge that agrees with your annotators 70% of the time is not measuring what you think it is measuring.