Groundedness in LLM evaluation

Why it is independent of correctness

This is the distinction people collapse, and the four combinations are all real:

Grounded and correct. The context said it and it is true. What you want.

Grounded and wrong. The context said it and the context is out of date or simply incorrect. The model behaved perfectly; your corpus is the problem. Groundedness scores 1.0 and the user is misinformed.

Ungrounded and correct. The model knew the answer from pretraining and the context never mentioned it. Harmless-looking, and it is the failure mode that erodes trust in a RAG system: you have no idea which answers came from your documents and which came from the model's memory, so the citations mean nothing.

Ungrounded and wrong. A hallucination, in the ordinary sense.

Groundedness catches rows three and four without needing to know the truth of anything. That is why it is cheap: it requires the context and the answer, and no reference and no domain expert.

How it is measured in practice

LLM-as-judge, per claim. Split the answer into claims, then for each ask a strong model whether the context entails it, with the answer justified. This is the standard approach and it works well, because entailment against a supplied passage is a much easier task than open-ended judgement.

Natural-language inference models. A dedicated NLI classifier scores entailment between context and claim. Cheaper and faster than a large judge, weaker on long or technical passages.

Citation checking. Require the model to cite a chunk id per sentence, then verify the cited chunk actually supports it. This has the useful property of being auditable by a human in seconds, and it changes the generator's behaviour for the better even before you measure anything.

What a low score is actually telling you

A groundedness problem is often a retrieval problem wearing a generation costume. If the context did not contain the answer, a model asked to answer anyway will fill the gap from memory — so low groundedness correlates with low recall@k, and the fix is upstream.

The other common cause is a prompt that does not permit abstention. A model told to "answer the question using the context" will answer; one told "if the context does not contain the answer, say so" will often decline correctly. That single sentence moves groundedness more than most model changes, and it is why corrective RAG treats declining as a first-class outcome.

Watch for the degenerate optimum, too. An answer that quotes the context verbatim and says nothing else scores 1.0 on groundedness and may be useless. Groundedness has to be read next to completeness and relevance, or you will optimise your way into a system that recites.

Is every claim supported by the sources?

Groundedness — also called faithfulness — asks a narrow question: does each statement in the answer follow from the retrieved context?

It deliberately does not ask whether the answer is true. Those are separate measurements, and separating them is what makes the pair diagnostic.

Context: "Parental leave is 39 weeks. The first 6 weeks are paid at 90% of salary."

Answer A: "You get 39 weeks, with the first 6 at 90% pay." — grounded

Answer B: "You get 39 weeks, with the first 6 at full pay." — not grounded (the context says 90%)

Answer C: "You get 39 weeks. You must give 8 weeks' notice." — the second sentence is unsupported, even if true

Answer C is the case that matters most in practice. The claim may well be correct, and it did not come from the provided documents — so the citation is misleading and the system will produce similar unsupported claims where they happen to be wrong.

Measuring it, claim by claim

Groundedness is scored per claim rather than per answer, because an answer is usually part supported and part not.

The standard procedure:

1. Decompose the answer into atomic claims.
2. Check each claim against the retrieved context.
3. Score as the share of claims that are supported.

Context:
{retrieved_chunks}

Claim: {claim}

Is this claim fully supported by the context above? Answer "supported",
"contradicted" or "not_found", with the sentence from the context that
supports it if applicable. Reply as JSON.

groundedness = supported claims / total claims

The decomposition step is where the difficulty lies. "Parental leave is 39 weeks, of which the first 6 are at 90% pay, and it must be requested 8 weeks in advance" is three claims, and scoring it as one loses the information that two are supported and one is not.

Frameworks such as RAGAS implement this pipeline with a model as judge. The three-way verdict — supported, contradicted, not found — is more useful than a binary one, because contradiction and absence have different causes: contradiction usually means the model mis-read the context, absence means it drew on its own knowledge.

The four-way diagnosis

Groundedness is most useful read against correctness:

The third row deserves emphasis because it presents as success. A system answering correctly from parametric memory rather than from its sources passes any correctness test built on well-known facts, and fails the moment a question concerns something specific to your organisation — which is the entire reason the system exists.

Measuring groundedness is how you detect that before it reaches users.

Scoring an answer against its sources, claim by claim

Groundedness asks whether every statement in an answer is supported by the retrieved context, and it is the one RAG metric that can be checked without knowing the right answer. This runs the claim-level procedure and shows why the sentence-level shortcut disagrees with it.

import numpy as np

CONTEXT = [
    "Refunds are available within 30 days of purchase.",
    "The item must be returned unused.",
    "Refunds are issued to the original payment method.",
]

# an answer, broken into the claims it makes, each marked with whether
# the context actually supports it
ANSWER_CLAIMS = [
    ("You have 30 days to request a refund.", True),
    ("The item must be unused.", True),
    ("The refund goes back to your original card.", True),
    ("You will also receive a 10 dollar credit.", False),
    ("Processing takes 2 business days.", False),
]

print("THE RETRIEVED CONTEXT:")
for c in CONTEXT:
    print("   - %s" % c)
print()
print("THE MODEL'S ANSWER, broken into the claims it makes:")
print("%-46s %14s" % ("claim", "in the context?"))
for text, ok in ANSWER_CLAIMS:
    print("%-46s %14s" % (text, "yes" if ok else "NO"))
supported = sum(1 for _, ok in ANSWER_CLAIMS if ok)
g = supported / float(len(ANSWER_CLAIMS))
print()
print("GROUNDEDNESS = supported claims / total claims = %d / %d = %.4f"
      % (supported, len(ANSWER_CLAIMS), g))
print("   the last two claims are fluent, plausible, and appear nowhere in")
print("   the context. that is what an ungrounded claim looks like -- not")
print("   nonsense, but detail the model supplied itself.")
print()

print("WHAT THIS METRIC DOES NOT MEASURE, and the distinction matters:")
print("%-30s %-34s %s" % ("metric", "compares", "needs a golden answer?"))
for row in (("groundedness", "answer against the context", "no"),
            ("correctness", "answer against the truth", "yes"),
            ("relevance", "answer against the question", "no"),
            ("context precision", "context against the question", "no")):
    print("%-30s %-34s %s" % row)
print("   groundedness is cheap precisely because it needs no golden")
print("   answer. it can run on production traffic, continuously, on")
print("   questions nobody has ever labelled.")
print("   the price is that a perfectly grounded answer can be WRONG:")
print("      if the context says 30 days and the real policy is 14, an")
print("      answer saying 30 days is fully grounded and misinforms the")
print("      user. groundedness measures faithfulness to the retrieved")
print("      text, and nothing about whether that text was right.")
print()

print("NOW THE SHORTCUT PEOPLE ACTUALLY IMPLEMENT, and where it goes")
print("wrong. instead of extracting claims, score each SENTENCE by its")
print("similarity to the context:")
AX = ["days", "unused", "payment", "credit", "processing"]
CTX_VEC = np.array([
    [0.95, 0.05, 0.05, 0.00, 0.05],
    [0.05, 0.95, 0.00, 0.00, 0.00],
    [0.05, 0.00, 0.95, 0.05, 0.05],
], float)
SENT_VEC = np.array([
    [0.95, 0.05, 0.05, 0.00, 0.05],     # 30 days       -- supported
    [0.05, 0.95, 0.00, 0.00, 0.00],     # unused        -- supported
    [0.05, 0.00, 0.92, 0.05, 0.05],     # original card -- supported
    [0.10, 0.00, 0.60, 0.75, 0.05],     # 10 dollar credit -- NOT
    [0.92, 0.03, 0.05, 0.00, 0.30],     # 2 days        -- NOT, but close
], float)

def unit(v):
    return v / np.linalg.norm(v)

print("%-46s %14s %14s" % ("claim", "max sim", "verdict at 0.80"))
flags = []
for (text, truth), v in zip(ANSWER_CLAIMS, SENT_VEC):
    s = max(float(unit(v) @ unit(c)) for c in CTX_VEC)
    flags.append((s >= 0.80, truth))
    print("%-46s %14.4f %14s"
          % (text[:44], s, "grounded" if s >= 0.80 else "UNSUPPORTED"))
agree = sum(1 for p, t in flags if p == t)
print("   the similarity method agrees with the claim-level answer on %d"
      % agree)
print("   of %d claims -- and the one it gets wrong is the one that" % len(flags))
print("   matters." if agree < len(flags) else "   claims.")
for (text, truth), (pred, _) in zip(ANSWER_CLAIMS, flags):
    if pred != truth:
        print("      DISAGREES on: %s" % text)
        print("         truth: %s, similarity says: %s"
              % ("supported" if truth else "not supported",
                 "grounded" if pred else "unsupported"))
print("   similarity measures TOPIC OVERLAP, and an invented detail is")
print("   usually on topic. '2 business days' talks about days, and the")
print("   context talks about days -- so a threshold on similarity is")
print("   asking a question that cannot separate 'the context says this'")
print("   from 'the context is about this'.")
print()

print("WHY CLAIM EXTRACTION IS THE EXPENSIVE PART. one answer needs:")
print("%-40s %s" % ("1. split the answer into claims", "one LLM call"))
print("%-40s %s" % ("2. check each claim against the context",
                    "one call per claim"))
print("%-40s %s" % ("total for this answer",
                    "%d calls" % (1 + len(ANSWER_CLAIMS))))
print("   at %s answers a day that is %s calls, which is why groundedness"
      % ("{:,}".format(10000), "{:,}".format(10000 * (1 + len(ANSWER_CLAIMS)))))
print("   is usually run on a SAMPLE rather than on everything, and why")
print("   the cheap similarity version keeps getting reached for despite")
print("   disagreeing with the real measure.")
print()

print("A MIDDLE PATH worth knowing: use the similarity check as a FILTER,")
print("not as the answer. it is cheap and it has few false negatives:")
fn = sum(1 for p, t in flags if t and not p)
fp = sum(1 for p, t in flags if p and not t)
print("      false negatives (called it ungrounded, it was fine): %d" % fn)
print("      false positives (called it grounded, it was not):     %d" % fp)
print("   so send only the claims it flags to the expensive checker,")
print("   and accept the rest. here that would be %d of %d claims sent"
      % (sum(1 for p, _ in flags if not p), len(flags)))
print("   on, at %.0f%% of the LLM cost, and it would still miss the %d"
      % (100.0 * sum(1 for p, _ in flags if not p) / len(flags), fp))
print("   claim%s the similarity check wrongly approved."
      % ("" if fp == 1 else "s"))
print("   that residual is the honest cost of the shortcut, and it is")
print("   worth stating in whatever dashboard reports the number.")

Things to try

1. Click the digital-goods claim, which the context does not support. Groundedness falls while correctness is unaffected — the two dimensions move independently.
2. Flip every claim to supported. Groundedness reaches 1.0 while relevance and completeness stay where they were: a perfectly grounded answer can still be padded and incomplete.
3. Note that the 24/7 phone claim is marked correct but off topic. True, plausible, and not what was asked — groundedness alone would not catch it.

What to remember

Groundedness is the proportion of an answer's claims that the retrieved context actually supports. It is the direct measure of hallucination, it needs no reference answer, and it is independent of truth — a claim can be grounded and wrong, or true and ungrounded. Measure it per claim rather than per answer so it tells you which span to look at. A low score usually means a retrieval failure or a prompt that does not allow the model to decline. Never optimise it alone, because verbatim quotation scores perfectly.

These four are usually measured with an LLM as judge: a strong model is shown the question, the retrieved context, the answer and sometimes a reference, and asked to score one dimension at a time. Judging one dimension per call is not a stylistic choice — a single prompt asking for all four produces correlated, mushy scores, because the model settles on a general impression and applies it everywhere.

Two habits make judge scores trustworthy. Ask for a decision and a justification, so a human can audit disagreements. And calibrate against a small human-labelled set before believing any of it — a judge that agrees with your annotators 70% of the time is not measuring what you think it is measuring.

What improves it

Better retrieval, first. Most ungrounded answers are the model doing its best with context that did not contain the answer. If recall@k is 0.6, groundedness cannot be high — and prompt work will not fix it.

An explicit refusal option. A model with no permission to say "not in the documents" will always produce something. One sentence in the prompt removes a large class of unsupported claims.

Required citations. Asking for a source id per claim makes unsupported statements structurally awkward and independently verifiable.

Fewer, better chunks. Contradictory or irrelevant context actively causes unsupported claims. Three relevant chunks beat ten containing three.

Temperature 0. Sampling variation in factual output is error.

Structured output. Requiring JSON with a source field per claim makes the constraint explicit.

The prompt that does most of this work:

Answer using only the context below. Cite the source number after each
claim. If the context does not contain the answer, reply exactly:
"Not found in the provided documents."

The judge's limitations

Groundedness scored by a model has known weaknesses, and knowing them determines how much weight to put on the number.

Implicit inference. If the context says "39 weeks" and the answer says "just under 10 months", is that supported? A strict judge says no; a reasonable reader says yes. Judges are inconsistent on paraphrase and arithmetic.

Multi-hop support. A claim following from two chunks combined may be marked unsupported by a judge examining chunks individually.

Decomposition quality. Badly split claims produce meaningless verdicts.

Judge bias. Judges tend to be lenient towards fluent text and towards answers from the same model family.

The mitigation is the same as for any model-graded metric: validate against human judgement on a sample of 50, and if agreement is below about 80%, treat the absolute number as unusable while relative comparisons between configurations may still hold.

Where it sits in the metric suite

Groundedness is the one that specifically measures whether the retrieval-augmented part of the system is doing anything. A high correctness score with low groundedness means you have built an expensive pipeline whose documents the model is ignoring.

Questions people ask

Is groundedness the same as correctness? No. Groundedness asks whether claims follow from the context; correctness asks whether they are true. Both are needed.

Can an answer be grounded but wrong? Yes — if the retrieved document is wrong or out of date. That is a retrieval or corpus problem, not a generation one.

How do I decompose an answer into claims? With a model, prompted to split into atomic factual statements. It is imperfect and works well enough in practice.

Should a refusal count as grounded? Yes — there are no unsupported claims. Track refusal rate separately so over-refusal is visible.

What score should I aim for? Above 0.9 for systems where citations matter. Below 0.7 indicates the model is regularly drawing on its own knowledge.

Does it detect subtle numeric errors? Inconsistently — "39 weeks" against "40 weeks" is often caught, unit and percentage changes less so. Worth spot-checking numbers separately.

Recap in one screen

• Groundedness asks whether every claim follows from the retrieved context, not whether it is true.
• Score it per claim: decompose the answer, check each against the context, take the supported share.
• Correct-but-ungrounded means the model answered from memory — it looks like success and fails silently.
• Better retrieval, an explicit refusal option and required citations are what move it.
• Validate the model judge against human grading before trusting absolute numbers.