Hallucination and Grounding

Where hallucinations come from

• Thin evidence. A fact that appeared once in pretraining is a weak signal competing with strong, generic patterns.
• Sampling. Temperature exists to make text less repetitive, and it spends probability mass on the tail to do so.
• The training objective. Nothing in pretraining rewards abstaining, and preference tuning often rewards a confident, complete answer over a hedge.
• Prompts that presuppose. Ask about a paper that does not exist and the likeliest continuation is a description of it, not a correction.
• Stale or absent knowledge. Anything after the cutoff, or behind a login, was never there to be recalled.

Why a model invents things

A language model is trained to produce text that looks like its training data. It is not trained to be true, and it has no mechanism for checking.

So when it does not know something, it does not stop — it produces the most plausible continuation. A fabricated citation looks exactly like a real one, because "author, year, journal" is a pattern the model has seen tens of thousands of times.

That is the whole explanation, and it has an important consequence: fluency and accuracy are unrelated. The confident tone is a property of the text distribution, not evidence about the content.

The recognisable categories:

The fourth is the one that matters most in RAG systems, and the one that is measurable.

Grounding: constraining the answer to sources

Grounding means every claim in the output traces to a supplied document. It is the main defence, and it works at three levels.

Retrieval. Put the relevant documents in the context. If the answer is not there, no prompt will conjure it.

Prompting. Instruct the model to use only the context, to cite the source of each claim, and — crucially — to say when the answer is absent.

Answer using only the context below. Cite the source number after each
claim. If the context does not contain the answer, reply exactly:
"Not found in the provided documents."

Context:
[1] {chunk}
[2] {chunk}

Question: {question}

Verification. Check the output against the context afterwards, either with a model asked "is this claim supported by this passage?" or by matching cited spans.

The escape hatch is the part people leave out. A model given no permission to fail will always produce something, and what it produces will be plausible.

Measuring it

Two metrics, and they answer different questions:

Groundedness (faithfulness) — is every claim in the answer supported by the retrieved context? Measured by splitting the answer into claims and checking each against the context, usually with a model as judge.

Correctness — is the answer right, against a known ground truth?

They can diverge, and the divergence is informative:

The third row is the dangerous one, because it looks like success. A system that answers correctly from parametric knowledge rather than from sources will fail silently the moment the question moves outside what the model memorised.

Why a model invents things, and what actually reduces it

A model hallucinates because its objective rewards a plausible next token and never rewards silence. That is a property of the training signal, not a bug, and it explains which interventions work and which cannot.

import numpy as np

VOCAB = ["1998", "2001", "2004", "2011", "I", "do", "not", "know"]
# what the model has learned about "the company was founded in ___"
# when it has genuinely seen the fact, and when it has not
KNOWN = np.array([0.5, 1.2, 8.4, 0.6, -2.0, -2.5, -3.0, -2.8])
UNKNOWN = np.array([2.1, 2.3, 2.0, 2.2, -1.9, -2.4, -2.9, -2.7])

def softmax(z):
    z = np.asarray(z, float)
    z = z - z.max()
    e = np.exp(z)
    return e / e.sum()

print("'THE COMPANY WAS FOUNDED IN ___'. two situations, and the model's")
print("output in each:")
print("%-12s %20s %20s" % ("token", "the fact was learned", "it was not"))
pk, pu = softmax(KNOWN), softmax(UNKNOWN)
for t, a, b in zip(VOCAB, pk, pu):
    print("%-12s %20.6f %20.6f" % (t, a, b))
print()
print("   in the second column the model is genuinely uncertain -- the")
print("   four years are nearly tied at about %.2f each. and yet the mass"
      % pu[0])
print("   on 'I do not know' is %.6f."
      % sum(pu[i] for i, t in enumerate(VOCAB) if t in ("I", "do", "not", "know")))
print("   THAT is the whole mechanism. the model is uncertain and has no")
print("   way to SAY so, because during training every example had a")
print("   continuation and none of them was an admission of ignorance.")
print("   sampling from that distribution produces a year -- a specific,")
print("   confident, fluent, wrong year -- about %.0f%% of the time."
      % (100 * sum(pu[:4])))
print()

print("AND THE ENTROPY KNOWS, even though the output does not:")
def H(p):
    return float(-(p * np.log(p + 1e-12)).sum())
print("%-24s %14s %16s" % ("", "entropy", "top probability"))
print("%-24s %14.4f %16.4f" % ("the fact was learned", H(pk), pk.max()))
print("%-24s %14.4f %16.4f" % ("it was not", H(pu), pu.max()))
print("   the uncertainty is measurable, at the token that matters, for")
print("   free. that is the basis of every confidence-based abstention")
print("   scheme: watch the entropy at the tokens carrying the facts and")
print("   refuse when it is high.")
print("   it is also why those schemes are unreliable -- a model that")
print("   confidently learned something WRONG has low entropy too, and")
print("   this signal cannot tell the two apart.")
print()

print("WHAT REDUCES IT, and the ordering is not what most teams try first:")
INTERVENTIONS = [
    ("retrieve the fact and put it in context", "large", "the model no longer has to recall"),
    ("ask for citations, then verify them", "large", "unverifiable claims become visible"),
    ("let the model say 'I do not know'", "moderate", "gives the mass somewhere to go"),
    ("lower the temperature", "small", "changes sampling, not the distribution"),
    ("say 'do not hallucinate' in the prompt", "none", "it cannot detect its own ignorance"),
]
print("%-44s %10s %s" % ("intervention", "effect", "why"))
for name, eff, why in INTERVENTIONS:
    print("%-44s %10s %s" % (name, eff, why))
print()
print("   the last row is worth being blunt about. the model has no")
print("   internal flag saying 'this next token is invented'. asking it")
print("   not to hallucinate asks it to detect a state it cannot observe.")
print("   what such a prompt DOES do is shift the style toward hedging,")
print("   which reads like caution and is not the same thing.")
print()
print("   and the temperature row deserves the arithmetic, because it is")
print("   the intervention people reach for first:")
def soft_T(z, T):
    z = np.asarray(z, float) / T
    z = z - z.max()
    e = np.exp(z)
    return e / e.sum()
print("%-14s %22s %22s" % ("temperature", "P(some year)", "P(the right one)"))
for T in (0.1, 0.5, 1.0, 1.5):
    q = soft_T(UNKNOWN, T)
    print("%-14.1f %22.4f %22.4f" % (T, q[:4].sum(), q[2]))
print("   at every temperature the answer is a year with probability")
print("   %.2f or higher. temperature decides WHICH invented year you get,"
      % min(soft_T(UNKNOWN, T)[:4].sum() for T in (0.1, 0.5, 1.0, 1.5)))
print("   not whether you get one. at temperature 0 the model returns the")
print("   single most likely year every time -- deterministically wrong")
print("   rather than randomly wrong, which is worse for detection because")
print("   asking twice no longer reveals the uncertainty.")
print()

print("GROUNDING IS THE INTERVENTION THAT WORKS, and it works by changing")
print("the question the model is answering:")
print("%-30s %s" % ("without context", "'what year was it founded?'"))
print("%-30s %s" % ("with context", "'what year does this text say?'"))
print("   the second is a reading task, and models are far better at")
print("   reading than at recalling. that is the entire mechanism of RAG's")
print("   effect on hallucination, and it is why it works even with a")
print("   model that never saw your documents in training.")
print("   what it does NOT do is eliminate the problem:")
print("%-40s %s" % ("if retrieval returns nothing", "the model recalls again"))
print("%-40s %s" % ("if retrieval returns the wrong thing", "grounded and wrong"))
print("%-40s %s" % ("if the answer needs two documents", "a plausible bridge"))
print("   that last one is the subtle failure. given two facts and a gap")
print("   between them, a model will produce the connective tissue, and")
print("   the invented part is the JOIN rather than any of the facts --")
print("   which is exactly the kind of hallucination a claim-level")
print("   groundedness check catches and a document-level one does not.")

Guided tour

1. Start on thin ice. With evidence at 2.0 the correct name holds about half the mass and three plausible wrong ones hold the other half, so the hallucination rate reads 49.6%. Nothing here is broken; this is what a weak memory looks like written as a distribution.
2. Turn the temperature up. At 2.0 the distribution flattens: the correct answer falls from 50.4% to 36.9% and the hallucination rate climbs past 63%. Temperature is a fluency control that doubles as a hallucination control.
3. Ground it. Tick Ground It With Retrieval. The supporting passage is in the prompt, the correct answer takes almost all the mass, and the hallucination rate collapses. This is why RAG is the first thing anyone reaches for.
4. Let it decline. Turn retrieval off again and tick Allow “I Don’t Know”. The top candidate is 50.4%, under the 0.60 threshold, so the system declines outright: the hallucination rate goes to zero, and so does the chance of getting an answer at all.
5. Now lower the bar. Drag Confidence Threshold down to 0.45. It answers again, and the hallucination rate comes straight back. A system that never answers is safe and useless; a system that always answers is neither. The threshold is where you choose between them.
6. Give it real evidence. Raise Evidence In Training to 5. The correct answer dominates without any of the machinery — models are not unreliable in general, they are unreliable at the edges of what they saw.

What actually helps

• Retrieval with citations. Move the fact from the weights into the prompt, and show where it came from so a reader can check.
• An explicit escape hatch. Tell the model it may answer "not in the provided context", and mean it. Without permission to decline, declining is off-distribution.
• Constrain the output. Enums, schemas and IDs from a real list cannot be invented if the decoder is constrained to them.
• Verify afterwards. Check every claimed citation resolves, every quote appears in the source, every number matches the table. Cheap, and catches the failures that matter.
• Lower the temperature for factual work. Creative writing wants a warm tail; a support answer does not.

What does not help: asking the model whether it is sure. Self-reported confidence is generated by the same process that produced the answer, and is largely uncorrelated with being right.

What to remember

A model hallucinates because it answers by sampling a distribution that has no state for "I do not know" — the mass has to go somewhere, and when the true fact is thinly represented it goes onto whatever is most plausible instead. That makes hallucination a property of the setup rather than a defect to be patched out: thin evidence, high temperature and prompts that presuppose all raise it, while putting the fact in the context, allowing an explicit abstention, constraining the output and verifying claims afterwards all lower it. Every mitigation trades coverage for precision, so the honest question is not "has it stopped hallucinating" but "how often does it answer, and how often is it right when it does".

What reduces it, ranked by effect

1. Better retrieval. The largest single factor. Most "hallucinations" in a RAG system are the model doing its best with context that did not contain the answer. Measure recall@k first.

2. An explicit "I don't know" option. Costs one sentence in the prompt and removes a large class of fabrication.

3. Required citations. Asking for a source per claim makes the answer checkable, and appears to reduce unsupported claims by itself.

4. Temperature 0. Sampling introduces variation, and variation in factual output is error.

5. Smaller, cleaner context. Contradictory or irrelevant chunks actively cause wrong answers. Fewer, better chunks beat more.

6. A verification pass. A second call asking whether each claim is supported. Doubles cost, catches a lot.

7. Structured output. Requiring JSON with a source field per claim makes unsupported statements structurally awkward to produce.

Note what is not on that list: telling the model to "be accurate" or "do not hallucinate". Instructions of that kind have little measurable effect, because the model is not knowingly inventing.

Where it cannot be fixed by prompting

Some limitations are structural and need a different tool:

Arithmetic and counting. Tokenisation fragments numbers, so calculation is unreliable. Give the model a calculator tool.

Current events past the training cut-off. No amount of prompting supplies information the model does not have. Retrieval or a search tool.

Precise quotations from memory. Verbatim recall is unreliable. Retrieve the source and quote from it.

Its own uncertainty. A model's stated confidence is poorly calibrated, and asking "how sure are you?" produces a plausible number rather than a measured one.

The general principle: if a fact must be right, it should come from a source in the context, not from the weights.

Questions people ask

Can hallucination be eliminated? Not entirely with current models. It can be reduced substantially and, importantly, made detectable through citations and verification.

Do bigger models hallucinate less? Generally yes on well-covered topics, and they remain confident when wrong, which can make errors harder to spot.

Does RAG solve it? It addresses the root cause for domain facts, and only if retrieval works. A RAG system with poor recall hallucinates while appearing grounded.

How do I detect it automatically? A model-as-judge groundedness check per claim, or entailment models. Neither is perfect; both are far better than nothing.

Does temperature 0 stop it? It removes sampling variance, not fabrication. A deterministic wrong answer is still wrong.

Why does it invent citations specifically? Because citation format is a very strong pattern in the training data, and filling a pattern is exactly what the model does.

Recap in one screen

• Models produce plausible text, not true text, and fluency carries no information about accuracy.
• Grounding constrains answers to supplied documents: retrieve, instruct, verify.
• Give an explicit "not in the documents" option — without one, the model will always produce something.
• Groundedness and correctness are different metrics, and a correct-but-ungrounded answer is a warning, not a success.
• Better retrieval reduces hallucination more than any prompt wording.