Hallucination and Grounding

Quick Context

Ask a model something it has barely seen and it does not go quiet. Next-token prediction produces a distribution over the whole vocabulary, that distribution sums to one, and decoding picks from it. There is no branch in the architecture where "insufficient evidence" lives.

So the mass lands on whatever is most plausible: a real researcher who works in the area, a plausible-looking citation, an API method that ought to exist. The output is fluent because fluency is what was trained, and being wrong does not make it any less fluent.

Where hallucinations come from

• Thin evidence. A fact that appeared once in pretraining is a weak signal competing with strong, generic patterns.
• Sampling. Temperature exists to make text less repetitive, and it spends probability mass on the tail to do so.
• The training objective. Nothing in pretraining rewards abstaining, and preference tuning often rewards a confident, complete answer over a hedge.
• Prompts that presuppose. Ask about a paper that does not exist and the likeliest continuation is a description of it, not a correction.
• Stale or absent knowledge. Anything after the cutoff, or behind a login, was never there to be recalled.

Interactive Exploration Guide

1. Start on thin ice. With evidence at 2.0 the correct name holds about half the mass and three plausible wrong ones hold the other half, so the hallucination rate reads 49.6%. Nothing here is broken; this is what a weak memory looks like written as a distribution.
2. Turn the temperature up. At 2.0 the distribution flattens: the correct answer falls from 50.4% to 36.9% and the hallucination rate climbs past 63%. Temperature is a fluency control that doubles as a hallucination control.Run this
3. Ground it. Tick Ground It With Retrieval. The supporting passage is in the prompt, the correct answer takes almost all the mass, and the hallucination rate collapses. This is why RAG is the first thing anyone reaches for.
4. Let it decline. Turn retrieval off again and tick Allow “I Don’t Know”. The top candidate is 50.4%, under the 0.60 threshold, so the system declines outright: the hallucination rate goes to zero, and so does the chance of getting an answer at all.
5. Now lower the bar. Drag Confidence Threshold down to 0.45. It answers again, and the hallucination rate comes straight back. A system that never answers is safe and useless; a system that always answers is neither. The threshold is where you choose between them.Run this
6. Give it real evidence. Raise Evidence In Training to 5. The correct answer dominates without any of the machinery — models are not unreliable in general, they are unreliable at the edges of what they saw.Run this

What actually helps

• Retrieval with citations. Move the fact from the weights into the prompt, and show where it came from so a reader can check.
• An explicit escape hatch. Tell the model it may answer "not in the provided context", and mean it. Without permission to decline, declining is off-distribution.
• Constrain the output. Enums, schemas and IDs from a real list cannot be invented if the decoder is constrained to them.
• Verify afterwards. Check every claimed citation resolves, every quote appears in the source, every number matches the table. Cheap, and catches the failures that matter.
• Lower the temperature for factual work. Creative writing wants a warm tail; a support answer does not.

What does not help: asking the model whether it is sure. Self-reported confidence is generated by the same process that produced the answer, and is largely uncorrelated with being right.

Key Takeaway

A model hallucinates because it answers by sampling a distribution that has no state for "I do not know" — the mass has to go somewhere, and when the true fact is thinly represented it goes onto whatever is most plausible instead. That makes hallucination a property of the setup rather than a defect to be patched out: thin evidence, high temperature and prompts that presuppose all raise it, while putting the fact in the context, allowing an explicit abstention, constraining the output and verifying claims afterwards all lower it. Every mitigation trades coverage for precision, so the honest question is not "has it stopped hallucinating" but "how often does it answer, and how often is it right when it does".