Abstention in RAG

Why saying "I do not know" is hard

Language models are trained to be helpful. Give one a question and some context, and it will write a fluent answer, even when the context holds nothing useful. That is not a bug. It is the model doing what it was trained to do. Your system has to override it, and no single check is enough.

Three situations call for a refusal. First, nothing relevant was retrieved. A question about vacation policy against a store of engineering docs is the plain case.

Second, the context is close but does not contain the answer. You ask for the maximum PTO carryover and the chunks only describe how PTO accrues.

Third, the model joins facts from two chunks into a conclusion that neither one states. That is multi-hop hallucination.

A good abstention system uses three gates, ordered by cost. A retrieval gate checks whether any chunk is relevant. A prompt tells the model that "I do not know" is a valid answer. A faithfulness check looks at what the model wrote. Cheap checks come first, so the costly one sees the fewest cases.

This page builds all three, shows how to set their thresholds, and covers the trade-off between refusing too often and answering wrongly.

Gate 1: is the retrieval confident?

The first gate runs before the model is called. If no chunk is relevant enough, stop there. You spend nothing on generation.

It checks two things. The top score must clear a minimum, because below it even the best chunk is unlikely to hold the answer. The average of the top three must clear a lower bar, which catches the case of one lucky chunk in a poor result set.

def retrieval_gate(scores, min_top1=0.45, min_top3_avg=0.35):
    if not scores:
        return False, "no results"
    if scores[0] < min_top1:
        return False, f"top1 {scores[0]:.2f} is below {min_top1}"
    top3 = sum(scores[:3]) / min(3, len(scores))
    if top3 < min_top3_avg:
        return False, f"top-3 average {top3:.2f} is below {min_top3_avg}"
    return True, "passed"

tests = [("strong retrieval", [0.82, 0.75, 0.68, 0.55]),
         ("weak top result", [0.38, 0.35, 0.30, 0.28]),
         ("one good, rest bad", [0.55, 0.20, 0.15, 0.10]),
         ("no results", [])]
for name, scores in tests:
    ok, why = retrieval_gate(scores)
    print(f"{name:20s} {'PROCEED' if ok else 'ABSTAIN':8s} {why}")

Cosine score alone is noisy. A chunk on "Python programming" can score 0.65 against a question on "Python snakes". A cross-encoder reranker is more reliable here, because it reads the question and the chunk together. The score you gate on matters as much as the rule. See similarity threshold gating for how to set the number.

Gate 2: let the model refuse

Even when retrieval passes, the chunks may not hold the exact answer. The prompt has to allow a refusal. Add a line such as: "If the context does not contain enough information, reply: I do not have enough information to answer this question."

Without that line, the model fills the gap with a plausible guess. You can also ask it to report its own confidence. That signal is weak on its own, but it adds something the other gates cannot see.

PHRASES = ["i don't have enough information", "i cannot find",
           "the context does not contain", "insufficient information",
           "not able to answer", "no information available"]

def model_abstained(response):
    text = response.lower()
    for phrase in PHRASES:
        if phrase in text:
            return True, phrase
    return False, None

responses = [
    "Employees receive 15 days of PTO annually.",
    "I don't have enough information to answer this question.",
    "The context does not contain details about the carryover limit.",
    "The maximum carryover is 5 days per year.",
]
for r in responses:
    hit, phrase = model_abstained(r)
    print("ABSTAINED" if hit else "ANSWERED ", "|", r[:55], "|", phrase)

Matching phrases is brittle. Ask for a fixed marker instead, such as ABSTAIN on its own line, so you do not depend on wording.

Gate 3: check the answer afterwards

The last gate runs after generation. It splits the answer into claims, then checks each one against the retrieved context. Each claim is marked supported, not supported or contradicted. If more than 20% of the claims are unsupported, reject the answer.

This catches confabulation that got past the prompt. It costs two to four extra model calls, so it goes last.

def faithfulness(verdicts, limit=0.20):
    if not verdicts:
        return True, 1.0
    supported = sum(1 for v in verdicts if v == "SUPPORTED")
    score = supported / len(verdicts)
    return (1 - score) <= limit, score

cases = {
    "good answer":  ["SUPPORTED"] * 4,
    "one made up":  ["SUPPORTED", "SUPPORTED", "NOT_SUPPORTED", "SUPPORTED"],
    "mostly wrong": ["SUPPORTED", "NOT_SUPPORTED", "NOT_SUPPORTED", "CONTRADICTED"],
}
for name, verdicts in cases.items():
    ok, score = faithfulness(verdicts)
    print(f"{name:14s} {'PASS' if ok else 'FAIL'}  {score:.0%} of claims supported")

Notice that one made-up claim out of four fails the check at a 20% limit. That is the intent: a single invented fact is enough to mislead.

The hardest case: multi-hop claims

A model reads "Project X has a $2M budget" in one chunk. It reads "Projects over $1M need VP approval" in another. It then writes "Project X needs VP approval". The inference is reasonable. It is also stated nowhere, and it may be wrong if there are exceptions.

A plain faithfulness check can miss it, because each fact on its own is supported. The hallucination is the join. To catch it, ask the checker a second question: does any claim need two or more chunks combined to reach a conclusion that no single chunk states? If yes, flag it.

How the three gates work together

Run a thousand simulated queries through all three gates and count where each one stops. The code below does that with fixed random seeds, so you can read the shares straight from the output.

import random
random.seed(42)

counts = {"gate1": 0, "gate2": 0, "gate3": 0, "answered": 0}
N = 1000
for _ in range(N):
    if random.uniform(0.2, 0.9) < 0.45:
        counts["gate1"] += 1
    elif random.random() < 0.12:
        counts["gate2"] += 1
    elif random.uniform(0.5, 1.0) < 0.80:
        counts["gate3"] += 1
    else:
        counts["answered"] += 1

for name, n in counts.items():
    print(f"{name:9s} {n:4d}  {n / N:.1%}")
print("model calls saved by gate 1:", counts["gate1"])

The point is the order. Gate 1 costs almost nothing and stops 33.8% of queries before any model call. Gate 2 adds no cost and stops another 7.8%. Gate 3 is the expensive one. Only the 584 queries that survive the first two reach it, and it stops 352 of them. In production, you can run gate 3 in the background: send the answer, and flag it for review if the check fails later.

Setting the thresholds

Too strict, and you refuse questions you could answer. Too loose, and hallucinations pass. You tune this on data, not by feel.

Build a labelled set where each question is marked answerable or not, given your knowledge base. Run the pipeline and count four outcomes: correct answers, correct refusals, false refusals and hallucinated answers. For a general product, aim to keep false refusals under 5% while catching most hallucinations. A medical or legal system can accept more false refusals, up to about 10%, to avoid any hallucination. A support bot can accept fewer.

Expose the thresholds as settings, so each deployment can choose its own balance.

What the user sees

"I do not know" on its own is a dead end. A good refusal says what the system did find and where to look next. For example: "I found how PTO accrues, but not the carryover limit. It may be in section 4.3 of the employee handbook."

That turns a refusal into a signpost. If retrieval was the weak point, corrective RAG can try again with a better query before giving up.

Things to try with Abstention in RAG

1. Set the Top retrieval score to 30. It falls under the Gate 1 minimum, so the system stops before any model call.
2. Set the Share of the answer in the corpus to 10 with every gate passing. The question cannot be answered, and the system answers it anyway.
3. Turn on Model says it does not know. Gate 2 stops the answer.
4. Raise Unsupported claims in the answer to 40 with the Faithfulness limit at 20. Gate 3 rejects it.

What to remember about Abstention in RAG

Models answer even when they should not, so one gate is never enough. Check retrieval confidence first, let the prompt allow a refusal, and verify the finished answer against its sources. Tune the thresholds on answerable and unanswerable questions, and make every refusal point the user somewhere useful.

Questions people ask about Abstention in RAG

Why not let the model decide when to abstain? Models are trained to be helpful, so they answer even with weak context. Self-reported confidence helps, but it misses many cases that the other gates catch.

How many model calls does the faithfulness check add? Two to four: one to extract claims and one to three to check them. That is why it can run in the background.

What false refusal rate is acceptable? Under 5% for general products. Up to about 10% for medical or legal ones.

Does refusing hurt trust? The opposite. Users lose trust when a system is confidently wrong. A refusal with a pointer to the right source builds it.

How do I build the labelled set? Start with 200 questions, half answerable and half not. Label each by hand and add new edge cases as you find them.

Can I skip gate 3? You can run it asynchronously. Send the answer straight away and flag it if the check fails later. You keep the quality signal without the delay.

Abstention in RAG: recap in one screen

• Models answer even without evidence, so build refusal in.
• Gate 1 checks retrieval scores before any model call.
• Gate 2 lets the prompt permit "I do not know".
• Gate 3 splits the answer into claims and checks each against the context.
• Tune thresholds on a labelled set, and make every refusal useful.