Retrieval-Augmented Generation

Quick Context

A language model knows what was in its training data, frozen at some cutoff date. It does not know your company's handbook, last week's incident report. Or a product invented for a teaching page. Asked anyway, it will answer — fluently, and wrongly.

Retrieval-augmented generation is the fix that does not involve training anything. Find the passages that look relevant, paste them into the prompt, and ask the question with the evidence attached. The model stops recalling and starts reading.

The pipeline has three stages: indexing, retrieval and generation. At indexing time, documents are split into chunks, each chunk is embedded into a vector. And the vectors are stored in an index. At query time, the question is embedded with the same model, the index returns the closest chunks. And those chunks are injected into the prompt alongside the question. The language model then makes an answer grounded in the retrieved evidence.

Each stage has its own failure modes. Chunking too coarsely means relevant data is buried in noise; chunking too finely means the chunk lacks enough context to be useful on its own. The retrieval step can miss relevant passages if the embedding model does not capture the right notion of similarity, or if the question is phrased differently from the stored text. The generation step can ignore the retrieved evidence fully, hallucinate beyond it, or contradict it. A failure called unfaithfulness.

RAG works because it separates knowledge from reasoning. The language model handles reasoning, and the retrieval system handles knowledge. Updating the knowledge requires re-indexing the documents, not retraining the model. This makes the system far cheaper to maintain than fine-tuning. It also provides attribution: each answer can cite the chunks it was based on. This gives the user a way to verify the response.

The pipeline

1. Chunk. Split the documents into passages small enough to be specific and large enough to stand alone.
2. Embed. Turn each chunk into a vector with an embedding model, and store it.
3. Retrieve. Embed the question the same way and find the nearest chunks, usually by cosine similarity.
4. Augment. Paste the top k chunks into the prompt above the question.
5. Generate. The model answers from what is in front of it, and can cite it.

Only steps 3 to 5 happen per question. Steps 1 and 2 happen once, when the documents change — which is why RAG updates in the time it takes to re-index. Rather than the time it takes to retrain.

The scores on this page are real cosine similarities over word-count vectors rather than a neural embedding. So the retrieval you are watching truly runs. A production system embeds meaning rather than words, which is exactly why it can match "descale" to "limescale" where this cannot.

Giving a model documents instead of retraining it

A language model knows what was in its training data, up to a cut-off date. It does not know your company's handbook, last week's incident report. Or the contents of a customer's account.

Two ways to fix that. Fine-tuning updates the weights on your data — expensive, slow to iterate. And it teaches style and format far better than it teaches facts. Retrieval-augmented generation leaves the model alone and puts the relevant documents into the prompt.

The pipeline has two halves, and the first is done once:

Indexing (offline). Split documents into chunks, embed each chunk into a vector, store the vectors in an index.

Retrieval and generation (per query). Embed the question, find the nearest chunks, paste them into the prompt with the question, and let the model answer from them.

question → embed → search → top k chunks → prompt → answer + citations

Why it is usually the right first choice

The citation row is the one that decides most enterprise deployments. If an answer must be traceable to a source document, retrieval gives you that for free and fine-tuning cannot give it at all.

The access-control row matters nearly as much. Fine-tuning on documents means the model can reproduce them for anyone; retrieval means permissions can be enforced at query time, per user.

Where RAG systems actually fail

Almost never in the language model. The failures are in retrieval, and they have distinct causes:

The right chunk was never retrieved. The most common failure by a wide margin. Poor chunking, an embedding model that does not cover the domain vocabulary. Or a query phrased differently from the document.

The chunk was retrieved but incomplete. A table split down the middle. Or a paragraph whose subject was named in the previous chunk.

The right chunk was buried. Retrieved at position 9 of 10, where models attend to it less. The "lost in the middle" effect.

Contradictory chunks. Two versions of a policy retrieved together, and the model picks one arbitrarily.

The model ignored the context and answered from its parametric knowledge instead.

That distribution is why the practical advice is: measure retrieval apart from generation. Recall@k on a set of question-and-known-answer pairs tells you whether the right chunk is even reaching the model. Debugging the prompt when recall is 0.4 is wasted effort.

Where the quality actually comes from

A RAG pipeline is a chain, and a chain's output is bounded by its weakest link --. This means the arithmetic of stage-by-stage success rates tells you where to spend your effort. This computes that, and then shows why the obvious first instinct is usually the wrong one.

import numpy as np

STAGES = [
    ("chunking", "the answer survives in one chunk", 0.90),
    ("embedding", "the chunk is findable by this query", 0.85),
    ("retrieval", "it lands in the top k", 0.80),
    ("generation", "the model uses it correctly", 0.95),
]

print("A RAG PIPELINE IS A CHAIN. every stage has to succeed:")
print("%-16s %-40s %10s" % ("stage", "succeeds when", "rate"))
for name, cond, p in STAGES:
    print("%-16s %-40s %10.2f" % (name, cond, p))
end_to_end = float(np.prod([p for _, _, p in STAGES]))
print("%-16s %-40s %10.4f" % ("end to end", "all four", end_to_end))
print()
print("   four individually respectable numbers multiply to %.2f. that is"
      % end_to_end)
print("   the arithmetic everyone meets eventually: a chain of 90-percent")
print("   stages is not a 90-percent system.")
print()

print("SO WHERE SHOULD YOU SPEND YOUR TIME? improve each stage by 5 points")
print("in turn and see what happens end to end:")
print("%-16s %12s %14s %14s" % ("improve", "that stage", "end to end", "gain"))
for i, (name, _, p) in enumerate(STAGES):
    ps = [q for _, _, q in STAGES]
    ps[i] = min(1.0, p + 0.05)
    new = float(np.prod(ps))
    print("%-16s %12.2f %14.4f %+14.4f"
          % (name, ps[i], new, new - end_to_end))
print("   every stage gives almost the same gain, because they multiply.")
print("   5 points on ANY factor moves the product by about the same")
print("   proportion, so 'which stage should I improve' has no answer from")
print("   this table alone.")
print()

Building on the same setup, the next block shows the left cases.

import numpy as np
STAGES = [
    ("chunking", "the answer survives in one chunk", 0.90),
    ("embedding", "the chunk is findable by this query", 0.85),
    ("retrieval", "it lands in the top k", 0.80),
    ("generation", "the model uses it correctly", 0.95),
]

print("THE QUESTION THAT DOES HAVE AN ANSWER: which stage is CHEAPEST to")
print("improve, and which is nearest its ceiling?")
print("%-16s %10s %14s %-22s %s"
      % ("stage", "rate", "headroom", "typical fix", "cost"))
FIXES = [("chunking", 0.90, "change the splitter", "an afternoon, re-index"),
         ("embedding", 0.85, "a better model", "re-embed everything"),
         ("retrieval", 0.80, "add a reranker", "latency, per query"),
         ("generation", 0.95, "prompt or model", "cheap to try")]
for (name, p, fix, cost) in FIXES:
    print("%-16s %10.2f %14.2f %-22s %s" % (name, p, 1.0 - p, fix, cost))
print("   now the picture is different. retrieval has the most headroom")
print("   (%.2f) AND a well-understood fix, and generation has the least"
      % (1.0 - 0.80))
print("   (%.2f) while being the cheapest to fiddle with -- which is"
      % (1.0 - 0.95))
print("   exactly why most teams spend their first month on prompts.")
print()

The following block covers the left steps.

import numpy as np
STAGES = [
    ("chunking", "the answer survives in one chunk", 0.90),
    ("embedding", "the chunk is findable by this query", 0.85),
    ("retrieval", "it lands in the top k", 0.80),
    ("generation", "the model uses it correctly", 0.95),
]

print("MEASURE IT BEFORE YOU FIX IT, because the guess is usually wrong.")
print("the cheapest diagnostic is to remove one stage's uncertainty at a")
print("time -- feed the generator the RIGHT chunk by hand:")
print("%-42s %14s %s" % ("experiment", "answer rate", "what it isolates"))
print("%-42s %14.4f %s"
      % ("the pipeline as built", end_to_end, "everything"))
print("%-42s %14.4f %s"
      % ("hand the generator the correct chunk", 0.95, "generation alone"))
print("%-42s %14.4f %s"
      % ("does the right chunk reach the model?", 0.80 * 0.85 * 0.90,
         "everything but generation"))
print("   if answers improve dramatically when you paste the right chunk")
print("   in by hand, the problem is retrieval and no amount of prompt")
print("   work will touch it. if they do NOT improve, the problem is")
print("   generation, and a better retriever is wasted money.")
print("   that one experiment costs an hour and routinely redirects a")

The next block goes on with the left demonstrations.

print("   month of work.")
print()

print("AND THE DECISION THAT COMES BEFORE ALL OF IT: RAG or fine-tuning?")
print("they solve different problems, and the confusion is common:")
print("%-34s %-14s %s" % ("what you need", "RAG", "fine-tuning"))
for row in (("facts that change weekly", "yes", "no"),
            ("citations to a source", "yes", "no"),
            ("per-user access control", "yes", "no"),
            ("a house writing style", "weakly", "yes"),
            ("a domain's vocabulary", "weakly", "yes"),
            ("a new output format", "weakly", "yes"),
            ("lower per-query cost", "no", "yes")):
    print("%-34s %-14s %s" % row)
print("   the split is KNOWLEDGE against BEHAVIOUR. RAG supplies facts at")
print("   query time and can cite them; fine-tuning changes how the model")
print("   writes and cannot be asked where it got something.")
print("   fine-tuning a model on your documents to 'teach it the facts' is")
print("   the expensive way to get an uncitable, undeletable, stale copy")
print("   of a database you already have.")
print()
print("   and they compose: fine-tune for the format and the tone, retrieve")
print("   for the facts. the fine-tuned model is then better at USING the")
print("   retrieved context, which shows up as the generation row in the")
print("   first table rather than as a replacement for any of the others.")

Guided tour

1. Turn retrieval off. Set Chunks Retrieved to 0. The prompt is the bare question, and the answer is what a model does when it has nothing. Something plausible and unsupported, flagged in red.
2. Turn it back on. At k = 1 the top-scoring chunk is pasted in and the answer arrives with a citation. Same model, same question, different prompt.
3. Meet the distractor. Ask about the warranty and read the scores. Chunk 1 wins on 0.213, but chunk 7. The warranty for the previous model. Is right behind it on 0.183,. This is because it talks about warranties too. At k = 1 the ranking has to be right; at k = 2 breadth covers you instead. Retrieval fails quietly, and this is the shape it fails in.
4. Watch the prompt grow. Push k to 6 and the prompt words climb steeply. All of it you retrieve is paid for on every request, in latency and in cost. And the useful chunk gets harder for the model to find among the noise.
5. Ask something the documents do not cover. Choose the voltage question. Retrieval still returns its best matches — it always returns something — but none of them answer it. So the honest response is "not in the documents". A system that cannot say that is a system that will invent.

Why not just fine-tune?

Fine-tuning changes how a model behaves; retrieval changes what it knows right now. Facts that change weekly, documents with access rules, anything that needs a citation. All of those want retrieval. Tone, format, a domain's vocabulary, a stubborn output schema. Those want fine-tuning. They are not competitors, and production systems often use both.

Retrieval also has a property fine-tuning cannot offer: you can show the user where the answer came from, and delete a document and have it truly gone.

Common mistakes

• Chunking badly. Too small and a passage loses the context that made it meaningful; too large and the embedding averages a few topics into mush. Overlapping windows of a few hundred tokens are the usual compromise.
• Assuming retrieval succeeded. Similarity search always returns its best k, but bad they are. Without a relevance floor, an unanswerable question quietly becomes a confident wrong answer.
• Stuffing the context. More chunks is not more accuracy. Models attend unevenly across a long context. And a fact buried in the middle of twenty passages can be missed fully.
• Pure vector search. Embeddings are poor at exact identifiers. Part numbers, error codes, names. Hybrid retrieval, combining keyword search with vectors, is standard for a reason.
• A stale index. The documents changed and nobody re-embedded them. Now the system cites yesterday's policy with complete confidence.

What to remember

RAG answers questions a model was never trained on by finding relevant passages and putting them in the prompt. So generation becomes reading rather than recall. The retrieval step is ordinary similarity search and it always returns something. This makes the quality of your chunking, your ranking and your "I could not find it" path the whole ballgame. Raising k buys robustness against a bad ranking and costs latency, money and attention. So it is a trade rather than a setting. Use retrieval for facts that change or need citing, fine-tuning for behaviour that needs to change, and never assume the top result was actually relevant.

The decisions that find quality

Chunk size. 200–500 tokens is the usual range. Too small and the chunk lacks context; too large and the embedding is a blurred average of a few topics. And the prompt fills up. Overlap of 10–20% keeps sentences that straddle a boundary intact.

Chunking strategy. Splitting on structure — headings, sections, paragraphs — beats splitting on a fixed character count almost always,. This is because it respects where topics actually change.

Hybrid retrieval. Combine dense embedding search with keyword search (BM25). Embeddings catch paraphrase; keywords catch names, codes, numbers and exact terms that embeddings blur. Reciprocal rank fusion merges the two ranked lists. This is the single highest-value improvement in most systems.

Reranking. Retrieve 50 candidates cheaply, then score each against the query with a cross-encoder that reads both together. Far more accurate than embedding similarity, affordable because it runs on 50 items rather than a million.

How many chunks to pass. 3–10 usually. More is not better: irrelevant context degrades answers. And the middle of a long context receives less attention.

Prompting the generation step

Three instructions do most of the work:

Answer using only the provided context. If the context does not contain
the answer, say "I don't know based on the available documents."
Cite the source id after each claim.

Context:
[1] {chunk_1}
[2] {chunk_2}

Question: {question}

"Only the provided context" reduces the model answering from memory, which is what gives confident wrong answers about your domain.

An explicit escape hatch matters more than it looks. Without permission to say "I don't know", a model will construct something plausible from any it was given.

Numbered sources make citations checkable, which is what turns an answer into something an auditor can verify.

Measuring it

Evaluate the two stages apart, then end to end:

Build a small evaluation set. 50–100 real questions with known correct answers and known source documents. Before optimising anything. Without it, changes to chunk size and prompt wording are guesswork. And the usual outcome is a system that feels better and measures worse.

Questions people ask

RAG or fine-tuning? RAG for facts, currency, citations and access control. Fine-tuning for format, tone and task-specific behaviour. They combine.

How large a corpus can this handle? Millions of chunks, with an approximate index. Below about 100,000, a plain vector matrix and a dot product is adequate.

Do long context windows make RAG obsolete? No. Retrieval is cheaper, more precise, and it tells you which source an answer came from. Stuffing a million tokens is expensive and dilutes attention.

Why does it hallucinate even with the right context? Usually the prompt does not constrain it firmly enough. Or the context is contradictory. Or the answer truly is not there and it has no permission to say so.

Which embedding model? Start with a strong general one. Then test domain-specific options on your own evaluation set. The benchmark leaderboard is a weaker signal than your own data.

Should I re-embed when I change models? Yes — vectors from different models are not comparable. So the whole corpus must be re-indexed.

Recap in one screen

• Index documents as embedded chunks; at query time retrieve the nearest and put them in the prompt.
• It adds facts, supports citations and enforces permissions. None of which fine-tuning does well.
• Almost all failures are retrieval failures, so measure recall apart before touching the prompt.
• Hybrid search plus reranking is the highest-value improvement in most systems.
• Instruct the model to use only the context and to admit when the answer is absent.