Context-aware chunking

Every chunking strategy answers "where do I cut?". This one answers a different question: what did this chunk lose by being cut? A fragment full of "it" and "the above" is unretrievable and unusable, so context is added back — a title, a heading path, a generated summary, resolved pronouns.

Overview

Every chunking strategy answers "where do I cut?".

In this guide
  1. The problem is coreference, not boundaries
  2. What gets added
  3. What it costs, and what to watch
  4. Contextual retrieval, and what it costs
  5. Embedded text and returned text can differ
  6. Measuring whether the enrichment helped
  7. The missing-context problem
  8. Four ways to add the missing context
  9. Contextual retrieval, concretely
  10. Testing whether a chunk can stand alone
  11. Things to try with Context-aware chunking
  12. What to recall
  13. Choosing among them
  14. Testing whether a chunk stands alone
  15. Common mistakes with Context-aware chunking
  16. Questions people ask about Context-aware chunking
  17. Context-aware chunking: recap in one screen

Parameters

Visualisation

—

Readout

What to watch

  • The boundary is unchanged; what is stored changes.
  • Pronouns and back-references are what make a raw chunk unusable.
  • The enrichment goes into the embedded text, so retrieval improves too.

Context-aware chunking: A Practical Guide

What is context-aware chunking, and what problem does it solve that the other strategies do not?

The problem is coreference, not boundaries

Prose is written to be read in order, so it leans on all of it before it: "it", "this policy", "as described above", "the latter". Cut one paragraph out and those references dangle.

Two results. The generator cannot answer from the chunk because it does not know what "it" is. And retrieval never surfaces the chunk in the first place,. This is because the embedding is of text that does not hold the words a user would search for.

The fix is to add context that the prose assumes. The cheapest approach is prepending the document title and heading path to each chunk before embedding — deterministic, no model call, and often the largest single improvement. A chunk from "Authentication > OAuth2 > Token Refresh" that opens with "This process repeats every hour" becomes far more retrievable when its heading trail is prepended.

A more powerful approach is made context: an LLM writes a sentence situating the chunk within the document. And that sentence is prepended before embedding. Anthropic's contextual retrieval paper showed this improves retrieval precision much. The cost is one model call per chunk at indexing time. This prompt caching over the shared document makes affordable for most corpus sizes.

Neighbour context stores the previous and next chunks alongside each chunk for generation but does not embed them. The retrieval model still sees only the target chunk (keeping the embedding focused). But the generator gets a wider window. This is the idea behind the parent-document retriever. Retrieve the small, precise chunk but return the larger parent to the model.

The trade-off in all three approaches is dilution. Prepending 200 tokens of context to a 300-token chunk means the embedding is partly about the context. So chunks from the same section start looking alike to the retriever. The window between "too little context" and "too much" is narrow and corpus-dependent. Measuring retrieval recall on a held-out set is the only reliable way to tune it.

What gets added

Cheap and free: document title and heading path, prepended. Deterministic, no model call, and often the largest single improvement.

Neighbour context: the previous and next chunk, or a window around the chunk, stored for generation but not embedded. The parent document retriever is exactly this idea, retrieving small and returning large.

Made context: an LLM writes a sentence situating the chunk in the document, prepended before embedding. Anthropic's "contextual retrieval" is this, and it works well. It costs one model call per chunk at index time. This prompt caching over the shared document makes affordable.

What it costs, and what to watch

Index time and money, both once. The subtler cost is dilution: prepending 200 tokens of context to a 300-token chunk means the embedding is largely about the context. So chunks from the same section start looking alike and the retriever loses its ability to distinguish them.

Keep the enrichment short relative to the chunk, and measure on a retrieval evaluation set rather than assuming. This is the plan with the best evidence behind it and it is not free of trade-offs.

Contextual retrieval, and what it costs

The strongest published form of this is to have an LLM write a short sentence situating each chunk inside its document. What it is about, what it follows. And prepend that before embedding. Reported reductions in retrieval failure are large. And the technique is simple enough to implement in an afternoon.

The cost is one model call per chunk at index time. This sounds prohibitive on a large corpus and mostly is not. The document is the same for every chunk in it. So prompt caching over that shared prefix makes the marginal call cheap. It is the clearest practical case of two of these techniques composing.

It is still an index-time cost paid on every re-index, so it belongs in your pipeline design. Rather than being bolted on: changing the chunker means regenerating every context sentence.

Embedded text and returned text can differ

The enrichment does not have to be what the model reads. You can embed the enriched text — so the chunk is findable — while storing and returning the first. So the generator is not fed repetitive boilerplate.

Separating the two is the general form of a pattern that appears everywhere in retrieval. Embed something optimised for matching, return something optimised for reading. Parent-document retrieval embeds a small chunk and returns its parent. Summary indexing embeds a summary and returns the document. Contextual retrieval embeds chunk-plus-context and can return either.

Once you see the pattern, the design question stops being "how should I chunk?" and becomes two questions with different answers. What should be matched against, and what should be read?

Measuring whether the enrichment helped

Enrichment is easy to add and easy to overdo. And the only way to tell which you have done is to measure both halves.

Did findability improve? Recall@k on a fixed query set, before and after. This is the number the technique is supposed to move. And it often does.

Did distinguishability degrade? The failure mode is chunks from one section becoming alike. A cheap proxy is the mean pairwise similarity between chunks that share a parent. If it climbs sharply after enrichment, the shared context is ruling and the retriever is losing its ability to pick between them.

Watching only the first will lead you to enrich more and more,. This is because recall keeps improving right up until the point where the top-k fills with near-the same chunks from the same section. At which point recall still looks fine and answers get worse.

The missing-context problem

A chunk is retrieved and passed to the model on its own. Often it cannot be understood on its own:

"This must be requested at least 8 weeks in advance, and the form needs a manager's signature."

What must? Requested from whom? The chunk before it said "Parental leave", and that chunk is not in the context.

This is the central weakness of chunking, and it has nothing to do with where the boundaries fall — even a perfectly-placed boundary gives chunks that reference their surroundings. Documents are written to be read in order; chunks are retrieved out of order.

Context-aware chunking is the family of techniques that give each chunk enough surrounding data to stand alone.

Four ways to add the missing context

TechniqueWhat is addedCost
Heading path prefixThe document's section hierarchyFree
OverlapThe neighbouring boundary textStorage
Parent-childRetrieve small, return the parent sectionStorage, complexity
Contextual summaryA generated sentence describing the chunk's placeOne model call per chunk

The heading path is the cheapest and often the most effective. Prepending "Handbook > Leave > Parental Leave > Applying" to that chunk resolves "this" at once, improves the embedding, and costs nothing beyond capturing the structure at indexing time.

Overlap of 10–20% means a sentence spanning a boundary appears whole in at least one chunk. It handles the boundary case and not the "referenced three paragraphs earlier" case.

Parent-child indexes small chunks for precise matching and returns the larger enclosing section for context. It resolves references within the section by construction.

Contextual retrieval is the newest of the four: ask a model, at indexing time, to write a sentence or two situating each chunk within its document, and prepend that to the chunk before embedding.

Contextual retrieval, concretely

The prompt is short and the effect is large:

Here is a document:
<document>{full_document}</document>

Here is a chunk from it:
<chunk>{chunk_text}</chunk>

Write 1-2 sentences of context situating this chunk within the document,
so it can be understood on its own. Reply with the context only.

The made context is prepended to the chunk text, and the combined text is embedded and stored.

Made: "This passage is from the parental leave section of the employee handbook, describing the use process for eligible employees."
Chunk: "This must be requested at least 8 weeks in advance..."

Now the chunk is retrievable by a query about applying for parental leave, and interpretable when retrieved.

The reported effect on retrieval failure rates is large. Large reductions when combined with hybrid search and reranking. The cost is one model call per chunk at indexing time. This for a large corpus is big. Two mitigations: use a small fast model, and exploit prompt caching. Since the full document is repeated across all of its chunks' prompts.

Testing whether a chunk can stand alone

The problem is coreference -- a chunk that says "it" or "the policy" is meaningless once separated from what it called. This measures that damage right, applies the four repairs the article names, and shows which one is worth its cost.

example_01.pyNumPy
Output

Now query this index and watch the retriever pick the wrong chunk.

example_02.pyNumPy
Output

Four repairs add context before embedding. Each one widens the gap between the correct chunk and the misleading one.

example_03.pyNumPy
Output

Price the repairs before choosing. The first three are free; only the LLM context costs money.

example_04.pyNumPy
Output

Enriched text improves retrieval, but the model should see the first chunk, not your made context.

example_05.pyNumPy
Output

A quick heuristic tests whether a chunk can stand alone. Does it open with a pronoun or bare article?

example_06.pyNumPy
Output

Things to try with Context-aware chunking

  1. Start with all of it off. The raw chunk is about refunds, never says so, and cannot be retrieved.
  2. Turn on resolve pronouns. Similarity jumps, because the searchable nouns are now in the text.
  3. Turn on over-enrich. The chunk stays findable and stops being distinguishable from its sibling. The cost of adding too much shared context.

What to recall

Context-aware chunking changes what is stored rather than where the cut falls. A chunk full of pronouns and back-references is unusable by the generator and invisible to retrieval. So title, heading path and resolved references are added back. Keep the enrichment short relative to the chunk, or shared context rules the embedding and chunks stop being distinguishable.

Choosing among them

The techniques are not options so much as layers, and they have a clear cost ordering.

Always do: capture the structure and prepend the heading path. Free, and it fixes the most common case.

Often do: 10–20% overlap. Cheap insurance against boundary loss.

Often worth it: parent-child retrieval, if chunks need to be small for precision. It resolves intra-section references without a model call.

Think of: contextual retrieval, when the corpus is high-value, the documents are long, and retrieval quality is the bottleneck. Measure the gain against the indexing cost.

Emerging: late chunking — embed the whole document with a long-context embedding model. Then pool token embeddings into chunk vectors. Each chunk's vector is informed by the entire document without any per-chunk generation. Promising, and it needs a suitable embedding model.

Testing whether a chunk stands alone

There is a simple and effective test, and it is worth running on a sample before indexing a whole corpus.

Take twenty chunks at random and read each one cold. For each, ask:

  1. Can I tell what topic this is about?
  2. Do all the pronouns and references resolve?
  3. If someone asked a question this chunk answers, would this text answer it?

If the answer to any is no, the chunk needs more context. That is a five-minute exercise that predicts retrieval quality better than most metrics. And it right finds which of the four techniques above is needed.

Following it up quantitatively: measure recall@k with and without the context addition on a question set. The heading path alone often moves recall by a few points.

Common mistakes with Context-aware chunking

  • Adding context after embedding. The context must be present in the text that gets embedded. Or it improves readability without improving retrieval.
  • Prepending so much context that it rules the embedding. A 200-token context on a 100-token chunk means the vector is mostly about the context. Keep it to a sentence or two.
  • Not storing the first chunk apart, so citations quote made context as if it were source text.
  • Making context per chunk without prompt caching, paying for the full document repeatedly.
  • Assuming overlap solves reference resolution. It handles boundaries, not distant antecedents.

Questions people ask about Context-aware chunking

Is the heading path enough? Often, for well-structured documents. It is always worth doing first because it is free.

How much does contextual retrieval cost? One model call per chunk. With a small model and prompt caching over the shared document, much less than it first appears.

Does the made context risk hallucination? It can misdescribe a chunk. Keep it to situating rather than summarising content, and store the first text apart for citation.

Should context be embedded or only stored? Embedded — that is the point. Store the first apart for display.

Does this replace good chunking? No, it complements it. Bad boundaries plus added context is worse than good boundaries plus added context.

What is late chunking? Embedding the whole document first and pooling token vectors into chunk vectors. So each chunk's embedding already carries document context.

Context-aware chunking: recap in one screen

  • Chunks routinely reference material outside themselves, and retrieval gives them out of order.
  • Prepend the heading path always — free, and it fixes the most common case.
  • Overlap handles boundary loss; parent-child retrieval handles intra-section references.
  • Contextual retrieval makes a situating sentence per chunk at indexing time, and measurably reduces retrieval failures.
  • Read twenty chunks cold and ask whether each stands alone. The cheapest diagnostic available.

Recall check

0 of 3

Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.

  1. What does this module say about “The problem is coreference, not boundaries”?

  2. What does this module say about “What gets added”?

  3. What does this module say about “What it costs, and what to watch”?

Cheat sheet

Context-aware chunking

Chunk a document by headings and semantic boundaries instead of fixed token counts and see the retrieval accuracy gained by preserving context.

GEN AI · vizlearn.in/gen_ai/context_aware_chunking.html

About the author

Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.

Teaching a course with this? Individual teachers can use it freely; for a whole department or programme, see institutional licences.