Chunking Strategies for RAG
Before anything can be embedded or retrieved, a document has to be cut into pieces. Where you cut decides how much sense each piece makes on its own.
Chunking
Resulting Chunks
—Counts
Overview
Before the details
Every piece of RAG downstream of chunking — embedding, indexing, retrieval, reranking — operates on any the chunk boundaries gave. Cut a sentence in half and both halves lose the context that made the first sentence meaningful; the embedding of a fragment is not a fragment of the embedding.
The strategies form a spectrum. Fixed-size chunking splits by token count. This ignores content fully — simple and fast. But it regularly cuts mid-sentence. Recursive chunking tries paragraph breaks first, then sentence breaks, then word breaks. This keeps units coherent when the document cooperates. Structure-aware chunking reads headings and section markers and splits at the boundaries the author provided. Semantic chunking embeds every sentence and splits where the topic shifts, at the cost of one embedding call per sentence at indexing time.
Chunk size controls a direct trade-off. Small chunks (100–200 tokens) are precise — each one is about one idea. So retrieval is accurate — but they lose surrounding context. Large chunks (500–1000 tokens) carry more context but dilute the embedding. A chunk about three topics is weakly relevant to all three and strongly relevant to none. Most production systems settle between 200 and 500 tokens, with 50–100 tokens of overlap to soften boundary effects.
Overlap is a patch, not a fix. It copies tokens across chunks so that a sentence split at a boundary still appears whole in at least one chunk. The cost is storage (every overlapped token is embedded and indexed twice) and rare copy retrieval results that need deduplication downstream. The benefit is measurable: removing overlap usually drops recall by 5–15% on boundary-sensitive queries.
Before anything can be embedded or retrieved, a document has to be cut into pieces.
In this guide
- Before the details
- Fixed-size vs semantic
- Why overlap exists
- Why documents have to be split
- The strategies, from crude to careful
- Size and overlap
- Cutting documents up without cutting answers in half
- Tests to try
- The short of it
- Parent-child, and sentence windows
- What to store alongside the vector
- Document types that need special handling
- Questions people ask
- Recap in one screen
Chunking: A Practical Guide
The retrieval step can only ever be as good as the pieces it is choosing between.
Before the details
Every piece of RAG downstream of chunking — embedding, indexing, retrieval, reranking — operates on any the chunk boundaries gave. Cut a sentence in half and both halves lose the context that made the first sentence meaningful; the embedding of a fragment is not a fragment of the embedding.
The strategies form a spectrum. Fixed-size chunking splits by token count. This ignores content fully — simple and fast. But it regularly cuts mid-sentence. Recursive chunking tries paragraph breaks first, then sentence breaks, then word breaks. This keeps units coherent when the document cooperates. Structure-aware chunking reads headings and section markers and splits at the boundaries the author provided. Semantic chunking embeds every sentence and splits where the topic shifts, at the cost of one embedding call per sentence at indexing time.
Chunk size controls a direct trade-off. Small chunks (100–200 tokens) are precise — each one is about one idea. So retrieval is accurate — but they lose surrounding context. Large chunks (500–1000 tokens) carry more context but dilute the embedding. A chunk about three topics is weakly relevant to all three and strongly relevant to none. Most production systems settle between 200 and 500 tokens, with 50–100 tokens of overlap to soften boundary effects.
Overlap is a patch, not a fix. It copies tokens across chunks so that a sentence split at a boundary still appears whole in at least one chunk. The cost is storage (every overlapped token is embedded and indexed twice) and rare copy retrieval results that need deduplication downstream. The benefit is measurable: removing overlap usually drops recall by 5–15% on boundary-sensitive queries.
Fixed-size vs semantic
Fixed-size chunking counts a fixed number of words or tokens and cuts there, with no regard for what is at that position — a heading, mid-word, mid-sentence, anywhere. It is simple, predictable, and blind. Semantic chunking respects natural boundaries — sentences, paragraphs, headings — and only splits at those boundaries, accepting some change in chunk size in exchange for every chunk being a coherent unit.
Why overlap exists
Even a well-placed boundary loses something: any came just before a chunk starts is not in it, even though it might be exactly the context needed to make the chunk's first sentence make sense. Overlap re-includes the last few words of the previous chunk at the start of the next one, at the direct cost of storing and embedding the same words twice.
Why documents have to be split
Three reasons, and they compound.
Embeddings blur when the input is long. A vector showing a 20-page document is an average of all of it in it. So it is close to nothing notably. Retrieval precision collapses.
The context window is finite, and shared with the question and the answer. Ten full documents will not fit.
Precision needs granularity. The user asked one question; the answer is one paragraph. Retrieving the whole document buries it.
So documents are split into chunks, each embedded apart. The chunk becomes the unit of retrieval, and how you choose its boundaries decides the ceiling on the whole system's quality.
The strategies, from crude to careful
| Strategy | Splits on | Quality |
|---|---|---|
| Fixed size | Character or token count | Poor — cuts mid-sentence |
| Recursive | Paragraph, then sentence, then word | Good default |
| Structure-aware | Headings, sections, list items | Better for structured documents |
| Semantic | Where the topic shifts, measured by embedding distance | Best quality, highest cost |
| Sentence windows | One sentence, with neighbours as context | Good precision, needs expansion |
| Parent-child | Retrieve small, return large | Best of both |
Fixed-size splitting at 500 characters cuts sentences in half, separates a heading from its content and splits tables. It is the default in many tutorials and the first thing worth replacing.
Recursive splitting tries separators in order. Paragraph break. Then newline. Then sentence end. Then space. Taking the largest unit that fits. It respects natural boundaries most of the time and costs nothing extra. This is the sensible default.
Structure-aware splitting uses the document's own markup: Markdown headings, HTML sections, PDF outline. A chunk is then a semantically coherent unit by construction. And the heading path can be prepended to give the chunk context ("Handbook > Leave > Parental leave").
Semantic chunking embeds each sentence and splits where in a row sentences become dissimilar. It finds real topic boundaries and costs an embedding call per sentence at indexing time.
Size and overlap
Size is a trade-off between context and precision:
| Chunk size | Effect |
|---|---|
| 100 tokens | Precise retrieval, often missing context |
| 200–500 tokens | The usual sweet spot |
| 1,000+ tokens | Good context, blurred embeddings, fills the prompt |
Overlap of 10–20% means in a row chunks share their boundary text, so a sentence spanning a split appears whole in at least one chunk. The cost is duplicated storage and sometimes two near-the same chunks in the results. This reranking or maximal marginal relevance handles.
The number that matters is not the token count in isolation but whether a chunk can answer a question on its own. A chunk beginning "This means that the limit is 30 days" is useless without knowing what "this" means —. This is the argument for prepending the heading path, or for the parent-child pattern.
Cutting documents up without cutting answers in half
Chunking decides what a retriever can ever return. And the default of a fixed character count fails in a specific way. This measures four strategies against the same document, and scores them on the only thing that matters -- whether a complete answer survives in one piece.
Building on the same setup, the next block shows the left cases.
The following block covers the left steps.
The next block goes on with the left demonstrations.
Building on the same setup, the next block shows the left cases.
The following block covers the left steps.
The next block goes on with the left demonstrations.
Building on the same setup, the next block shows the left cases.
Tests to try
- Start at 30 words, fixed-size. A few chunks end mid-sentence — the cut counter is not zero.
- Switch to semantic. The cut counter drops to zero at any size — chunks vary a little in length but every one ends on a real sentence boundary.
- Shrink fixed-size chunks to 15 words. More chunks, and more of them are mid-sentence cuts — small fixed windows are the worst case for this failure.
- Add overlap. The highlighted words at the start of each chunk (after the first) are the ones repeated from the end of the previous chunk — visible context carried across the boundary, at the cost of storing it twice.
The short of it
Fixed-size chunking is simple and fast but blind to meaning, and will cut sentences in half when the count lands mid-sentence. Semantic chunking respects real boundaries at the cost of variable chunk sizes. Overlap recovers some of the context lost at any boundary by copying a small window of text between neighbouring chunks. None of this is free — every choice trades ease, storage, or coherence against the others. And the right trade depends on how structured the source documents already are.
Parent-child, and sentence windows
Two patterns resolve the size trade-off rather than compromising on it.
Parent-child (small-to-big). Index small chunks for precise matching, but return their larger parent section to the model. Retrieval operates on 100-token units; generation sees the 800-token section they came from. You get the precision of small chunks and the context of large ones.
Sentence windows. Index each sentence one by one. And at retrieval time expand to include the two or three sentences either side. Same idea, finer granularity.
Both require storing the link between the indexed unit and its context. This every serious vector store supports through metadata.
What to store alongside the vector
Metadata is what makes a retrieval system usable in production. And it is easy to under-invest in at indexing time:
- Source — document id, title, URL. Needed for citations.
- Location — page number, section heading, character offsets.
- Heading path — prepend it to the chunk text and storing it.
- Dates — created and modified, so stale content can be filtered or down-weighted.
- Access control — the groups permitted to see it, filtered at query time.
- Document type — so a query about policy can be restricted to policy documents.
Adding metadata after the fact means re-indexing. Deciding it up front costs nothing.
Document types that need special handling
Tables. Splitting a table mid-row destroys it. Extract tables apart, keep each whole, and store a text summary alongside for embedding.
Code. Split on function and class boundaries, not line counts. A half-function is not retrievable knowledge.
PDFs. Multi-column layouts are often extracted in the wrong reading order. Check the extracted text before indexing thousands of documents; a layout-aware extractor is worth the effort.
Transcripts. Split on speaker turns or topic shifts, and keep timestamps in the metadata.
Slides. One slide per chunk usually works, with the deck title prepended.
Questions people ask
What chunk size should I use? Start at 300–500 tokens with 15% overlap and recursive splitting. Then evaluate against your own question set.
Is semantic chunking worth the cost? Often yes for heterogeneous documents. And the gain over structure-aware chunking on well-structured documents is usually small.
Should chunks overlap? Yes, 10–20%. It is cheap insurance against boundary loss.
How do I keep context in small chunks? Prepend the heading path, or use parent-child retrieval.
Can chunk size differ within one index? Yes, and it is often sensible. A table or a code function is one chunk any its length.
How do I know my chunking is bad? Low recall@k on questions whose answers you know are in the corpus. Then read the retrieved chunks: fragments and mid-sentence cuts are the diagnosis.
Recap in one screen
- Chunks are the unit of retrieval, so their boundaries set the ceiling on system quality.
- Recursive splitting on natural separators is the sensible default; structure-aware is better where structure exists.
- 200–500 tokens with 10–20% overlap covers most cases.
- The real test is whether a chunk can answer a question alone. Hence heading paths and parent-child retrieval.
- Store source, location, dates and permissions at indexing time; adding them later means re-indexing.