Chunking Strategies for RAG
Before anything can be embedded or retrieved, a document has to be cut into pieces. Where you cut decides how much sense each piece makes on its own.
Chunking
Resulting Chunks
—Counts
Chunking: A Practical Guide
The retrieval step can only ever be as good as the pieces it is choosing between.
Quick Context
Every piece of RAG downstream of chunking — embedding, indexing, retrieval, reranking — operates on whatever the chunk boundaries produced. Cut a sentence in half and both halves lose the context that made the original sentence meaningful; the embedding of a fragment is not a fragment of the embedding.
Fixed-size vs semantic
Fixed-size chunking counts a fixed number of words or tokens and cuts there, with no regard for what is at that position — a heading, mid-word, mid-sentence, anywhere. It is simple, predictable, and blind. Semantic chunking respects natural boundaries — sentences, paragraphs, headings — and only splits at those boundaries, accepting some variation in chunk size in exchange for every chunk being a coherent unit.
Why overlap exists
Even a well-placed boundary loses something: whatever came just before a chunk starts is not in it, even though it might be exactly the context needed to make the chunk's first sentence make sense. Overlap re-includes the last few words of the previous chunk at the start of the next one, at the direct cost of storing and embedding the same words twice.
Interactive Exploration Guide
- Start at 30 words, fixed-size. Several chunks end mid-sentence — the cut counter is not zero.
- Switch to semantic. The cut counter drops to zero at any size — chunks vary a little in length but every one ends on a real sentence boundary.
- Shrink fixed-size chunks to 15 words. More chunks, and more of them are mid-sentence cuts — small fixed windows are the worst case for this failure.
- Add overlap. The highlighted words at the start of each chunk (after the first) are the ones repeated from the end of the previous chunk — visible context carried across the boundary, at the cost of storing it twice.
Key Takeaway
Fixed-size chunking is simple and fast but blind to meaning, and will cut sentences in half whenever the count lands mid-sentence. Semantic chunking respects real boundaries at the cost of variable chunk sizes. Overlap recovers some of the context lost at any boundary by duplicating a small window of text between neighbouring chunks. None of this is free — every choice trades simplicity, storage, or coherence against the others, and the right trade depends on how structured the source documents already are.