Semantic chunking
Put the boundary where the topic changes, not where the character count runs out. Embed each sentence, measure the similarity between consecutive ones, and cut at the troughs. Chunks come out semantically coherent and wildly variable in length.
Overview
How the boundary is chosen
Split into sentences, embed each one, and compute the similarity between each consecutive pair. Where the text stays on topic the similarity is high; where the subject changes it dips. Cut at the dips.
The threshold should be relative. A fixed number like 0.7 means something different in every corpus, so implementations use a percentile of the similarities observed in this document — the 5th percentile of gaps, say — which adapts automatically to how varied the writing is.
Parameters
Visualisation
—Readout
What to watch
- The boundary lands at the similarity trough, not at a size limit.
- Chunk lengths become uneven, which is the point.
- The threshold is a percentile of this document, not a global constant.
Semantic chunking: A Practical Guide
What is semantic chunking, and when is it worth the extra cost?
What it costs
One embedding call per sentence at index time, against one per chunk for the alternatives. On a large corpus that is a real bill and a slow re-index, though it is paid once rather than per query.
It is also brittle on short sentences. "Yes." and "See above." have unstable embeddings, so a document full of them produces noisy similarity and boundaries in odd places. Buffering each sentence with its neighbours before embedding is the usual mitigation.
When to reach for it
Worth it for long unstructured prose — transcripts, interviews, reports without headings — where no formatting signal exists and a fixed-size split reliably cuts mid-argument.
Not worth it when the document already carries structure. If there are headings, they are an explicit, author-provided, free topic boundary, and structure-aware chunking will beat semantic chunking at a fraction of the cost. Measure before adopting: on a retrieval evaluation set the gain over a well-tuned recursive splitter is often small.
Things to try
- Raise the split percentile to 70. It splits almost everywhere, producing chunks too small to carry an idea.
- Drop it to 10. Only the single largest topic change survives as a boundary.
- Note that the threshold is derived from this document's own gaps — a fixed number would behave differently on every corpus.
What to remember
Semantic chunking embeds each sentence and cuts where the similarity between neighbours drops, so boundaries land at topic changes rather than at character counts. It costs one embedding per sentence at index time, and it is usually not worth it on documents that already carry headings.