Semantic chunking
Put the boundary where the topic changes, not where the character count runs out. Embed each sentence, measure the similarity between consecutive ones, and cut at the troughs. Chunks come out semantically coherent and wildly variable in length.
Overview
How the edge is chosen
Split into sentences, embed each one, and compute the likeness between each in a row pair. Where the text stays on topic the likeness is high; where the subject changes it dips. Cut at the dips.
The threshold should be relative. A fixed number like 0.7 means a thing other in each corpus. So implementations use a percentile of the similarities observed in this document. The 5th percentile of gaps, say. Which adapts by itself to how varied the writing is.
The result is chunks that align with topic shifts rather than arbitrary token counts. A paragraph about authentication stays in one chunk; the next paragraph about rate limiting starts a new one. This is the grade edge over fixed-size chunking. The embedding of each chunk represents a coherent topic, not a random slice of text that starts mid-thought.
Three parameters control the output. The embedding model decides how likeness is measured — a model trained on semantic likeness (like sentence-transformers) works; a general-purpose model may not distinguish topic edges cleanly. The percentile threshold sets how sensitive the splitting is. A lower percentile (more aggressive) gives more, smaller chunks; a higher percentile gives fewer, larger ones. The min chunk size prevents degenerate one-sentence chunks that carry too little context to be useful on their own.
Semantic chunking pairs well with metadata enrichment. Because each chunk represents a coherent topic, it is straightforward to generate a summary or title for each chunk and store it alongside the embedding. At retrieval time, the summary can be used for hybrid search (keyword plus vector) or for reranking. This improves precision without raising the embedding cost at query time.
The implementation in LangChain is SemanticChunker, and in LlamaIndex it is SemanticSplitterNodeParser. Both follow the same algorithm: split into sentences, embed, compute pairwise likeness, cut at dips. The gaps are in defaults and the API surface, not in the underlying logic.
Put the boundary where the topic changes , not where the character count runs out.
In this guide
- How the edge is chosen
- What it costs
- When to reach for it
- Making the likeness sign usable
- Is it worth it? Measure before adopting
- How it relates to the other three plans
- Splitting where the meaning changes
- The threshold, and how to set it
- The likeness curve, and where it dips
- Things to try with Semantic chunking
- What to recall
- Where it helps and where it does not
- Variants
- Real guidance
- Questions people ask about Semantic chunking
- Semantic chunking: recap in one screen
Parameters
Visualisation
—Readout
What to watch
- The boundary lands at the similarity trough, not at a size limit.
- Chunk lengths become uneven, which is the point.
- The threshold is a percentile of this document, not a global constant.
Semantic chunking: A Practical Guide
What is semantic chunking, and when is it worth the extra cost?
How the edge is chosen
Split into sentences, embed each one, and compute the likeness between each in a row pair. Where the text stays on topic the likeness is high; where the subject changes it dips. Cut at the dips.
The threshold should be relative. A fixed number like 0.7 means a thing other in each corpus. So implementations use a percentile of the similarities observed in this document. The 5th percentile of gaps, say. Which adapts by itself to how varied the writing is.
The result is chunks that align with topic shifts rather than arbitrary token counts. A paragraph about authentication stays in one chunk; the next paragraph about rate limiting starts a new one. This is the grade edge over fixed-size chunking. The embedding of each chunk represents a coherent topic, not a random slice of text that starts mid-thought.
Three parameters control the output. The embedding model decides how likeness is measured — a model trained on semantic likeness (like sentence-transformers) works; a general-purpose model may not distinguish topic edges cleanly. The percentile threshold sets how sensitive the splitting is. A lower percentile (more aggressive) gives more, smaller chunks; a higher percentile gives fewer, larger ones. The min chunk size prevents degenerate one-sentence chunks that carry too little context to be useful on their own.
Semantic chunking pairs well with metadata enrichment. Because each chunk represents a coherent topic, it is straightforward to generate a summary or title for each chunk and store it alongside the embedding. At retrieval time, the summary can be used for hybrid search (keyword plus vector) or for reranking. This improves precision without raising the embedding cost at query time.
The implementation in LangChain is SemanticChunker, and in LlamaIndex it is SemanticSplitterNodeParser. Both follow the same algorithm: split into sentences, embed, compute pairwise likeness, cut at dips. The gaps are in defaults and the API surface, not in the underlying logic.
What it costs
The cost is real and worth stating plainly: one embedding call per sentence at indexing time.
A 100,000-document corpus averaging 200 sentences each is 20 million embedding calls, against perhaps 400,000 for fixed chunking. That is a fifty-fold increase in indexing cost.
| Strategy | Indexing cost | Quality |
|---|---|---|
| Fixed size | Lowest | Poor — cuts mid-sentence |
| Recursive | Lowest | Good |
| Structure-aware | Low | Better where structure exists |
| Semantic | High | Best on unstructured prose |
Whether that trade is worth it depends on one thing: does the document have usable structure? If it has headings and sections, structure-aware chunking gets most of the gain for almost none of the cost,. This is because the author now marked where the topics change. Semantic chunking earns its cost on smooth unstructured prose. Transcripts, scanned text, articles without headings.
When to reach for it
Worth it for long unstructured prose. Transcripts, interviews, reports without headings. Where no formatting sign exists and a fixed-size split reliably cuts mid-input.
Not worth it when the document now carries structure. If there are headings, they are an explicit, author-provided, free topic edge, and structure-aware chunking will beat semantic chunking at a fraction of the cost. Measure before adopting: on a retrieval evaluation set the gain over a well-tuned recursive splitter is often small.
Making the likeness sign usable
Raw sentence-to-sentence likeness is noisy. Short sentences — "Yes.", "See above.", a heading fragment — have unstable embeddings. So the gap sequence holds dips that are artefacts rather than topic changes.
Two standard mitigations. Buffering: embed each sentence together with its neighbours, so a short sentence inherits context and its vector stops swinging. Smoothing: take a rolling mean over the gap sequence before looking for troughs. So a single anomalous sentence cannot open a edge on its own.
Both trade response for stability, and both are worth having. Without them the technique looks impressive on a clean essay and falls apart on a support transcript. This is just the sort of unstructured text it was supposed to be good at.
Is it worth it? Measure before adopting
Semantic chunking is the most cited of the plans and the hardest to justify on evidence. It costs an embedding call per sentence at index time, gives chunks of unpredictable size that complicate context budgeting. And on published checks its edge over a well-tuned recursive splitter is often small or absent.
Where it does earn its cost: long unstructured prose with no formatting sign. Interview transcripts, meeting notes, scanned reports. And corpora where topics shift within a paragraph. So paragraph edges mislead.
The honest order of steps is to tune the separators and the size first, measure. And only then try semantic chunking against that baseline. It is a real technique that is often adopted before the cheap options have been exhausted.
How it relates to the other three plans
The four plans answer truly other questions, which is why they compose rather than compete.
Recursive asks where the safest edge is given a size budget. It is the default and the baseline.
Structure-aware asks what edges the author now marked. Free when the format carries them, and strictly better than guessing when it does.
Semantic asks where the meaning changes. It is the only one that can find a edge the author did not mark. This is why it is for unstructured prose in detail.
Context-aware asks what the chunk lost by being cut, and is orthogonal to all three. You can enrich chunks gave by any splitter.
A sensible production pipeline is often structure first, recursive to enforce the size limit, and context enrichment on the result. Semantic chunking enters when the documents have no structure to exploit.
Splitting where the meaning changes
Fixed-size chunking cuts each 500 characters no matter what is there. Recursive chunking respects paragraph and sentence edges. Neither knows whether a edge is a topic edge.
Semantic chunking finds those edges by gauging meaning right:
- Split the document into sentences.
- Embed each sentence.
- Compute the likeness between each in a row pair.
- Split where the likeness drops below a threshold.
Where two in a row sentences are semantically close, they belong together. Where the likeness falls sharply, the topic has changed and that is where the chunk should end.
sim(sᵢ, sᵢ₊₁) high → same chunk sim low → edge
The result is chunks that are each about one thing, at any length that takes — which is what an embedding wants. Since a vector showing one topic is sharp and one showing three is a blurred average.
The threshold, and how to set it
An absolute likeness threshold does not transfer between documents or models, because typical likeness levels vary enormously.
The reliable approach is a percentile of the distances within this document. Compute all in a row-sentence distances, and split at the ones above the 90th or 95th percentile.
| Percentile | Effect |
|---|---|
| 80th | Many boundaries, small chunks |
| 90th | A reasonable default |
| 95th | Fewer boundaries, larger chunks |
Two refinements that materially improve results:
Use a buffer. Compare a small window of sentences either side of the candidate edge rather than single sentences. A single short sentence — "This is key." — gives a spurious dip on its own.
Enforce size limits. Semantic edges can give a 4,000-token chunk or a 30-token one. Cap the max and merge below a min, or the downstream context budget turns unpredictable.
import numpy as np
sents = split_sentences(text)
vecs = model.encode(sents, normalize_embeddings=True)
sims = [float(vecs[i] @ vecs[i+1]) for i in range(len(vecs)-1)]
threshold = np.percentile(sims, 10) # lowest 10% of similarities
boundaries = [i+1 for i, s in enumerate(sims) if s < threshold]
The likeness curve, and where it dips
Semantic chunking splits where in a row sentences stop resembling each other. That description leaves the key question open -- how big a dip counts -- so here is the actual sign from a passage. And the edges each threshold gives.
The next question is how big a dip has to be before it counts as a edge. Try an absolute cutoff first.
A threshold next to the document's own spread of similarities fixes that.
Both threshold styles share one blind spot: they only compare next sentences. This fails when a passage returns to an earlier topic.
Set against fixed-size and recursive splitting, semantic chunking's real cost and its real payoff are both worth pricing out plainly.
Things to try with Semantic chunking
- Raise the split percentile to 70. It splits almost all over, giving chunks too small to carry an idea.
- Drop it to 10. Only the single largest topic change survives as a edge.
- Note that the threshold is derived from this document's own gaps. A fixed number would behave else on each corpus.
What to recall
Semantic chunking embeds each sentence and cuts where the likeness between neighbours drops. So edges land at topic changes rather than at character counts. It costs one embedding per sentence at index time. And it is often not worth it on documents that now carry headings.
Where it helps and where it does not
It helps on:
- Meeting and interview transcripts, where topics shift with no markup.
- Long-form articles and books without useful headings.
- OCR output, where structure was lost in extraction.
- Documents whose headings are decorative rather than topical.
It does not help on:
- Well-structured technical documentation. The headings are better edges than any likeness measure, and free.
- Short documents now about one thing — FAQ entries, product descriptions, tickets.
- Highly uniform text, where in a row-sentence likeness is flat and no clear edges exist.
- Code, where structural edges (functions, classes) are what matter.
There is a further honest caveat: the measured gain is often smaller than expected. Published checks on well-structured corpora often find semantic chunking roughly level with recursive chunking. It is worth evaluating rather than adopting on principle.
Variants
Numeric / gradient-based. Instead of a percentile threshold, look for the largest local drops in likeness. The points of steepest change. Which adapts better to documents with varying likeness levels.
Double-pass. Chunk semantically, then merge next chunks whose embeddings are close. Cleans up over-splitting.
LLM-based (agentic) chunking. Ask a model to propose edges, or to group sentences into topics. Highest grade on difficult text, and by far the most costly — often reserved for small high-value corpora.
Late chunking. A recent other worth knowing: embed the whole document with a long-context embedding model. Then pool the token embeddings into chunk vectors. Each chunk's vector is informed by the whole document, which fixes the missing-context problem without needing per-sentence calls.
Real guidance
Start with recursive or structure-aware chunking and a proper evaluation set. Measure recall@k. Then, if retrieval is the bottleneck and the documents lack structure, try semantic chunking and measure the gap.
That sorting matters because chunking is one of a few levers, and it is not often the largest. Hybrid search and reranking often give bigger gains for less effort. And they are worth exhausting first.
Any plan you choose, two things help regardless: prepend the heading path to each chunk so it carries context, and think of parent-child retrieval so small precise chunks can return larger context.
Questions people ask about Semantic chunking
Is semantic chunking worth the cost? On unstructured prose, often. On documents with real headings, often not — measure it.
What embedding model for the sentence checks? A small fast one is adequate; it is comparing next sentences, not doing retrieval.
How do I stop chunks turning enormous? Enforce a max size and split at the weakest internal edge when exceeded.
Does it replace overlap? Largely — edges at genuine topic shifts lose less than arbitrary ones. A small overlap is still cheap insurance.
Can I combine it with structure? Yes, and this is often the best answer: split on headings first. Then semantically within any oversized section.
Is late chunking better? Promising, and it needs a long-context embedding model. Worth evaluating if one is available.
Semantic chunking: recap in one screen
- Embed sentences, measure in a row likeness, and split where it drops. Edges at real topic changes.
- Use a percentile of the document's own distances rather than an absolute threshold.
- Add a check buffer and enforce min and max sizes.
- It costs one embedding call per sentence, which is a large indexing cost.
- Worth it on unstructured prose; structure-aware chunking often wins where headings exist.