Structure-aware chunking

Carrying the heading path

The single most valuable thing this approach enables is context injection.

A chunk taken from deep inside a handbook might read: "The entitlement is 39 weeks, of which 6 are at full pay." Retrieved on its own, it does not say what kind of leave, or for whom.

Structure-aware chunking knows exactly where it came from, so the path can be prepended:

Employee Handbook > Leave > Parental Leave > Entitlement

The entitlement is 39 weeks, of which 6 are at full pay.

That transforms the chunk in two ways. Its embedding now contains the topic words, so a query about parental leave matches it. And the retrieved text is self-contained, so the model can answer without guessing.

This is cheap, mechanical, and one of the highest-value improvements available to a retrieval pipeline over naive chunking.

import re

def chunk_markdown(text, max_chars=1500):
    chunks, path, buf = [], [], []
    for line in text.split("\n"):
        m = re.match(r"^(#{1,6})\s+(.*)", line)
        if m:
            if buf:
                chunks.append((" > ".join(path), "\n".join(buf)))
                buf = []
            level = len(m.group(1))
            path = path[: level - 1] + [m.group(2)]
        else:
            buf.append(line)
            if sum(len(b) for b in buf) > max_chars:
                chunks.append((" > ".join(path), "\n".join(buf)))
                buf = []
    if buf:
        chunks.append((" > ".join(path), "\n".join(buf)))
    return chunks

What it needs from you

A parser per format. Markdown is easy, HTML is manageable, PDF is genuinely hard — a PDF has no structure, only positioned glyphs, so headings must be inferred from font size and spacing. Most RAG quality problems on PDFs are really extraction problems.

Sections also vary wildly in length, so structure-aware chunking is usually combined with a recursive splitter: split on structure first, then recursively split any section still over the limit, keeping the heading path on every resulting piece.

Extraction is the hard part, not splitting

The splitting is easy once the structure is known. Knowing it is the problem, and it varies enormously by format.

Markdown and HTML carry structure explicitly; a parser gives you the tree.

PDF has no structure at all — only positioned glyphs. Headings must be inferred from font size, weight and spacing, multi-column layouts interleave text if read naively, and tables lose their rows. Most RAG quality complaints about PDFs are extraction failures rather than retrieval failures, and swapping the embedding model will not touch them.

Office documents and slides sit in between: real structure exists in the file format, and most extraction libraries flatten it away before you see it.

Budget for extraction seriously. It is unglamorous and it decides the ceiling on everything downstream.

Tables, code and the things that must not be split

Some structures are atomic. A table cut between its header row and its data leaves rows of unlabelled numbers; a code block cut in half is not code; a numbered list split mid-way loses the numbering that made it readable.

The usual handling is to keep these whole even when they exceed the size limit, and to accept the occasional oversized chunk. For a genuinely large table, the better answer is often to store a generated summary for retrieval and the full table for generation — the summary is what matches the query, the table is what answers it.

The same reasoning applies to images and diagrams: retrieve on a caption or description, return the artefact. Once you separate what is embedded from what is returned, you are already at the parent-document pattern.

What the heading path does to retrieval

Prepending Refund policy > Exceptions to a chunk is often described as a readability improvement, and its larger effect is on retrieval.

The section's vocabulary is now inside the embedded text. A query using the words of the section — "refund exceptions" — matches a chunk whose body never uses either word, because the body says "digital goods are non-refundable once downloaded". Without the path, that chunk is close to unfindable by its own section's name.

Store the path as metadata as well as prepending it. As metadata it supports filtering ("only search the exceptions"), it gives you a citation to display, and it lets you reconstruct the document's hierarchy for a parent-document lookup.

One caution: on a deep hierarchy the path can grow long enough to dominate a short chunk's embedding, which is the dilution problem described under context-aware chunking. Two or three levels is usually the useful limit.

Using the boundaries the author already wrote

Every structured document contains explicit markers of where topics change: headings, sections, list items, table boundaries, code blocks. Structure-aware chunking uses those instead of guessing.

The advantage over recursive splitting is that a heading is a semantic boundary asserted by the person who wrote the document. No similarity computation can be more authoritative than that, and it is free to use.

The corresponding rule: split on the highest-level heading that produces chunks within your size budget, and fall back to finer boundaries inside sections that are too large.

Handling elements that must not be split

Structure also tells you what to keep whole, which fixed-size chunking cannot know.

Tables. Splitting a table mid-row destroys it — the header is in one chunk and the data in another, and neither is interpretable. Extract each table as its own chunk, and store a short text description alongside it for embedding, since a grid of numbers embeds poorly.

Code blocks. Keep whole. A half-function is not retrievable knowledge.

Lists. Keep short lists whole. A list of eight steps split after step three is worse than useless, because the retrieved fragment looks complete.

Definitions and glossary entries. One entry per chunk is usually right.

Figures and captions. Keep the caption with its reference.

The general rule: any element whose meaning depends on its own completeness gets its own chunk, whatever the size budget says. Variable chunk sizes are correct here, not a compromise.

The heading path, and what it does to retrieval

Structure-aware splitting uses boundaries the author already wrote. The splitting is the easy half; the two things that decide whether it is worth doing are what you carry along with each chunk, and what you refuse to split.

import numpy as np

DOC = "\n".join([
    "# Product Guide",
    "## Billing",
    "### Refunds",
    "Requests are reviewed and settled within 30 days.",
    "## Support",
    "### Escalation",
    "Requests are reviewed and answered within 4 hours.",
])

print("A DOCUMENT WITH REAL STRUCTURE. the author already marked every")
print("boundary worth having:")
for line in DOC.split("\n"):
    print("   %s" % line)
print()

chunks = []
path = {}
for line in DOC.split("\n"):
    if line.startswith("#"):
        level = len(line) - len(line.lstrip("#"))
        title = line.lstrip("# ").strip()
        path = {k: v for k, v in path.items() if k < level}
        path[level] = title
    elif line.strip():
        chunks.append((" > ".join(path[k] for k in sorted(path)), line.strip()))

print("SPLITTING ON THE HEADINGS gives chunks that are whole sections --")
print("but the useful part is the PATH each one sits under:")
print("%-46s %s" % ("heading path", "text"))
for p, t in chunks:
    print("%-46s %s" % (p, t[:34]))
print("   nobody had to guess where these boundaries were. the author")
print("   wrote them, and a markdown parser reads them exactly.")
print()

print("NOW WHAT THE PATH BUYS. embed the chunk text alone, then embed the")
print("path plus the text, and score a query against both:")
# a bag-of-words vector over the words that distinguish anything here
VOCAB = ["billing", "refund", "support", "escalation", "request",
         "review", "settle", "answer", "within", "days", "hours"]

def vec(text, with_path=""):
    full = (with_path + " " + text).lower()
    v = np.array([1.0 if w in full else 0.0 for w in VOCAB])
    return v + 0.15

def cos(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

QUERY = vec("how long are billing requests reviewed within")
print("%-46s %14s %14s" % ("chunk", "text only", "path + text"))
for p, t in chunks:
    print("%-46s %14.4f %14.4f"
          % (p, cos(QUERY, vec(t)), cos(QUERY, vec(t, p))))
print()
print("   query: 'how long are billing requests reviewed within'")
tp = [cos(QUERY, vec(t)) for _, t in chunks]
wp = [cos(QUERY, vec(t, p)) for p, t in chunks]
print("   on TEXT ALONE the two chunks score %.4f and %.4f -- a gap of"
      % (tp[0], tp[1]))
print("   %.4f, which is to say a tie. the retriever picks whichever the"
      % abs(tp[0] - tp[1]))
print("   sort happened to put first.")
print("   with the PATH the scores are %.4f and %.4f, a gap of %.4f, and"
      % (wp[0], wp[1], abs(wp[0] - wp[1])))
print("   the Billing one wins for the right reason.")
print("   look at the two chunk TEXTS: 'reviewed and settled within 30")
print("   days' and 'reviewed and answered within 4 hours'. they share")
print("   almost every word, and neither contains 'billing' or 'support'.")
print("   on text alone the retriever is choosing between two nearly")
print("   identical strings and has no basis for preferring either.")
print("   the PATHS are what separate them -- one sits under Billing and")
print("   one under Support -- and once the path is embedded with the")
print("   text, the chunk answers a query using a word it does not")
print("   contain.")
print("   this is the same coreference repair that context-aware chunking")
print("   pays an LLM for, obtained free from structure the author already")
print("   provided.")
print()

print("AND THE PART THAT IS ACTUALLY HARD: THINGS THAT MUST NOT BE SPLIT.")
TABLE = ["| plan    | price | seats |",
         "|---------|-------|-------|",
         "| starter | 10    | 3     |",
         "| team    | 40    | 20    |",
         "| scale   | 150   | 100   |"]
print("   a table, split by a character limit that lands mid-way:")
LIMIT = 60
joined = "\n".join(TABLE)
naive = [joined[i:i + LIMIT] for i in range(0, len(joined), LIMIT)]
for i, c in enumerate(naive):
    print("      [%d] %r" % (i, c.replace("\n", " / ")))
print("   chunk 1 onward has no header row, so 'starter costs 10' is no")
print("   longer recoverable from it -- the column names are in a")
print("   different chunk. a table split anywhere is not a smaller table,")
print("   it is a list of numbers with no labels.")
print()
print("   the same for these, and each needs its own rule:")
print("%-24s %s" % ("element", "why splitting destroys it"))
for row in (("a table", "the header row carries the meaning"),
            ("a code block", "half a function does not run or parse"),
            ("a numbered list", "step 4 of 9 means nothing alone"),
            ("a figure and caption", "either alone is unusable")):
    print("%-24s %s" % row)
print("   the standard handling is to treat each as ATOMIC: never split")
print("   it, and if it exceeds the chunk limit, let it be an oversized")
print("   chunk rather than a broken one. a chunk that is too big costs")
print("   context; a chunk that is cut in half costs the answer.")
print()

print("WHICH IS WHY THE EXTRACTION, NOT THE SPLITTING, IS THE WORK:")
print("%-26s %-24s %s" % ("source format", "structure available", "effort"))
for row in (("markdown", "headings, tables, code", "trivial to parse"),
            ("HTML", "the full DOM", "easy, if it is clean"),
            ("a well-made PDF", "tagged sections", "moderate"),
            ("a scanned PDF", "none until OCR", "the whole project"),
            ("a transcript", "speaker turns, maybe", "little to work with")):
    print("%-26s %-24s %s" % row)
print("   splitting markdown on headings is ten lines. getting a usable")
print("   heading hierarchy out of a scanned PDF is a document-AI project,")
print("   and that is where the time in a real RAG build actually goes --")
print("   not in the chunking strategy, in the extraction that has to")
print("   happen before any strategy can be applied.")

Things to try

1. Compare structure first with fixed size at the same chunk size. The character splitter cuts through the heading; the structure-aware one does not.
2. Lower the chunk size until a section still exceeds it. Structure gives boundaries, not size control, so a recursive splitter has to finish the job.
3. Note the heading path on each chunk. A fragment retrieved alone still says which section it belongs to.

What to remember

Structure-aware chunking splits along the document's own markup — headings, list items, table rows — because those boundaries were placed by the author and cost nothing to find. Carrying the heading path into each chunk is the half people miss: it makes a fragment self-describing and puts the section's vocabulary into the embedded text.

Extraction is the hard part

The method depends on being able to see the structure, and for several common formats that is where the work is.

Markdown and HTML are easy — the structure is explicit and parseable.

PDF is the difficult one. A PDF is a description of marks on a page, not a document tree. Headings must be inferred from font size, weight and position; multi-column layouts are frequently extracted in the wrong reading order; and tables often come out as a jumble.

The practical advice is to check the extracted text before indexing thousands of documents. Extract ten, read them, and look for interleaved columns, missing headings and mangled tables. A layout-aware extractor is worth the effort where quality matters.

Scanned documents need OCR first, which loses structure and introduces character errors. Expect to fall back to semantic or recursive chunking for these.

Word documents carry heading styles, which is usable structure if the author applied them — and many do not, using bold text instead.

Where it sits among the strategies

Structure-aware chunking is the right default where structure exists, and it composes with the others:

• Split on headings first, then recursively within any oversized section.
• Split on headings, then use parent-child so small precise chunks return their section.
• Fall back to semantic chunking for documents whose extraction produced no usable structure.

That layered arrangement handles a mixed corpus, which is what real corpora are.

Metadata worth capturing while you have it

The structural pass is the moment when this information is available, and capturing it later means re-indexing:

• Heading path — prepend to the text and store as a field.
• Heading level — useful for weighting or filtering.
• Page number and character offsets — needed for precise citations.
• Element type — paragraph, table, code, list. Enables filtering to tables when a question asks for figures.
• Document title, date, source and access control.

Citations are the practical payoff: "Handbook, section 4.2, page 17" is verifiable, and "one of the retrieved chunks" is not.

Questions people ask

What if the document has no headings? Fall back to recursive splitting, or semantic chunking if it is continuous prose.

How do I handle a very long section? Split it recursively inside, keeping the heading path on every resulting chunk.

Should chunk sizes be uniform? No. A table, a code block or a glossary entry should be one chunk whatever its length.

Is prepending the heading path really worth it? Yes — it improves both the embedding and the usability of the retrieved text, for almost no cost.

What about PDFs? Use a layout-aware extractor and inspect the output. This is where most structure-aware pipelines actually fail.

Can I combine this with semantic chunking? Yes, and it is a good arrangement: headings first, semantic splitting within oversized sections.

Recap in one screen

• Headings and sections are boundaries the author asserted — more authoritative than any inferred split.
• Prepend the heading path to each chunk: it sharpens the embedding and makes the retrieved text self-contained.
• Keep tables, code blocks and short lists whole, whatever the size budget says.
• Extraction is the hard part, and PDFs are where it usually goes wrong — inspect before bulk indexing.
• Capture heading path, page, element type and permissions during the structural pass, because adding them later means re-indexing.