Chunking converted documents for retrieval

Why fixed-size chunking is the fallback, not the default

Splitting every eight hundred characters is simple, uniform and completely indifferent to meaning. It is popular because it always works and never works well.

Its failure mode is specific: it cuts in the middle of the unit a reader would treat as atomic. Half a table. A heading separated from the section it introduces. The first two items of a five-item list. Each of those chunks is retrievable and each is misleading on its own, which is the worst combination — the retriever finds it, the model reads it, and nothing in the chunk signals that it is a fragment.

Structural chunking avoids that by using boundaries the document already has. Which is why the heading ladder matters more than any other property of a converted file.

Chunk on headings first

The rule is straightforward. Split at the deepest heading level that produces chunks in your target size range, keep the full heading path with each chunk, and fall back to paragraph boundaries inside any section that is still too long.

Splitting at every level from the top down, rather than picking one level for the whole corpus, is what handles documents with uneven section lengths — which is to say, all of them.

What to do when the ladder is flat

Some documents convert with no headings at all. This is not a rare edge case: it is what happens whenever the original styled its headings by weight or colour rather than size, and a bold eleven-point heading over eleven-point body text is geometrically not a heading.

You have three options, in order of how much work they are and how well they work.

• Fix the file. If the corpus is small enough, adding the heading levels by hand is an hour that improves every retrieval afterwards. Skim the Markdown and promote the lines that were clearly headings.
• Convert from a better source. If a Word original exists, its heading styles are stated outright and the problem disappears entirely.
• Chunk on paragraphs with overlap. The genuine fallback. Group consecutive paragraphs up to your size limit and overlap by one paragraph, so a statement split across a boundary appears whole in one of the two chunks.

Never split a table

A pipe table cut in half produces two chunks, one with a header and no data, and one with data and no header. The second is genuinely dangerous: rows of numbers with no column names, which a model will interpret against whatever context it can find.

Treat a table as atomic. If it exceeds your chunk size, the right move is to split it by rows and repeat the header row in each piece, so every chunk is a valid table that says what its columns mean. A table too large even for that is usually a table that should be a document of its own.

Sizes worth starting from

These are starting points to measure against, not recommendations. The right size depends on your embedding model, your retrieval depth and how specific your questions are.

Carry the context into the chunk itself

A chunk is retrieved alone and read alone, so anything it needs has to be inside it. The heading path is the cheapest and most valuable thing to add: the document title, then the chain of headings above the chunk, then the text.

This costs a few dozen characters and it fixes the most common retrieval failure there is, which is a chunk that discusses limits without saying what has limits. Rate limits, size limits and retention limits all read identically once separated from their section heading, and a query about one will happily retrieve the chunk about another.

The same argument applies to the source document name and, where it exists, the page number. A retrieved answer a reader cannot trace back to a page is an answer they have to take on faith.

Check what boilerplate is doing to your index

Repeated headers and footers are detected and removed during conversion when they repeat in the same position on at least half the pages, which handles the usual case of a running title and a page number.

It does not handle a document that varies its furniture — a different chapter title on each page, a confidentiality notice that only appears on some. Those fragments survive into the Markdown, and at corpus scale they become hundreds of near-identical chunks that crowd out real content in every result set.

The check is quick. Sort your chunks by similarity to each other and look at the top of the list. If the most similar pairs in your corpus are boilerplate, strip it before indexing rather than after.

Measure rather than guess

Every choice above has a defensible argument behind it and none of them survives contact with a specific corpus unchanged. Chunk size in particular is not knowable in advance.

Build a set of thirty real questions with known answers before you index anything, then run the same set against two or three chunking strategies. It takes an afternoon and it replaces a permanent argument with a number. The strategies worth comparing first are heading-based against fixed-size with overlap, since that comparison tells you how much your conversion quality is actually worth.