Preparing PDFs for LLMs and retrieval pipelines
Last reviewed
Feeding PDFs to a language model — for retrieval-augmented generation, for summarisation, or just to paste into a long context window — is now one of the most common reasons anyone converts a document at all.
It is also the use where the quality of the conversion matters most and is noticed least, because a retrieval pipeline fails quietly. A badly converted document does not throw an error. It simply returns the wrong passage, or a passage with its table mangled, and the model answers confidently from it.
9 minute read
Why not just extract the raw text
Every PDF library will give you the text of a document in a couple of lines of code, and for a single-column memo that is genuinely enough. For anything else it is not, for three reasons that all cost you retrieval quality.
First, reading order. Raw extraction returns text in the order the glyphs were drawn, which on a two-column page interleaves the columns. Half your chunks become sentence fragments alternating between unrelated paragraphs, and they embed to a point in vector space that corresponds to nothing.
Second, boundaries. Without headings there is nothing to chunk on except character count, so chunks split mid-sentence and mid-table, and a chunk that begins halfway through an explanation cannot be understood on its own — which is precisely what a retrieved chunk has to be.
Third, tables. Extracted as raw text, a table becomes a stream of numbers with no indication of which column any of them came from. A model reading that will still answer your question about it. The answer will just be wrong.
What Markdown gives the pipeline
Markdown is a good intermediate format here for reasons that have nothing to do with it being pretty.
- Headings give you semantic chunk boundaries, so a chunk is a section rather than a thousand-character window.
- The heading path gives each chunk its context — you can prepend "Chapter 3 › Rate limits › Burst behaviour" to the chunk before embedding, which measurably improves retrieval on documents where the same word means different things in different sections.
- Pipe tables keep rows and columns associated, so a model reading a retrieved table can tell which figure belongs to which heading.
- It is compact. Markdown carries structure at a cost of a few characters per element, where HTML or JSON layout formats spend tokens on markup that tells the model nothing.
- It is a format every current model has seen enormous amounts of in training, so it needs no explanation in the prompt.
Chunking, concretely
The approach that works on converted documents is to split on headings first and only fall back to size limits within an oversized section.
Rough parameters that hold up across most document types:
| Primary split | On H2, falling back to H3 in long sections |
|---|---|
| Target chunk size | 500–1,000 tokens |
| Hard maximum | Whatever leaves room for several chunks in context |
| Overlap | 1–2 sentences, or none if chunks are heading-bounded |
| Chunk prefix | Document title plus the full heading path |
| Tables | Never split — keep whole, even if oversized |
Never split a table
This is the rule most pipelines get wrong, because a naive splitter has no idea it is inside one.
A GFM table split down the middle produces two chunks, neither of which is a valid table: the first has a header and some rows, the second has rows with no header at all. The second chunk is worse than useless, because the numbers in it are unlabelled and a model will label them by guessing.
Detect table blocks before chunking and keep each one intact, even where that means a chunk over your size target. If a table is genuinely too large for one chunk, repeat the header row in each piece.
Where scanned documents fit
A scan has no text, so there is nothing to chunk. Text recognition has to happen first — and its output has properties a pipeline needs to account for, because recognition is an estimate rather than a transcription.
Two things in particular. Confidence varies by page, so a document can be near-perfect for forty pages and unreliable for three; if your recognition step reports per-page confidence, carry it into the chunk metadata so a low-confidence answer can be flagged. And unreadable words are usually dropped rather than marked, which means the text looks fluent and complete while missing exactly the tokens the engine found hardest — often names, reference codes and figures.
For anything where an error has consequences, recognised text should be treated as a lead to verify against the original, not as a source of record.
What to check before you index a corpus
Indexing is the expensive, slow, hard-to-undo step, so it is worth spending twenty minutes on a sample first. Convert five documents chosen to be different from each other and read the Markdown.
- Do the headings form a sensible ladder? If they are flat, your chunker has nothing to split on and you will fall back to fixed-size windows across the whole corpus.
- Did the tables survive? Count columns against the original in two or three.
- Is the reading order right on any multi-column pages? Interleaved columns are obvious once you look and invisible once indexed.
- Are there empty or near-empty conversions? Those are scans, and they need a different path.
- Is the same boilerplate repeated in every document? Repeated headers and footers become near-duplicate chunks that crowd out real content in retrieval results.
The privacy dimension, which is not incidental
RAG corpora are built from exactly the documents organisations are least willing to hand to a third party: contracts, internal reports, customer records, research that is not published yet.
It is worth being deliberate about which steps in your pipeline send those documents somewhere. Conversion does not have to be one of them — a browser-based converter processes the file on the machine it is already on, so nothing about the conversion step adds a party to the chain. Whether the later steps do is a separate decision, but it is a decision you can make separately.