Chunking

Chunking is the step in a retrieval pipeline where a document is split into smaller units that are embedded, indexed, and retrieved independently. Chunk boundaries decide what a retriever is capable of returning: a passage can only be retrieved if it sits inside a single chunk, and a chunk that packs three unrelated sections together dilutes the embedding that represents it.

Four families of strategy are in common use. Fixed-size chunking splits on a token count with an overlap window, which is fast and predictable but cuts sentences and tables in half. Recursive chunking walks a priority list of separators – paragraph, then sentence, then word – falling down the list until each piece fits the size budget. Semantic chunking embeds sentences and cuts where consecutive embeddings diverge, so boundaries follow topic shifts rather than character counts. Structure-aware chunking uses the document's own markup, splitting on headings, list items, table rows, or page breaks. Chroma's July 2024 technical report "Evaluating Chunking Strategies for Retrieval", by Brandon Smith and Anton Troynikov, scored these families on recall, precision, and intersection-over-union, and proposed two more: a cluster-based semantic chunker and an LLM prompted to chunk the corpus directly. NVIDIA's own evaluation, published on its technical blog, reported page-level chunking as the strongest default across its test datasets, at an average accuracy of 0.648 with the lowest variance of the strategies it compared.

Structure-aware chunking is only available if the structure survived extraction, which is the constraint most chunking advice leaves out. Web content arrives as HTML, and the extractor that converts it decides whether headings stay headings, whether a table stays a table, and whether the sidebar and the cookie banner end up interleaved with body text. If extraction flattens a page into a wall of plain text, every downstream strategy collapses to counting tokens, because there are no headings left to split on. This is why Markdown-producing extraction APIs matter to retrieval quality: Firecrawl, Jina AI's Reader, and Crawl4AI all preserve heading levels and table structure, and Trafilatura's whole purpose is separating main content from boilerplate before anything downstream sees it. Bad boundaries at the extraction step produce bad chunks that no chunker can repair.

For a product team the practical sequence is: fix extraction first, then pick a chunking strategy, then tune size. Start with structure-aware splitting on headings under a token ceiling, and reach for semantic or LLM-based chunking only once you can measure that it helps on your own queries. Both cost an embedding or inference call per document at ingest, which is real money at corpus scale and buys nothing if the retriever was already surfacing the right passage. Size also interacts with query type: NVIDIA's write-up found short factoid queries did well on 256 to 512 token chunks while analytical queries did better at 1,024 tokens or whole pages, so a corpus serving both kinds of question may need two indexes rather than one compromise size.