Chunking

Every RAG system makes a decision before any query arrives: how to cut documents into the segments that will be embedded, indexed, and retrieved. The decision looks like plumbing and acts like destiny. An embedding represents its chunk's overall meaning — a chunk spanning five topics embeds as a muddy average no query matches well; a chunk of one sentence matches sharply but arrives stripped of the context that made it true. Retrieval quality, and therefore generation quality, is bounded by what the chunks made findable.

The granularity trade is the core tension. Large chunks preserve context but dilute embeddings and waste prompt budget on irrelevant surroundings; small chunks sharpen matching but fragment meaning — the retrieved sentence whose critical caveat lived in the paragraph above it. Overlap between adjacent chunks softens boundary damage, and the working optimum for most corpora sits in the few-hundred-token range — but the honest answer is empirical, and corpus-specific.

Strategy has converged on respecting structure. Documents carry their own segmentation — sections, headings, paragraphs, list items, table boundaries — and splitting along it consistently beats fixed-size cutting, because human-authored boundaries already wrap coherent units of meaning. Semantic chunking goes further, detecting topic shifts by embedding similarity; hierarchical schemes index small chunks for matching while returning their parent sections for context; metadata (source, section, headings) rides along to preserve provenance. Real-world parsing — PDFs, tables, slide decks — is where these pipelines earn their keep or quietly fail.

The operational guidance is unglamorous: chunking is a tunable system parameter, not a set-and-forget default. Retrieval evaluation against representative queries — testing sizes, boundaries, and overlap — routinely yields double-digit retrieval gains over framework defaults, making it among the highest-ROI optimizations in any RAG stack. When a RAG system underperforms, audit the chunks before blaming the model: read what retrieval actually returned, and the fragmentation usually explains the hallucination.