RAG Chunking Is an Evaluation Question, Not a Magic Number
Chunk size affects retrieval recall, context noise, and citation quality. Use an evaluation set to choose boundaries instead of copying a popular token count.
Why one chunk size cannot be universal
A policy document, a source file, and a support ticket do not carry meaning in the same shape. Splitting every input into 500-token blocks may be convenient, but it can separate a heading from its conditions, combine unrelated sections, or make a citation too broad to verify.
The right question is not “what chunk size does everyone use?” It is “which boundaries preserve the evidence needed for this question?” For a policy, that may mean keeping a rule with its exceptions. For code, it may mean a function plus the relevant comment and signature. For a ticket, the resolution may need the original symptom and environment.
Build a small, representative set
Collect questions from actual users and label the evidence needed to answer them. Include queries that target headings, tables, versioned instructions, and documents with similar vocabulary. Add cases where the answer is not present. A retrieval system that returns a confident near-match for an unanswerable question is a quality failure, not a success.
For each chunking candidate, compare:
- retrieval recall for the relevant evidence
- retrieval precision among the top results
- whether the returned chunk is understandable without hidden context
- citation coverage and the effort required to verify it
- token cost and latency at the production top-k
Keep the embedding model, query set, filters, and top-k constant while comparing candidates. Otherwise you are testing several variables at once.
Prefer semantic boundaries, then add controlled overlap
Start with document structure: headings, paragraphs, list items, table rows, or code symbols. Add overlap only where a boundary regularly loses context. Large overlap can inflate the index with near-duplicates and make precision look worse.
Store metadata such as document ID, section path, version, access scope, and source URL. Metadata can improve filtering and citations, but it cannot repair text that was extracted incorrectly. Inspect the chunks themselves before blaming embeddings.
Read the failure examples
Aggregate scores tell you which candidate wins overall; failure examples tell you why. A higher recall configuration may retrieve a complete answer but also six irrelevant versions. A smaller configuration may be cheaper but omit the exception that changes the result. Make the tradeoff explicit for the product: support answers, compliance research, and internal search have different tolerance for each failure.
Run the RAG Retrieval Evaluation Workbook on a labeled sample, then keep the sample as a regression set. Chunking is part of the data contract. Treat it like code: version the change, record the evidence, and rerun the tests after ingestion changes.