Chunking

Article.

The highest-leverage decision in a RAG system, and the one most often made by copying chunk_size=1000 from a tutorial. It decides what can be retrieved at all — no reranker recovers a fact that got split across two chunks and neither half made the cut.

It is also the decision that is most expensive to change, which is a good reason to understand the mechanism before picking a number.

The mechanism: one vector for the whole chunk

An embedding model reads a chunk and produces a single vector. Everything in that chunk — every topic, every fact — is compressed into one point.

Which means a chunk containing two unrelated things produces a vector that sits between them, close to neither:

chunk A: "Refunds are processed within 5 days."
         -> vector near [refunds]

chunk B: "Refunds are processed within 5 days.
          Our Berlin office opens at 9am."
         -> vector between [refunds] and [hours]

Query “how long do refunds take” retrieves A confidently and B weakly, because B’s vector was pulled away by content the query does not care about. Mixing topics in one chunk dilutes every topic in it.

That single fact explains most of what follows: why boundaries should fall where topics change, why very large chunks retrieve badly, and why semantic chunking exists at all.

The opposite failure is just as real. Split too small and a fact loses the context that makes it findable — “it expires after 30 days” is unretrievable for any query that does not already know what “it” is.

Strategies

Fixed size with overlap

The baseline. Split every N tokens with a small overlap. Fast, predictable, and blind to structure — it will cut mid-sentence and mid-table.

Worth keeping as the thing you measure against, not as the answer.

Recursive character splitting

Try to split on paragraph breaks; failing that, sentences; failing that, whitespace; failing that, characters. Respects structure where structure exists and degrades gracefully where it does not.

This is the sensible default for prose, and what most libraries do when you do not choose.

Document-aware

Use the document’s own structure — markdown headings, HTML sections, function and class boundaries in code.

# Code split by structure, not by character count.
splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON, chunk_size=800, chunk_overlap=0,
)

For code this is not a preference. A function split at character 800 produces two chunks that are each syntactically meaningless, and an embedding of half a function matches almost nothing a developer would ask.

Semantic chunking

Embed sentences, and start a new chunk where consecutive similarity drops — placing boundaries exactly where the topic changes, which is precisely what the mechanism above says you want.

The cost is an embedding call per sentence at ingest, and boundaries that vary with the model. Reach for it when documents lack usable structure; skip it when they have headings, because headings already encode the same information for free.

Parent-document retrieval

The most useful pattern here, because it refuses the trade-off entirely: embed small, return large.

def index(doc):
    parent_id = store_parent(doc.text)               # the full section
    for chunk in split(doc, chunk_size=200):         # small, precise
        vector_db.add(embed(chunk), metadata={"parent_id": parent_id})

def retrieve(query, top_k=3):
    hits = vector_db.search(embed(query), top_k=top_k)
    parents = {h["parent_id"] for h in hits}         # dedupe
    return [load_parent(p) for p in parents]

Small chunks give a clean, undiluted vector; the parent gives the model enough context to answer. When someone asks how to get precision and context, this is the answer.

Note the set — several matching chunks often share a parent, and returning it three times wastes context window on duplicates.

Hierarchical indexing

Summaries at document level, section level, then the chunks themselves. Query routes down the levels. Genuinely useful on very large corpora, and overkill below that.

Choosing a size

These are starting points, not answers. The only honest method is to measure: build a held-out set of question/expected-chunk pairs and sweep the parameter.

for size in (200, 400, 800, 1200):
    index = build_index(docs, chunk_size=size, overlap=int(size * 0.15))
    print(size, recall_at_k(index, eval_set, k=10))

If that eval set does not exist, building it is the higher-priority task — see Building eval sets.

Overlap

Some content shared between adjacent chunks, so a fact spanning a boundary survives in at least one of them.

10-20% is the working range. Higher for technical documents where exact wording matters; lower for narrative. Overlap costs storage and produces near-duplicate results in the top-K, which is a reason to dedupe before the reranker sees them.

Where it breaks in production

Re-chunking is a full re-index. This is the one people discover late. Changing chunk_size means re-embedding the entire corpus: an API bill proportional to your document count, hours of ingest, and a window where the index is inconsistent unless you build into a new collection and swap.

1M chunks re-embedded = an invoice + hours + a swap

Treat chunk size as a schema decision, not a config value. Sweep it on a sample before the first full ingest.

Tables destroyed by a character splitter. A markdown table cut at 800 characters yields a header with no rows and rows with no header, both useless. Extract tables during parsing and chunk them as units — see Structured Document Pipelines — Parsing, Tables, OCR, Extraction.

Embedding the body but not the title. A chunk reading “Processed within 5 working days” is far more discriminative embedded as “Refund policy > Timing: processed within 5 working days”. Prepend the section path to the embedded text even when you do not show it to the model.

Chunks with no route back. A chunk that cannot name its source document, page or section cannot be cited, and an answer without citations cannot be verified. Store the provenance at ingest; adding it later means re-indexing.

A default nobody measured. chunk_size=1000, overlap=200 appears in a startling number of production systems because it is the first example in a tutorial. It is not wrong. It is unmeasured, which is different.

The trade not taken

The most under-considered option is not chunking at all.

If documents are small enough to pass whole, do that — retrieval at document granularity has no boundary problem, no dilution, and nothing to tune. With large context windows this covers more cases than it used to.

And if the corpus itself is small, skip retrieval entirely and put everything in the prompt. You remove chunking, embedding, a vector store and their failure modes in one decision. RAG earns its complexity on corpora too large to fit — which is a narrower set than the enthusiasm suggests.

Related

• Hybrid search and reranking - retrieving the chunks once they exist
• Structured Document Pipelines — Parsing, Tables, OCR, Extraction - getting clean text before you split it
• What are embeddings - what the vector encodes