Chunking Strategies — How You Split Decides What You Can Find

Most of a RAG system's quality is decided by how you split documents. The arithmetic of fixed-size chunks and overlap, structural and semantic splitting, parent-child chunks, and the two things that break every splitter: tables and equations.

Taking a thick manual to the guillotine

You have a 500-page operations manual, and you want to be able to pull out just the part you need. So you cut the binding off and file the loose pages. That is where the first decision shows up: what unit do you bundle them in?

File every sheet separately and you can pull an exact page. But if an explanation starts halfway down page 1 and finishes on page 2, either page alone is useless. Bundle by chapter instead and nothing gets cut mid-thought — but now your answer is "somewhere in these thirty pages," and the bundle itself no longer has a single subject.

Chunking in RAG (retrieval-augmented generation) is exactly this cutting job. The awkward part is that it happens before retrieval. Split badly and no embedding model or reranker downstream can recover what you threw away. The full pipeline is in RAG Fundamentals and Design Patterns; this article drills into one stage of it.

Why split at all

Three reasons, and they are independent of each other.

First, an embedding is a single vector. When you compress text into a few hundred numbers, the longer the input, the more distinct topics get crushed into one direction. A blob containing the leave policy, expense reimbursement, and fire drills ends up pointing somewhere blandly equidistant from all three. For how embeddings work, see Understanding Embeddings from Scratch.

Second, there is a ceiling on what you can hand the LLM. The context window is finite, and token count translates directly into cost and latency.

Third, the granularity of your citation is the granularity of your evidence. "The basis is somewhere in these thirty pages" is not something a user can verify.

Too big and too small fail in different shapes

When chunks are too big, you get dilution. Only three lines in the blob actually relate to the question, but the other few hundred lines join the average, similarity gets watered down, and the chunk never surfaces. Even when it does surface, irrelevant sentences occupy most of the context and the generator is more likely to latch onto the wrong basis.

When chunks are too small, you get missing context. Pronouns like "the company" or "in that case" point at something now living in a different chunk. A single table row gets cut out and loses its column headers. A proviso matches on its own, detached from the rule it modifies. Each of these is correct as a fragment and wrong as an answer.

So there are two different accidents waiting on either side, and the sweet spot moves with the document type. That is why no universal chunk size exists.

There is a subtler coupling too: changing chunk length shifts the distribution of embedding vector lengths (norms). If you search by unnormalized dot product, that alone lets long chunks muscle their way to the top. Drag the query in the figure below and switch the metric between dot product, cosine, and L2 — the same document set gives you a different top 5.

Fixed-size splitting: get the arithmetic straight first

The simplest approach cuts at a fixed length and softens the mid-sentence breaks with overlap — repeating the tail of one chunk at the head of the next. In practice this is a splitter like RecursiveCharacterTextSplitter with chunk_size and chunk_overlap, and it is where most teams start.

What matters is that these two knobs directly set the size and cost of your index. Let $L$ be the length of the document, $c$ the chunk length, and $o$ the overlap. The number of chunks is:

Read in words, the formula is only counting how many slices the document falls into. Each slice holds $c$ characters, but its last $o$ characters get copied onto the next slice as well, so you only advance $c - o$ per cut — and the brackets $\lceil\ \rceil$ say that a leftover partial slice still counts as one whole chunk.

The denominator $c - o$ is how much new text each step consumes — the stride. Increase the overlap and each step gets shorter, so the same document yields more chunks. Write the overlap ratio as $r = o / c$ and the total tokens you send to the embedding model takes a more intuitive form:

The left side, $N \cdot c$, is every chunk length added together — the total amount of text the embedding model actually reads — which says that number lands near the length of the original document divided by $1 - r$.

Meaning: 20% overlap costs about 1.25× in embedding spend and index size; 50% costs about 2×. Equation (2) is just saying you are buying the overlapped characters twice. Overlap is not free insurance.

The other trap is the unit of length. A single character often becomes several tokens in Japanese, Chinese, or Korean, so "1,000 characters" can quietly overflow the embedding model's input limit and have its tail silently dropped (see Understanding Tokenizers from Scratch). The rule is simple: hand the splitter the same tokenizer you use in production.

Structural splitting: use the boundaries the document already has

Fixed-size splitting sees a document as nothing but a stream of characters. Real documents already carry boundaries the author put there: Markdown headings, HTML sections, numbered clauses in a policy, one section per endpoint in API docs. Cut on those and you land close to the ideal of one chunk, one topic.

The bigger win, though, is less about where you cut than about prepending the heading hierarchy to the chunk body.

# Prepend the heading breadcrumb before embedding
header = " > ".join(["Employee Handbook", "Ch.5 Leave", "§21 Paid Leave"])
text_to_embed = f"{header}\n\n{chunk_body}"

Now even a fragment that reads only "may be carried over up to the number of days in the preceding paragraph" produces a vector that points toward "leave." You are letting the embedding do what a human does when they glance at the table of contents.

Semantic splitting: find where the topic turns

For documents with little structure — meeting notes, transcripts, long-form prose — you infer the boundaries from the content itself. This is semantic chunking.

Embed sentence by sentence and line up the similarity between neighbours. Writing $e_i$ for the embedding of sentence $i$:

Put in words: turn each of the two neighbouring sentences into an arrow and measure how closely the arrows point the same way. Near 1 means the same direction — the same thread continuing; near 0 means they have veered apart.

$s_i$ measures how much sentence $i$ and the sentence after it are talking about the same thing. Where this value drops off sharply is treated as a topic boundary, and that is where you cut. Setting the threshold by percentile within the document (say, the bottom 5% of $s_i$) holds up better across document types than a fixed absolute value.

Honestly, semantic chunking does not always beat fixed-size. The preprocessing is heavy, the threshold needs tuning per domain, and in documents that are mostly bullet lists the similarity bounces around and the boundaries stop meaning anything. The practical rule: structural splitting for structured documents, semantic splitting for prose that drifts between topics.

Parent-child chunks: search small, hand over big

"Smaller retrieves more precisely" and "bigger carries more context" are not actually in conflict. Keep the search index and the text you hand the LLM as separate things.

hits = index.search(query, k=20)              # match on small chunks
parents = {h.parent_id for h in hits}         # fold to parent IDs, dedup
context = [store.get(pid) for pid in parents] # hand over the big blocks

This goes by small-to-big, parent-child chunking, or — depending on the implementation — sentence-window (restore the $n$ sentences around the hit) and auto-merging (promote to the parent once enough of its children match). Same idea underneath: decouple the granularity you match on from the granularity you read from. Note the set in that snippet — it deduplicates, and without it you can end up with top_k of 10 where the same paragraph appears five times.

The hard cases: tables, equations, code

Prose-oriented splitters break on these three with high probability.

Tables lose their column headers the moment a row is cut out on its own. A hit on "120,000" tells you nothing about what that amount is. Either keep the whole table as one chunk, or, if you must split by row, repeat the column headers into every row — expand it into something self-contained like dept: Sales / category: Travel / cap: 120,000. Attaching the caption and the sentence introducing the table changes retrieval noticeably too.

Equations are awkward twice over. They tend to break during extraction from PDF or HTML in the first place. And even when the $\LaTeX$ comes through cleanly, a string like \frac{\partial L}{\partial w} is a nearly meaningless symbol sequence to an embedding model and will not connect to a query about "gradients." The practical move is to not make the equation itself the thing you match on: keep it in the same chunk as the surrounding prose, and let retrieval land on the explanation around it ("taking the partial derivative of the loss with respect to the weights"). Keep equation numbers and captions in that chunk as well.

Code should be cut at function and class boundaries. Fixed-size splitting produces fragments that have lost their def line, and nobody — model or human — can tell what the function does. Most splitters carry language-aware separators; use those, or parse and emit one chunk per top-level definition.

top_k moves with chunk length

On top of the norm problem from the figure earlier, the counts stop balancing. Halve your chunk length and you need to double top_k to deliver the same amount of information. A comparison that changes chunk size while holding top_k fixed is really a comparison that changed how much context you passed, and you will not be able to tell which change did the work.

Deciding: build an evaluation that survives a change of splitter

Comparing chunking strategies has a trap of its own. When you measure retrieval with Recall@k (how often the right item is in the top k), the definition of "the correct chunk" changes with every splitting method. Relabelling ground truth each time is not a comparison.

The fix is to store ground truth as a span in the source text, not as a chunk. Record the question plus the document ID and the character range where the answer lives; after splitting, count any chunk that overlaps that span as correct. Now fixed-size, structural, and semantic splits can all be measured against the same evaluation set, side by side.

How this shows up on the job

The people who actually touch chunking are the backend and ML engineers building internal document QA and support assistants. More often than designing from scratch, you inherit a RAG system that runs but does not perform — and this is the first stage to suspect.

Parameters and tools you will touch: the splitter's chunk_size / chunk_overlap, the tokenizer you hand it (the same one your production embedding model uses), the header levels for a Markdown header splitter, the percentile threshold for semantic splitting, the parent size for parent-child retrieval, and on the retrieval side top_k plus the reranker's maximum input length.

Failure modes that bite if you do not know them:

• Changing the splitter means reindexing everything. Add new chunks while the old ones are still there and the same text is registered twice, with duplicates eating your top_k budget
• Do not misdiagnose an extraction failure as a chunking failure. Two-column PDFs interleaved line by line, headers and footers bleeding into every page — no chunk size fixes a broken extractor. Read a few dozen extracted texts with your own eyes first
• Anything past the embedding model's input limit may be silently truncated. You configure long chunks and the tails quietly never make it into the index
• Rerankers have input limits too. If the large chunks from stage one get truncated at the reranker, it is scoring only the opening paragraph

How it gets asked in a design review: answering "how did you pick the chunk size?" with a number alone is weak. What lands is the reasoning chain — document type (structured or not) → choice of splitting method → Recall@k comparison on an evaluation set anchored to source spans → validation with top_k and every other condition held equal.

Summary

• Chunking runs before retrieval; information dropped here cannot be recovered by any model downstream
• Too big dilutes, too small strips context. The two failures differ in shape, so the sweet spot moves with the document type
• Fixed-size chunking has readable arithmetic (overlap ratio $r$ costs $1/(1-r)$×). Split structured documents on structure, drifting prose on meaning
• Parent-child chunking is the standard way to decouple match granularity from read granularity. Repeat column headers into every table chunk, and let equations be found through the prose around them
• Anchor evaluation ground truth to spans in the source text — otherwise you cannot compare splitting methods at all

Next, on how to pick those chunks back up: the design of hybrid search and reranking, where the granularity decisions made here start to pay off.