Chunking Strategies — How You Split Decides What You Can Find

Most of a RAG system's quality is decided by how you split documents. The arithmetic of fixed-size chunks and overlap, structural and semantic splitting, parent-child chunks, and the two things that break every splitter: tables and equations.

Taking a thick manual to the guillotine

You have a 500-page operations manual, and you want to be able to pull out just the part you need. So you cut the binding off and file the loose pages. That is where the first decision shows up: what unit do you bundle them in?

File every sheet separately and you can pull an exact page. But if an explanation starts halfway down page 1 and finishes on page 2, either page alone is useless. Bundle by chapter instead and nothing gets cut mid-thought — but now your answer is "somewhere in these thirty pages," and the bundle itself no longer has a single subject.

Chunking in RAG (retrieval-augmented generation) is exactly this cutting job. The awkward part is that it happens before retrieval. Split badly and no embedding model or reranker downstream can recover what you threw away. The full pipeline is in RAG Fundamentals and Design Patterns; this article drills into one stage of it.

Why split at all

Three reasons, and they are independent of each other.

First, an embedding is a single vector. When you compress text into a few hundred numbers, the longer the input, the more distinct topics get crushed into one direction. A blob containing the leave policy, expense reimbursement, and fire drills ends up pointing somewhere blandly equidistant from all three. For how embeddings work, see Understanding Embeddings from Scratch.

Second, there is a ceiling on what you can hand the LLM. The context window is finite, and token count translates directly into cost and latency.

Third, the granularity of your citation is the granularity of your evidence. "The basis is somewhere in these thirty pages" is not something a user can verify.

Too big and too small fail in different shapes

When chunks are too big, you get dilution. Only three lines in the blob actually relate to the question, but the other few hundred lines join the average, similarity gets watered down, and the chunk never surfaces. Even when it does surface, irrelevant sentences occupy most of the context and the generator is more likely to latch onto the wrong basis.

When chunks are too small, you get missing context. Pronouns like "the company" or "in that case" point at something now living in a different chunk. A single table row gets cut out and loses its column headers. A proviso matches on its own, detached from the rule it modifies. Each of these is correct as a fragment and wrong as an answer.

So there are two different accidents waiting on either side, and the sweet spot moves with the document type. That is why no universal chunk size exists.

There is a subtler coupling too: changing chunk length shifts the distribution of embedding vector lengths (norms). If you search by unnormalized dot product, that alone lets long chunks muscle their way to the top. Drag the query in the figure below and switch the metric between dot product, cosine, and L2 — the same document set gives you a different top 5.

Fixed-size splitting: get the arithmetic straight first

The simplest approach cuts at a fixed length and softens the mid-sentence breaks with overlap — repeating the tail of one chunk at the head of the next. In practice this is a splitter like RecursiveCharacterTextSplitter with chunk_size and chunk_overlap, and it is where most teams start.

What matters is that these two knobs directly set the size and cost of your index. Let $L$ be the length of the document, $c$ the chunk length, and $o$ the overlap. The number of chunks is:

Read in words, the formula is only counting how many slices the document falls into. Each slice holds $c$ characters, but its last $o$ characters get copied onto the next slice as well, so you only advance $c - o$ per cut — and the brackets $\lceil\ \rceil$ say that a leftover partial slice still counts as one whole chunk.

The denominator $c - o$ is how much new text each step consumes — the stride. Increase the overlap and each step gets shorter, so the same document yields more chunks. Write the overlap ratio as $r = o / c$ and the total tokens you send to the embedding model takes a more intuitive form:

The left side, $N \cdot c$, is every chunk length added together — the total amount of text the embedding model actually reads — which says that number lands near the length of the original document divided by $1 - r$.

Meaning: 20% overlap costs about 1.25× in embedding spend and index size; 50% costs about 2×. Equation (2) is just saying you are buying the overlapped characters twice. Overlap is not free insurance.