RAG Fundamentals and Design Patterns — Embeddings, Chunking, Reranking, and Evaluation from Scratch

Retrieval-Augmented Generation explained from zero: the core intuition, chunking strategies, hybrid search and reranking, and the evaluation design that matters most.

What RAG is: letting the LLM take an open-book exam

LLMs have two structural weaknesses. Their knowledge is frozen at training time, and when asked about things they don't know, they tend to produce plausible-sounding fabrications (hallucinations). They simply cannot answer questions about your internal policies or last week's product specs — that information was never in their training data.

RAG (Retrieval-Augmented Generation) fixes this by changing the format of the exam. Instead of a closed-book test answered from memory, it becomes an open-book exam:

1. When a question arrives, first search a document collection for relevant pages
2. Put the retrieved pages into the prompt alongside the question
3. The LLM answers with the instruction "base your answer on these documents"

The term RAG comes from a 2020 paper by Lewis et al., but today it refers broadly to any architecture that retrieves external knowledge and injects it into the prompt. Unlike fine-tuning, the model itself is untouched — which means updating knowledge is just swapping documents, and answers can cite their sources. Those two properties are why RAG is usually the first tool to reach for.

The pipeline: five stages

RAG splits into an offline phase (indexing) and an online phase (retrieval and generation):

[Indexing]  documents → chunking → embedding → store in vector DB
[Serving]   question → embedding → similarity search → (reranking) → prompt assembly → generation

In this article we take each stage apart in order, then finish with the part most teams skip: knowing where the pipeline breaks, via evaluation.

Embeddings: turning meaning into coordinates

An embedding converts text into a vector of hundreds or thousands of numbers. The single property that matters: the model is trained so that texts with similar meanings land close together in vector space.

v1 = embed("How do I request paid leave?")
v2 = embed("What's the process for taking time off?")
v3 = embed("Server restart procedure")

cosine_sim(v1, v2)  # high  (different wording, same meaning)
cosine_sim(v1, v3)  # low

Unlike keyword search, embeddings can match by meaning even when no words overlap. The flip side: they are weak at exact-string matches — part numbers, person names, error codes — which is precisely the motivation for hybrid search below.

The "similarity" here is nothing more than the dot product (cosine similarity) between vectors: the closer two vectors point in the same direction, the higher the score. It's exactly the geometry of the dot product from the linear algebra article — play with the figure below to feel how the angle drives the score.

Chunking: where most of your RAG quality is decided

Embed a whole document and its meaning averages out into a blur; chop too finely and you lose context. Guidelines for splitting:

• Fixed-size chunking: simplest to implement. A common starting point is roughly 512–1024 tokens with 10–20% overlap (overlap softens the damage of cutting mid-thought)
• Structure-based chunking: split on the document's logical structure — Markdown headings, paragraphs, clauses. For structured documents (policies, manuals, API docs) this consistently beats fixed-size splitting
• Semantic chunking: cut where the embedding similarity between adjacent sentences drops sharply. Costs more compute but respects topic boundaries
• Parent-child chunks (small-to-big): search over small chunks for precision, but hand the LLM the parent chunk (a larger block including surrounding context). A practical staple that gets you both retrieval precision and generation context

The difference between the metrics (dot, cosine, distance) is best learned in the shape of the accident it causes. Drag the query and watch the cluster of long vectors cut into the top-k only under the dot product.

The mathematics underneath — from the dot product to eigenvalues and low rank — is collected in The linear algebra under LoRA and RAG.