RAG Fundamentals and Design Patterns — Embeddings, Chunking, Reranking, and Evaluation from Scratch

Retrieval-Augmented Generation explained from zero: the core intuition, chunking strategies, hybrid search and reranking, and the evaluation design that matters most.

What RAG is: letting the LLM take an open-book exam

LLMs have two structural weaknesses. Their knowledge is frozen at training time, and when asked about things they don't know, they tend to produce plausible-sounding fabrications (hallucinations). They simply cannot answer questions about your internal policies or last week's product specs — that information was never in their training data.

RAG (Retrieval-Augmented Generation) fixes this by changing the format of the exam. Instead of a closed-book test answered from memory, it becomes an open-book exam:

1. When a question arrives, first search a document collection for relevant pages
2. Put the retrieved pages into the prompt alongside the question
3. The LLM answers with the instruction "base your answer on these documents"

The term RAG comes from a 2020 paper by Lewis et al., but today it refers broadly to any architecture that retrieves external knowledge and injects it into the prompt. Unlike fine-tuning, the model itself is untouched — which means updating knowledge is just swapping documents, and answers can cite their sources. Those two properties are why RAG is usually the first tool to reach for.

The pipeline: five stages

RAG splits into an offline phase (indexing) and an online phase (retrieval and generation):

[Indexing]  documents → chunking → embedding → store in vector DB
[Serving]   question → embedding → similarity search → (reranking) → prompt assembly → generation

In this article we take each stage apart in order, then finish with the part most teams skip: knowing where the pipeline breaks, via evaluation.

Embeddings: turning meaning into coordinates

An embedding converts text into a vector of hundreds or thousands of numbers. The single property that matters: the model is trained so that texts with similar meanings land close together in vector space.

v1 = embed("How do I request paid leave?")
v2 = embed("What's the process for taking time off?")
v3 = embed("Server restart procedure")

cosine_sim(v1, v2)  # high  (different wording, same meaning)
cosine_sim(v1, v3)  # low

Unlike keyword search, embeddings can match by meaning even when no words overlap. The flip side: they are weak at exact-string matches — part numbers, person names, error codes — which is precisely the motivation for hybrid search below.

The "similarity" here is nothing more than the dot product (cosine similarity) between vectors: the closer two vectors point in the same direction, the higher the score. It's exactly the geometry of the dot product from the linear algebra article — play with the figure below to feel how the angle drives the score.

Chunking: where most of your RAG quality is decided

Embed a whole document and its meaning averages out into a blur; chop too finely and you lose context. Guidelines for splitting:

• Fixed-size chunking: simplest to implement. A common starting point is roughly 512–1024 tokens with 10–20% overlap (overlap softens the damage of cutting mid-thought)
• Structure-based chunking: split on the document's logical structure — Markdown headings, paragraphs, clauses. For structured documents (policies, manuals, API docs) this consistently beats fixed-size splitting
• Semantic chunking: cut where the embedding similarity between adjacent sentences drops sharply. Costs more compute but respects topic boundaries
• Parent-child chunks (small-to-big): search over small chunks for precision, but hand the LLM the parent chunk (a larger block including surrounding context). A practical staple that gets you both retrieval precision and generation context

The difference between the metrics (dot, cosine, distance) is best learned in the shape of the accident it causes. Drag the query and watch the cluster of long vectors cut into the top-k only under the dot product.

The mathematics underneath — from the dot product to eigenvalues and low rank — is collected in The linear algebra under LoRA and RAG.

One more high-leverage practice: attach metadata. Give each chunk its document title, section, and last-updated date. Use it for filtering at query time (e.g. latest version only) and prepend the title to the chunk text — both reliably improve accuracy.

The classic way to get chunking wrong is to cut mechanically through the middle of tables, code blocks, or lists. A chunk that contains only a few table rows has lost the information about what the table even is: it won't be retrieved when it should be, and even if it is, it's useless as evidence for the answer. Even with fixed-size chunking, at minimum keep tables and code on one side of a boundary — that alone removes a whole class of "retrieval looks right but the answer is broken" failures. Likewise, if your PDF text extraction is garbled (multi-column reading order scrambled, headers and footers leaking into the body), no amount of downstream tuning will recover it. Before designing chunks, eyeball the extracted text. It looks like a detour; it's the shortcut.

Retrieval: dense, BM25, and hybrid

• Vector search (dense): nearest-neighbor search over embeddings. Strong on paraphrases
• BM25 (keyword / sparse): classic full-text scoring. Strong on proper nouns, part numbers, code fragments
• Hybrid: run both and merge the results. RRF (Reciprocal Rank Fusion) — summing the reciprocal of each result's rank across lists — is simple and widely used for the merge

In production, hybrid should be your default. Real user questions are always a mix of "asking by meaning" and "asking by exact string."

Reranking: retrieve coarsely, then sort intelligently

Vector search is fast because it embeds the question and the chunks separately and measures distance (the bi-encoder approach) — but that speed caps its precision. So use two stages:

1. Stage 1 (Retrieval): fetch the top 50–100 candidates quickly via vector or hybrid search
2. Stage 2 (Reranking): re-sort with a cross-encoder — a model that takes the question and a chunk together as one input and scores relevance directly — then pass only the top 5–10 to the LLM

A cross-encoder needs one inference per pair, so it's too slow for the whole corpus, but perfectly practical on a shortlist. "Cast a fast, coarse net wide; then select with slow, accurate eyes" — this two-stage setup is often the single most cost-effective upgrade to RAG quality.

Evaluation: graduating from "it feels better now"

RAG has many moving parts, and improving by gut feel gets you lost fast. The core principle: measure retrieval and generation separately.

Evaluating retrieval (before generation)

• Build a few dozen question → "correct chunk" pairs and measure Recall@k — how often the right chunk appears in the top k results
• No matter how good your generation is, the system cannot answer correctly if the right information was never retrieved. Fix retrieval in isolation first

Evaluating generation

• Faithfulness: is the answer grounded in the provided context (i.e., not hallucinated)?
• Answer relevance: does it actually answer the question?
• Human review is the gold standard; in practice most teams approximate it with LLM-as-a-judge (a separate LLM scores the outputs). Frameworks like RAGAS exist to automate this

Operational tips

• Grow the eval set continuously from production failure logs (don't try to make it perfect upfront)
• Change one stage at a time. If you swap chunk size and embedding model together, you'll never know which one helped
• Always include "no relevant document exists" cases in the eval set. Refusing to answer unanswerable questions is also RAG quality

How this shows up on the job

The people who build and run RAG in practice are backend and ML engineers shipping internal document Q&A bots and support assistants. And more often than building from scratch, the real assignment is "inherit a RAG system that works but isn't accurate enough, and improve it" — at which point what you touch are exactly the per-stage parameters from this article.

What you actually adjust: on the chunking side, chunk_size and chunk_overlap (in LangChain, the arguments to RecursiveCharacterTextSplitter). On the retrieval side, the top_k candidate count, the hybrid fusion weights or the RRF constant, and metadata filter conditions. The vector store is usually chosen for fit with existing infrastructure — FAISS, pgvector, Qdrant, Elasticsearch — and the reranker is typically just a cross-encoder model dropped in between retrieval and generation.

Pitfalls that cause real incidents:

• Changing the embedding model requires re-indexing every chunk. Vectors from the old and new models do not live in the same space; the moment they mix, distance comparisons become meaningless
• Mismatched normalization and distance metric. Take a model trained for cosine similarity, skip normalization, and search by raw dot product — and chunks with large norms dominate the top of every result list
• Raising top_k is not the same as improving quality. The more irrelevant chunks enter the context, the higher the risk that generation grabs the wrong evidence. If you widen the net, pair it with reranking
• Permission leakage. The vector index knows nothing about the source documents' access control. Without a per-user permission filter at retrieval time, content from documents a user must not see leaks into their answers

Questions you'll face in interviews and design reviews: the staple is "retrieval quality is poor — what do you check first?" The path is the one this article laid out: don't touch the prompt; measure retrieval in isolation (compute Recall@k), then inspect chunk boundaries and extraction preprocessing, and only then consider swapping the embedding model or going hybrid. "When RAG versus fine-tuning?" comes up just as often. The standard answer is by role: injecting and updating knowledge is RAG's job; shaping output format, style, and behavior is fine-tuning's.

Takeaways

• RAG is an open-book exam: knowledge updates are document swaps, and answers can cite sources
• Most quality is decided in chunking and retrieval; prompt tweaks for generation come last
• Hybrid search plus cross-encoder reranking is the cost-effective default architecture
• Evaluate retrieval with Recall@k and generation with faithfulness/relevance, and grow your eval set from failure logs

Next up: advanced RAG patterns — query rewriting, HyDE, GraphRAG, and agentic RAG — compared as design patterns. For the background on agents calling retrieval as a tool, see LLM agents fundamentals.