Evaluating RAG in Practice — Turning “Seems Better” Into a Number

RAG improvement work gets lost because quality is judged as one vague feeling. This article builds up the metrics that separate retrieval from generation, the definitions of faithfulness and relevance, how to build an eval set with synthetic QA and where that goes wrong, and how many questions you need before the number means anything.

RAGAS: Automated Evaluation of Retrieval Augmented Generation

Where do you actually look at a junior's report?

You ask a new hire to find out how your parental leave policy treats an employee whose spouse is self-employed. Thirty minutes later a report lands on your desk. If you skim it and say "looks fine," there is no reason the next report will be any better — you never told them what was good about this one.

There are at least three separate things to look at.

1. Did they find the right source at all? If they never opened Chapter 8 of the employment rules, everything downstream is guesswork.
2. Did they write only what the sources actually say? A plausible sentence that appears in none of the documents is the nastiest failure of the three.
3. Did they answer the question that was asked? A careful summary of parental leave in general is still a failed report if it never gets to the self-employed-spouse case.

Evaluating RAG (retrieval-augmented generation) has exactly this shape. The pipeline itself is covered in RAG Fundamentals and Design Patterns; this article is only about turning those three questions into numbers.

One number doesn't tell you what to fix

Suppose you treat RAG as a single box, score only the final answer, and get 0.62. What do you change tomorrow?

You can't say. RAG has at least two stages, so failure comes in four flavors.

The two cells on the right demand completely different responses: top-right points at your prompt and generator, bottom-right at your chunking and embedding model. And the "fluke" in the bottom-left is the most dangerous cell of all, because if your eval set is full of well-known general knowledge, retrieval can be broken without the score moving. Evaluate an internal-documents RAG with common-knowledge questions and you walk straight into this.

So there is one principle: measure retrieval and generation separately. And the order is fixed. A question whose answer never gets retrieved cannot be answered correctly no matter how much you polish the generator. Get retrieval healthy on its own first, then look at generation.

Measuring retrieval: is it in there, and is it near the top?

Retrieval evaluation starts by attaching a set of gold chunks to each question. Write the gold set for question $q$ as $G$, and the top $k$ results returned by the retriever as $R_k$.

In words: of the passages you wanted retrieved, what fraction landed in the top k? The denominator is the total number of gold chunks, so if three are relevant and you retrieve one, you get 0.33. This number is the ceiling on generation — if it sits at 0.6, your answer accuracy cannot exceed 0.6 either, flukes aside.

But Recall@k ignores rank. A gold chunk at position 1 and one at position 10 score identically, even though documents placed earlier in the context tend to get used more. That is what nDCG@k is for. First, sum the relevance scores while discounting by rank.

Here $\mathrm{rel}_i$ is the relevance of the document at rank $i$ (0 for irrelevant, 1 for relevant, 2 for highly relevant, and so on), and $\log_2(i+1)$ is a discount that grows as you go down the list. Rank 1 is divided by $\log_2 2 = 1$, so it counts in full; rank 2 is divided by about 1.58, rank 8 by about 3.17. In words, the formula says a hit further down the list is worth less. Divide by the DCG of the ideal ordering and you get nDCG, normalized to 0–1.

One more that comes up constantly: MRR (Mean Reciprocal Rank), the average of the reciprocal of the rank of the first correct hit. Third position gives $1/3$. For question types where a single correct passage is enough — FAQ lookup, say — this is the more honest number.

What is easy to miss here is that the choice of similarity metric alone reshuffles the top of the list. Drag the query point in the figure below and switch between dot product, cosine, and L2. Same document set, different top 5. Recall@k is what turns an unglamorous configuration difference like this into a number you can act on.

How you cut the documents in the first place is its own lever, covered in Chunking Strategies — and Recall@k and nDCG@k are exactly the yardsticks you use to compare one splitting scheme against another.

Measuring generation: faithfulness and relevance

Once retrieval is healthy, move to generation. What you want here is items 2 and 3 from the top of the article: faithfulness and answer relevance. They are independent, and it is entirely normal to be strong on one and weak on the other.

Faithfulness asks whether each claim in the answer is supported by the context you handed over. The basic recipe is to decompose the answer into individual claims and check each one against the context.

In words: of everything the answer asserted, what fraction has evidence behind it? The important part is not scoring the answer as binary hallucinated / not-hallucinated. Real answers break in the shape of four correct sentences and one invented one. Score it binary and you either miss that sentence entirely, or mark the whole answer wrong and lose the signal about where it went wrong.