RAG vs Fine-Tuning — Which One, and When

The two main ways to make an LLM better, compared on four axes: knowledge freshness, cost, hallucination, and data requirements. From metaphor to math to interactive demos to the mistakes that break production systems.

There are always two reasons someone can't answer

Imagine a new hire who fails to answer a customer's question. The cause splits cleanly in two.

Either they don't know the fact. They haven't read the refund policy that was revised last month. What they need isn't training — it's a copy of the policy on their desk.

Or they don't know how to answer. They've read the policy, but their phrasing isn't right for customers, or they don't follow the house format (conclusion first, then the reasoning, then where to escalate). Handing them a second copy of the policy changes nothing. What they need is to see many good answers until the shape sinks in.

Large language models are no different. RAG (retrieval-augmented generation) solves the first problem: at every question, it finds relevant documents and pastes them into the prompt. Fine-tuning solves the second: it rewrites the model's weights — the numbers inside it — through additional training.

This is one of the most-searched comparisons in the field, and most teams get stuck because they start comparing before asking which of the two failures they actually have. This article takes that diagnosis all the way to something you can run mechanically.

The intuition: where you put the knowledge

The difference in one line is where the knowledge lives.

RAG keeps it outside the model. Documents sit in a searchable database, and the model takes an open-book exam every time. Swap a document and the next answer changes.

Fine-tuning dissolves knowledge and behaviour inside the model. Once training is done, it speaks that way without any documents in the prompt. But what you dissolved in cannot be taken back out one line at a time.

That asymmetry decides most of what follows. What you put outside can be swapped. What you put inside cannot.

Mechanism 1: RAG finds what's similar

Retrieval is the heart of RAG. Both documents and queries become arrays of numbers that encode meaning — embedding vectors — and we pick the ones pointing in a similar direction.

In plain terms: the more the question's vector and the document's vector point the same way, the higher the score. $e_q$ is the question turned into a vector, $e_d$ is a chunk of a document turned into a vector, the numerator $e_q \cdot e_d$ is the sum of their element-wise products (the dot product), and $\lVert \cdot \rVert$ in the denominator is a vector's length. Dividing by length cancels out the advantage that merely long vectors would otherwise have — you can see exactly what happens without that division in the demo below.

Take the top $k$ results from equation (1), paste them into the prompt, and generate. That's all of RAG. The components are covered in depth in RAG fundamentals and design patterns.

Mechanism 2: Fine-tuning nudges the weights

Fine-tuning, meanwhile, adds a delta to a weight matrix $W$ inside the model. In LoRA, today's dominant method, that delta is expressed as the product of two thin matrices.

Put differently: freeze the original weights $W$ and stick a thin sticky note $BA$ next to them. $B$ is tall, $A$ is wide, and the narrow dimension between them is called $r$, the rank. Shrink $r$ and the note gets thinner — the number of values you actually train can drop to a tiny fraction of the original weights.

"Can a delta that thin really be enough?" is a fair question, and it's the subject of the original paper (LoRA, explained). Here you only need the fact that it sometimes is.

Deciding on four axes

Four axes carry almost all the weight in practice.

The most important thing in that table is that the hallucination axis behaves differently from the others. The first three are matters of degree; on hallucination the direction flips. When you fine-tune to inject facts, the model first learns the habit of speaking confidently about the topic. Confidence rises before the knowledge is fully in, and hallucination can get worse rather than better. We won't dig into why that happens internally, but as a decision rule it's simple: facts go in RAG, behaviour goes in fine-tuning.

The two-axis map: which quadrant are you in?

Fold those four axes into something you can decide with and you get two. Horizontal: how much you need the behaviour to change. Vertical: how fast the knowledge changes.

Clear the bottom-left quadrant first. If you can get away with building nothing, that's the cheapest possible system, and a handful of examples in the prompt clears the bar more often than people expect.

The top right — current information in a fixed format — is the both-at-once quadrant, and it's where a great many commercial systems land. RAG supplies the facts; fine-tuning stabilises the shape of the output.

The difference in code

In pseudocode it's immediately visible that the two touch different places.

# RAG: add documents from outside at every call (the model is untouched)
def answer(question, index, llm):
    docs = index.search(question, k=5)          # top 5 by equation (1)
    context = "\n\n".join(d.text for d in docs)
    return llm(f"Documents:\n{context}\n\nQuestion: {question}")

# Fine-tuning: rewrite the weights once (then call the model bare)
def train(base_model, pairs):                   # pairs = [(input, target output), ...]
    model = attach_lora(base_model, r=16)       # insert ΔW=BA
    for x, y in pairs:
        loss = cross_entropy(model(x), y)
        loss.backward(); step()
    return model                                # afterwards, just model(question)

RAG runs index.search every time; fine-tuning runs train once. The first keeps paying in latency and tokens, the second pays up front and keeps inference light.

Three common misconceptions

"Fine-tuning will teach it our internal documents." Sometimes it does, but never reliably, and you can't update it. When one clause out of a hundred thousand changes, RAG swaps one document. Fine-tuning means another training run for that one clause.

"RAG is cheap." Cheap to build, not necessarily cheap to run. Every question carries thousands of tokens of retrieved text, multiplied by your traffic. If the front of the prompt is fixed, prompt caching claws some of that back, but the retrieved documents change per question, so the benefit is bounded.

"You pick one." They aren't rivals. As the top-right quadrant shows, using both is the normal case.

How this plays out on the job

Who decides, and when. The person running internal AI infrastructure, or an ML engineer, decides this in the first thirty minutes after someone asks for "an LLM that does X." Skipping it and going straight to implementation is what makes the rework expensive.

The diagnosis can be a fixed procedure. (1) Collect twenty outputs that are failing. (2) For each one, check whether handing it the right document would fix it. (3) High fix rate means RAG; low fix rate means fine-tuning. Step (2) is just pasting documents into a prompt by hand, so you get the answer before writing any code.

Parameters and tools you'll actually touch. On the RAG side: chunk size and overlap, the retrieval count top_k, choice of embedding model, whether to blend in keyword search like BM25, and whether to add a reranker. For vector storage, pgvector, FAISS, and Qdrant are common. On the fine-tuning side: LoRA's r and alpha, learning rate, number of epochs, and above all the volume and quality of training pairs. PEFT, TRL, Axolotl, and Unsloth are the usual implementations.

Traps that turn into incidents.

• Personal data in training data can't be removed later. With RAG you delete the document from the index and you're done; anything baked into weights leaves you unable to honour a deletion request. In domains where such requests are plausible, this single point can decide the choice on its own.
• Documents updated, index not rebuilt. The most common RAG failure by far. Almost every "we updated it but it still returns the old answer" is a pipeline missing a re-indexing step.
• Catastrophic forgetting. Train too hard on narrow data and general capabilities the model already had degrade. Always measure tasks other than the one you trained on afterwards.
• Starting without an evaluation set. Neither approach can be judged without a way to measure whether the problem got fixed. For RAG you have to measure retrieval and generation separately (Evaluating RAG).

How this gets asked in interviews and design reviews. Given "build a bot that answers questions about our employee handbook — which approach?", an answer that shows you're using the four axes sounds like: "RAG, on three grounds — the handbook gets revised, citations will be demanded, and we have no question-answer pairs. If there's a mandated response format, add that afterwards with prompting or a light fine-tune, separately."

Summary

• RAG keeps knowledge outside; fine-tuning dissolves it inside. Outside can be swapped, inside cannot
• The four axes are freshness, cost, hallucination, and data. Hallucination is the one that runs backwards: fine-tuning for knowledge can make it worse
• Use the two-axis map to pick a quadrant, and clear the bottom-left (build nothing) first
• Most commercial systems end up using both. Don't frame it as a rivalry

The next step is measuring whichever side you picked. If you chose RAG, start with an evaluation design that tells you whether retrieval or generation is the thing that's broken.