Fine-tuning vs RAG: how to choose (and when to use both)

The short answer

Use retrieval-augmented generation (RAG) when the model needs knowledge it doesn’t have: facts that change, documents that are private, or citations a user must be able to check. Use fine-tuning when the model needs skill it doesn’t have: a consistent output format, a domain’s way of reasoning, a procedure, a tone, or the judgment to choose between plausible answers.

Most production systems eventually use both. A fine-tuned model that is good at reading retrieved context, extracting what matters and answering in your format beats either technique alone. But the order matters, and the first step is almost always to figure out which kind of failure you are looking at.

What RAG actually fixes

RAG adds a search step before generation. A query is embedded, matched against an index of your documents, and the top results are inserted into the prompt. The model then answers using that context. The strengths are real:

• Freshness. Update the index and the model “knows” the new policy in minutes, without retraining.
• Attribution. Answers can cite their sources, which matters in legal, medical and regulated settings.
• Access control. Retrieval can respect per-user permissions, so the model only sees documents the user is allowed to see.
• Low upfront cost. A basic pipeline can be built in days with off-the-shelf components.

RAG’s weaknesses show up at scale. Retrieval quality caps answer quality: if the right passage isn’t in the top results, nothing downstream can recover it. Long contexts are expensive and slow, and every request pays for them. And retrieval does nothing for tasks where the bottleneck is not knowledge but behavior — the model still has to know what to do with the context.

What fine-tuning actually fixes

Fine-tuning changes the model’s weights by training it on examples of the behavior you want. It is how you teach a model to:

• produce a strict schema every time, rather than most of the time;
• use a domain’s vocabulary and conventions correctly — ICD-10 codes, legal clause types, tariff classifications;
• follow a multi-step procedure the way your best people do it;
• exercise judgment — what to flag, when to escalate, which of two valid answers is more useful (via preference tuning);
• do all of this in a smaller, cheaper, faster model than the one you prototyped with.

What fine-tuning does not do well is inject large volumes of fast-changing facts. You can teach a model your product catalog, but you will have to retrain every time it changes, and the model may still confuse similar items. That is retrieval’s job.

Side-by-side comparison

Note the cost asymmetry. RAG is cheap to start and expensive to run; fine-tuning is the reverse. A use case serving a few thousand requests a day may never justify fine-tuning on cost alone. One serving millions will almost always benefit, even if RAG stays in the loop. Our cost guide works through the break-even math.

A five-question decision framework

We run every new use case through five questions before recommending an approach.

1. Does the model fail when the answer is in the prompt?

Take 50 failing examples and paste the correct source material directly into the prompt. If accuracy jumps, your problem is retrieval: improve the index, chunking and ranking. If the model still fails with perfect context, it lacks the skill, and fine-tuning is on the table.

2. How fast does the underlying knowledge change?

Prices, policies, inventory and news change daily — retrieve them. Coding conventions, clinical guidelines and legal reasoning change slowly — they can live in weights.

3. What is the request volume?

Above roughly a million requests a month, the per-request savings of a smaller fine-tuned model start to dominate the upfront cost. Below a hundred thousand, optimize prompts and retrieval first.

4. Is the output format strict?

If downstream systems parse the output, every malformed response is an incident. Fine-tuning drives format errors close to zero in a way prompting rarely does.

5. Do you have, or can you get, labelled examples?

Fine-tuning needs a few thousand high-quality demonstrations or preference judgments. Many companies already have them in historical decisions and corrections; others need a short annotation program.

Using both: retrieval-aware fine-tuning

The strongest pattern we deploy is a fine-tuned model trained on retrieval outputs. During data preparation we run the production retriever over each training example and include its results — including the irrelevant ones — in the training input. The model learns to find the useful passage, ignore distractors, cite correctly and say “not found” when retrieval comes up empty.

This approach, sometimes called retrieval-augmented fine-tuning, typically adds 5–15 points of accuracy over plain RAG with the same retriever, while letting you use a smaller and cheaper model. It keeps all of RAG’s strengths — freshness, attribution, access control — and removes most of its brittleness.

Retrieval decides what the model sees. Fine-tuning decides what the model does with it.

What we see by industry

The right mix of retrieval and fine-tuning follows the shape of the work, and the shape of the work tends to cluster by industry.

Financial services

Policies, product terms and regulations change constantly, so retrieval is non-negotiable for anything customer-facing. But the core analytical tasks — adverse-media screening, KYC document review, transaction narrative classification — are stable procedures performed millions of times. Banks typically end up with fine-tuned models for the procedures and retrieval for the reference material, often inside the same pipeline.

Insurance

Claims intake and underwriting submissions are dominated by extraction from messy documents: scanned forms, broker emails, loss runs. This is fine-tuning territory. Retrieval enters when the model must check a claim against the specific policy wording, which differs for every insured.

Healthcare

Clinical coding, prior-authorization review and note summarization are high-volume, schema-heavy and privacy-constrained — ideal for small fine-tuned models running inside the provider’s environment. Clinical guidelines and formularies, which change, belong in retrieval.

Legal

Contract review mixes both: the clause taxonomy and risk judgment are learned behavior, while the firm’s playbooks and precedent language are best retrieved, because partners revise them.

Customer support

Product documentation changes weekly, so knowledge lives in retrieval. What fine-tuning adds is the behavior: correct tool calls, consistent tone, and — most valuable of all — knowing when to hand off to a human.

A two-week evaluation plan

If you are unsure which way to go, this is the plan we run in a feasibility sprint:

1. Days 1–3: assemble 500–1,000 representative inputs with expert-verified answers. Freeze them as the evaluation set.
2. Days 3–5: score the current system and the strongest prompted frontier baseline. Run the “answer in the prompt” test on failures.
3. Days 5–8: improve retrieval (chunking, hybrid search, re-ranking) and measure the gain.
4. Days 8–12: fine-tune a small model on a few thousand examples, with and without retrieved context in training.
5. Days 12–14: compare accuracy, cost per thousand requests and latency across all variants and write the recommendation.

Two weeks is enough to replace opinions with numbers. In roughly half the sprints we run, the answer is “both”; in about a fifth, better retrieval alone closes the gap.

Common mistakes

• Fine-tuning to add facts. Teaching a model your knowledge base via fine-tuning produces confident hallucinations about the parts it half-learned. Retrieve facts; fine-tune behavior.
• Endless prompt engineering. If you have 4,000-token prompts full of exceptions and examples, you are doing fine-tuning by hand, expensively, on every request.
• No evaluation set. Without a frozen, adjudicated test set you cannot tell whether a change to retrieval or training helped. Build it first. See how to evaluate a fine-tuned LLM.
• Training without production context. A model fine-tuned on clean inputs falls over on real retrieval results. Train on what it will actually see.