Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 42 of 57

RAG vs fine-tuning

0 of 57 complete

Why it matters

Teams reach for fine-tuning when answers are wrong and for RAG when answers are stale, but the two failures have different causes. Wrong answers can come from missing knowledge, which retrieval fixes, or from bad format and refusal behaviour, which training fixes. Choosing the wrong lever produces an expensive change that does not move the metric, and often destroys the instruction-following behaviour the product depended on.

Mental model

RAG is an open-book exam; fine-tuning is practising until the method is automatic. The metaphor breaks because the open book can be swapped between questions, while the practised method is baked into the weights and cannot be unlearned for one query.

How it works

Mechanism, step by stepInteractive
  1. 1Classify the failure: missing fact, stale fact, wrong format, wrong tone, or wrong procedure.
  2. 2Route missing and stale facts to retrieval, because the corpus can be updated without touching weights.
  3. 3Route format, tone and tool-use procedure to fine-tuning, with examples that cover the edge cases.
  4. 4Check whether the base model already performs the behaviour with a better prompt before training anything.
  5. 5If both are needed, fine-tune for behaviour and keep retrieval as the fact source, then evaluate them together.
  6. 6Measure the update path: how long to change a fact with retrieval versus retraining and redeploying.

Step 1 of 6

Classify the failure: missing fact, stale fact, wrong format, wrong tone, or wrong procedure.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Classify the failure: missing fact, stale fact, wrong format, wrong tone, or wrong proced… 2. Route missing and stale facts to retrieval, because the corpus can be updated without tou… 3. Route format, tone and tool-use procedure to fine-tuning, with examples that cover the ed… 4. Check whether the base model already performs the behaviour with a better prompt before t… 5. If both are needed, fine-tune for behaviour and keep retrieval as the fact source, then e… 6. Measure the update path: how long to change a fact with retrieval versus retraining and r…

  1. Classify the failure: missing fact, stale fact, wrong format, wrong tone, or wrong procedure.
  2. Route missing and stale facts to retrieval, because the corpus can be updated without touching weights.
  3. Route format, tone and tool-use procedure to fine-tuning, with examples that cover the edge cases.
  4. Check whether the base model already performs the behaviour with a better prompt before training anything.
  5. If both are needed, fine-tune for behaviour and keep retrieval as the fact source, then evaluate them together.
  6. Measure the update path: how long to change a fact with retrieval versus retraining and redeploying.

Trade-offs

ChoiceGood forCostFails when
RAG onlyFresh facts and citationsRetrieval latency and context costThe model ignores instructions or formats poorly
Fine-tuning onlyConsistent behaviour and lower prompt costStale knowledge and no citationsFacts change often or must be attributable
BothBehaviour plus current factsTwo systems to build and evaluateThe team cannot maintain either well
Prompt engineering firstCheapest and fastest to tryLimited gains on knowledge gapsThe corpus is not indexed at all
Trade-off mapInteractive

Good for

Fresh facts and citations

Cost

Retrieval latency and context cost

Fails when

The model ignores instructions or formats poorly

  1. The model ignores instructions or formats poorly
  2. Facts change often or must be attributable
  3. The team cannot maintain either well
  4. The corpus is not indexed at all

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: RAG only is good for Fresh facts and citations, costs Retrieval latency and context cost, and fails when The model ignores instructions or formats poorly. Fine-tuning only is good for Consistent behaviour and lower prompt cost, costs Stale knowledge and no citations, and fails when Facts change often or must be attributable. Both is good for Behaviour plus current facts, costs Two systems to build and evaluate, and fails when The team cannot maintain either well. Prompt engineering first is good for Cheapest and fastest to try, costs Limited gains on knowledge gaps, and fails when The corpus is not indexed at all

Failure modes

  • Fine-tuning to memorise facts. The model recites training documents, including outdated versions, and cannot cite them. Debug: ask about a fact changed after training and check whether it answers with the old value.
  • RAG for a behaviour problem. Retrieval is added, answers remain badly formatted, and the index is blamed. Debug: hold retrieval fixed and test whether a better prompt or a few demonstrations fixes the format.
  • Forgetting after fine-tuning. Instruction following degrades and the model stops using retrieved context. Debug: run the pre- and post-training evaluation on grounded answering, not only on the target task.

In production

Track the cost of a knowledge update end to end for each approach. If a fact change requires a training run and a deploy, the system has a freshness problem that retrieval should own.

Common misconceptions

People think: Fine-tuning is how you add your company's knowledge.

Actually: Training teaches behaviour and style; facts that change need an external, updatable store.

People think: RAG and fine-tuning are alternatives.

Actually: They address different failures and are usually combined, with training for conduct and retrieval for content.

Interview angle

Start from the failure classification and name the update path as the deciding constraint. Mention the risk of degrading instruction following and the need to evaluate both together. Picking one technique and justifying it on cost alone is the answer that skips the problem.

Sources