Why it matters
Teams reach for fine-tuning because it sounds like the thorough option, then spend weeks teaching a model a format that a schema and two examples would have enforced. A decision tree makes the cheapest sufficient intervention the default. It asks what the model is failing at, because knowledge gaps, output-shape gaps and behaviour gaps have different remedies, and only some of them are training problems at all.
Mental model
Choosing an adaptation is triage in an emergency room: treat the condition that is actually dangerous, not the one that is loudest. The analogy breaks because a model can be treated twice; you can add retrieval and still tune later, and the order matters less than the cost of undoing a bad tune.
How it works
Step 1 of 7
Classify the failure
Write three examples and label each as missing knowledge, wrong shape, wrong tone or weak reasoning. The label is what makes a rung applicable; without it, any rung looks plausible.
question · current
Failure, written with 3 examples
Start from the observed failure, not from the tool. Three examples are enough to classify it, and the class chooses the first rung that can reach it.
Root of the hierarchy
Also in focus
- Rung 1 · Prompt + schema
- Rung 2 · Retrieval
- Rung 3 · Adapter (LoRA)
- Rung 4 · Full fine-tune
Figure. Four rungs in cost order, each with the condition that makes it the right first attempt and the stop that ends the descent. A fifth branch closes the ladder with train-from-scratch, outside the page's four rungs; prompt and schema come first in every case.
What to notice. The edge label is the decision: escalation is justified by the failure class, not by which team owns accelerators. Every rung is a stop point, so the cheapest sufficient fix is the default and each step up buys capacity with data, compute and maintenance. Train from scratch is drawn only to close the ladder — it sits outside the page's four rungs, and nothing here recommends it. Costs are relative, not measured.
Text version
A cost ladder drawn as a decision tree. The root is a failure written with three examples, labelled as missing knowledge, wrong shape, wrong tone or weak reasoning. Four rungs hang from it in cost order. Rung 1, prompt and schema, applies first in every case, with a stop if the twenty failures now pass. Rung 2, retrieval, applies when knowledge changes or must be cited, with a stop when grounding closes the gap. Rung 3, an adapter, applies when the behaviour is stable and survives prompting, with a stop when the held-out set and general slice hold. Rung 4, full fine-tuning, applies to a new tokenizer, a new modality or a large shift, with a stop only when the cheaper rung fails. A fifth branch, train from scratch, covers the case where no base model hosts the behaviour and is labelled as outside the tree. A final branch records the decision and its reversal plan beside the model version.
- Write down the observed failure with three examples, and label each as missing knowledge, wrong shape, wrong tone or weak reasoning.
- Try the prompt first, because a schema, two demonstrations and an explicit refusal rule fix most shape and tone complaints without training.
- If the gap is knowledge that changes or must be cited, add retrieval and measure whether grounding closes it.
- If the behaviour is stable and survives prompting, compare a small adapter against a hosted fine-tune on the same held-out set.
- Escalate to full fine-tuning only when the adapter cannot represent the change, for example a new tokenizer, a new modality or a large distribution shift.
- Record the decision and the evidence beside the model version so the next team does not relitigate it.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Prompt and schema only | Fast, reversible, no data pipeline | Ceiling on style and format consistency | The task needs behaviour the base model never shows |
| Retrieval | Fresh, citable knowledge without training | Retrieval latency and index maintenance | The failure is tone or format rather than facts |
| Adapter tuning | Cheap iteration, many variants per base | Data curation and an eval harness | The change needs new vocabulary or a new modality |
| Full fine-tuning | Maximum capacity for distribution shift | Cost, checkpoints and regression risk | A cheaper method already passes the held-out set |
Good for
Fast, reversible, no data pipeline
Cost
Ceiling on style and format consistency
Fails when
The task needs behaviour the base model never shows
- The task needs behaviour the base model never shows
- The failure is tone or format rather than facts
- The change needs new vocabulary or a new modality
- A cheaper method already passes the held-out set
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Prompt and schema only is good for Fast, reversible, no data pipeline, costs Ceiling on style and format consistency, and fails when The task needs behaviour the base model never shows. Retrieval is good for Fresh, citable knowledge without training, costs Retrieval latency and index maintenance, and fails when The failure is tone or format rather than facts. Adapter tuning is good for Cheap iteration, many variants per base, costs Data curation and an eval harness, and fails when The change needs new vocabulary or a new modality. Full fine-tuning is good for Maximum capacity for distribution shift, costs Cost, checkpoints and regression risk, and fails when A cheaper method already passes the held-out set
Failure modes
- Skipping the cheap rung. A prompt fix was available but a training run was funded first. Debug: run the prompt and retrieval baseline on the same twenty failures before approving any training budget.
- Decision by org chart. The team that owns accelerators gets the work regardless of the failure type. Debug: attach a failure label to each request and check whether the chosen remedy matches it.
- No reversal plan. The tuned model regresses on general traffic and there is no path back to the base model. Debug: keep the base model and prompt version deployable and compare on a general slice before cutover.
In production
Keep the decision record next to the deployed model and revisit it when the base model changes, because an upgrade can make an adapter redundant or harmful. Monitor a general-traffic slice, not only the target task.
Common misconceptions
People think: Fine-tuning is the serious option and prompting is the shortcut.
Actually: Prompting, schemas and retrieval are engineering interventions with their own evaluation; training is the expensive escalation, not the mature default.
People think: If fine-tuning works, it replaces the prompt.
Actually: A tuned model still runs inside a prompt, and the system message, tools and refusal rules usually stay.
Interview angle
Walk the tree out loud: name the failure class first, then the cheapest remedy that could fix it, then the evidence that would justify the next rung. Strong candidates mention what makes an adaptation reversible. Weak answers jump to a training recipe and never say which failure it addresses or how they would know it worked.
Sources
- Ouyang et al., Training language models to follow instructions with human feedback — instruction tuning and preference optimisation as the expensive end of the tree.
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — retrieval as the alternative for knowledge failures.
- Hugging Face, PEFT documentation — the adapter methods the middle rung relies on.