Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 20 of 22

When not to fine-tune

0 of 22 complete

Why it matters

Fine-tuning is attractive because it feels like a permanent fix. It is also the most expensive way to change behaviour: data collection, training runs, evaluation, a second artefact in production and a rollback plan. Many requests that arrive as fine-tuning are format problems, freshness problems or permission problems. Knowing when to refuse is a cost decision as much as a technical one.

Mental model

Fine-tuning to fix a knowledge gap is like repainting a sign because the address changed. The analogy breaks because repainting is harmless, while retraining on stale facts can make the model confidently wrong in a way that is harder to correct.

How it works

The refusal ladderInteractive
startquestionoutcomepath takenruled out
VerdictChoose· recommended hereRefuse· decline the request

Each edge carries the answer read at the fork; the side panel follows one rung at a time.

Step 1 of 6

Knowledge first

If the answer exists in a document you could retrieve, test retrieval before considering training. Repainting the sign does not change the address.

Condition · yes

Retrieval first

Choose

Fresh, auditable, reversible; costs retrieval latency and an index to operate.

Fails when the task needs a consistent style the corpus does not contain — that is the remaining case for training.

Path walked

  1. A request arrives as “fine-tune the model”
  2. Is the answer in a document you can retrieve?
  3. yesRetrieval first

Ruled out: Is it a format or policy rule code can enforce?, Prompt, schema or validator

Step 1/6 · Knowledge first

Figure. Five rungs: retrieval and a code-enforceable rule clear the cheaper remedies; the example count, the change cadence and the lifecycle cost then test whether a tune is supportable. Training survives only the last rung, where the life cycle cost fits — the page's stable-format, high-volume case, which no rung tests directly.

What to notice. The rungs are the page's order of refusal, not a universal policy. “A few hundred consistent examples” is the page's floor for a stable signal, not a measured threshold, and the lifecycle price is a judgement about your team. The figure does not price a training run; it shows where a refusal is cheaper than a checkpoint.

Text version

A decision ladder for refusing a fine-tune. A request arrives as “fine-tune the model”, then rung 1 asks whether the answer is in a document you can retrieve; yes leads to retrieval first, no continues. Rung 2 asks whether it is a format or policy rule code can enforce; yes leads to a prompt, schema or validator, no continues. Rung 3 asks whether you can assemble a few hundred consistent examples; no leads to do not tune yet, yes continues. Rung 4 asks whether the behaviour changes monthly; yes leads to keeping it in a prompt, no continues. Rung 5 asks whether the full lifecycle cost fits; no leads to recording the refusal with the evidence, yes leads to a fine-tune candidate. The three refusals carry the “Refuse” badge and the three positive outcomes carry “Choose”; the conditions sit on the edges, and the side panel walks one rung at a time.

  1. Ask whether the correct answer exists in a document you could retrieve, and if so test retrieval before considering training.
  2. Check whether the requirement is a format or policy rule that a schema, validator or system prompt can enforce.
  3. Count consistent examples; if you cannot assemble a few hundred, the signal is too thin for a stable update.
  4. Estimate the change cadence: if the behaviour changes monthly, a prompt is cheaper to revise than a checkpoint.
  5. Price the full lifecycle, including evaluation, serving, monitoring and rollback, not only the training run.
  6. Write the refusal down with the evidence, so the request can be revisited when the constraints change.

Trade-offs

ChoiceGood forCostFails when
Retrieval insteadFresh, auditable, reversibleLatency and index operationsThe task needs a consistent style the corpus lacks
Prompt and schemaImmediate and cheap to changeToken cost and a ceiling on reliabilityThe behaviour must hold across thousands of phrasings
Rules and validatorsDeterministic guaranteesBrittle on open-ended inputThe requirement is semantic
Fine-tune anywayConsistent behaviour at low latencyLifecycle cost and regression riskThe facts change or the data is thin
Trade-off mapInteractive

Good for

Fresh, auditable, reversible

Cost

Latency and index operations

Fails when

The task needs a consistent style the corpus lacks

  1. The task needs a consistent style the corpus lacks
  2. The behaviour must hold across thousands of phrasings
  3. The requirement is semantic
  4. The facts change or the data is thin

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Retrieval instead is good for Fresh, auditable, reversible, costs Latency and index operations, and fails when The task needs a consistent style the corpus lacks. Prompt and schema is good for Immediate and cheap to change, costs Token cost and a ceiling on reliability, and fails when The behaviour must hold across thousands of phrasings. Rules and validators is good for Deterministic guarantees, costs Brittle on open-ended input, and fails when The requirement is semantic. Fine-tune anyway is good for Consistent behaviour at low latency, costs Lifecycle cost and regression risk, and fails when The facts change or the data is thin

Failure modes

  • Training on stale facts. The model confidently states outdated information and retrieval cannot correct it. Debug: ask for facts you know changed and compare the answers with the source of truth.
  • Underspecified data. The tune memorises the few examples and fails on paraphrases. Debug: evaluate on human-written variations of the same requests.
  • Ownership gap. The model ships and nobody owns its evaluation, so it silently degrades. Debug: check that an owner and a recurring evaluation job exist before launch.

In production

If you ship anyway, cap the scope: one behaviour, one evaluation suite, one rollback path. Revisit the decision when the base model or the data source changes.

Common misconceptions

People think: Fine-tuning is how you add facts.

Actually: Weights are a poor store for facts that change; retrieval keeps them current and attributable.

People think: A small fine-tune is harmless to try.

Actually: It creates a model to maintain and a regression surface, which is a permanent cost for a temporary experiment.

Interview angle

Give the refusal criteria in order: knowledge, cadence, data volume, lifecycle cost. A strong answer names the case where fine-tuning is still correct, such as a stable format at high volume. Weak answers either refuse everything or treat training as the default and never mention retrieval.

Sources