Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 2 of 22

SFT

0 of 22 complete

Why fine-tune at all

Prompting can specify a format. It cannot always hold it. When an application needs the same structure across thousands of calls — a fixed JSON envelope, a house citation style, a proprietary query dialect, a specific tool-call sequence — a long instruction block is a recurring token cost and a recurring opportunity to drift. SFT moves that specification from the prompt into the weights.

There are three honest reasons to run it. Behaviour shaping: the model adopts a format, register or procedure reliably. Prompt compression: a demonstration set that would cost thousands of tokens per request becomes implicit. Distillation: a smaller, cheaper model learns to approximate a larger model or a hand-built pipeline on a narrow distribution.

There is one dishonest reason: "we want the model to know our data". Supervised fine-tuning can memorise, but it does so unevenly, cannot cite, and cannot be updated without retraining. Retrieval is the better tool for facts; SFT is the better tool for form. If a fact changes weekly, it does not belong in weights.

Mental model

An apprenticeship built entirely from worked examples. The apprentice sees a request and a finished answer, again and again, and learns to produce answers that look like the ones in the book.

The analogy breaks in two places. First, an apprentice can ask why; the loss function cannot. It rewards surface agreement, so a demonstration that is confidently wrong teaches confident wrongness at the same rate as a correct one. Second, an apprentice generalises from principles; a fine-tuned model generalises from the distribution it saw. Show it short answers and it will be terse on long questions too.

How it works

Pairs in, answer tokens out, one updateInteractive
signalbypassloop

Step 1 of 6

One pair

A real prompt and the answer a careful person would give. SFT moves form, not facts, so the pair should demonstrate the behaviour rather than carry knowledge.

Active layers

  • Demonstration pair

Hover a layer to read what it holds.

Next: Format and tokenise

Loss on
answer tokens only
Template
must match serving
Checkpoint
best, not last
Step 1/6 · One pair

Figure. The training stack for one supervised step: data is formatted, the model predicts every position, the mask selects answer positions, and only those contribute to cross-entropy. The dashed lane is the label path; the dash-dot lane is the next batch. Truncation, padding and distributed data loading are omitted, and the learning rate is illustrative.

What to notice. The instruction is in the forward pass but not in the loss: that is the claim the mask band makes. Every answer token counts equally, so a long answer dominates the gradient unless length is weighted or capped. The template must be the serving template; a mismatch here is invisible in this drawing and in the training loss, and shows up only against production traffic.

Text version

A vertical stack. A prompt and answer pair enters a batch construction group holding a chat template and tokenisation. The ids go to a forward pass; its logits and the ids meet at a select-answer-tokens band that masks the instruction, which feeds cross-entropy, then a backward and optimiser step, then a checkpoint. A dashed lane carries the labels from tokenisation to the selection band, and a dash-dot lane returns from the update to batch construction. Six steps cover the pair, formatting, the forward pass, selection, the loss, and the update.

  1. Define the behaviour and its test. Write the target output contract and a held-out evaluation set before collecting data. If you cannot measure the behaviour, you cannot tell learning from imitation.
  2. Collect demonstrations. Real production prompts with human-corrected answers are the strongest source; synthetic generation from a stronger model is faster but inherits its errors. Deduplicate near-identical examples, balance categories, and strip personal data.
  3. Format exactly as you will serve. Apply the model's chat template, including special tokens and role markers. A mismatch between training and inference formatting is the single most common silent failure.
  4. Mask the prompt. Compute loss only on answer tokens; otherwise the model spends capacity learning to reproduce your instructions. Some setups also mask tool-call scaffolding or system text.
  5. Choose modest hyperparameters. Fine-tuning starts from a competent model, so it needs a small learning rate and few epochs. More epochs fit the training set and degrade everything else.
  6. Train with checkpointing and a fixed seed. Save intermediate checkpoints; the best checkpoint is frequently not the last one.
  7. Evaluate on held-out tasks plus regressions. Compare against the base model with your best prompt, on both the target behaviour and a general capability sample.
  8. Package and pin. Record base model revision, dataset hash, hyperparameters and tokenizer. A fine-tune without provenance cannot be reproduced or rolled back.

The objective is ordinary cross-entropy over the answer tokens:

L=−1N∑t=1Nlog⁡pθ(yt∣x,y<t)\mathcal{L} = -\frac{1}{N}\sum_{t=1}^{N} \log p_\theta(y_t \mid x, y_{<t})

where xx is the prompt, yy the demonstrated answer, and NN the number of unmasked answer tokens. Note what this implies: every token contributes equally, so long answers dominate the gradient unless you weight or truncate. Truncation is worse than it looks — it teaches the model to stop mid-sentence.

A minimal loop makes the masking explicit:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
 
model_id = "your-base-model"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
 
def loss_for(prompt: str, answer: str) -> torch.Tensor:
    # Apply the same chat template used at inference time.
    full_messages = [
        {"role": "user", "content": prompt},
        {"role": "assistant", "content": answer},
    ]
    full_text = tok.apply_chat_template(full_messages, tokenize=False)
    prompt_text = tok.apply_chat_template(
        full_messages[:-1], tokenize=False, add_generation_prompt=True
    )
 
    full_ids = tok(full_text, return_tensors="pt").input_ids
    prompt_ids = tok(prompt_text, return_tensors="pt").input_ids
 
    labels = full_ids.clone()
    labels[:, : prompt_ids.shape[1]] = -100   # ignore prompt tokens
    return model(input_ids=full_ids, labels=labels).loss
 
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)
for prompt, answer in demonstrations:
    loss = loss_for(prompt, answer)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Production training pads and batches examples, masks padding, and usually trains adapters rather than full weights — see LoRA. The example is deliberately small: the mechanics are the masking and the loss, not the framework.

Trade-offs

ApproachGood forCostFails when
Prompt engineeringFast iteration, changing requirementsTokens per request, format driftThe contract is long or must hold under adversarial input
RetrievalFacts that changeIndexing, latency, context budgetThe gap is behaviour, not knowledge
SFTFormat, tone, procedure, distillationData curation, training run, eval burdenYou need new facts or preference trade-offs
Preference optimisationRanking between acceptable answersPreference data, more complex trainingYou have no reliable comparison signal
Trade-off mapInteractive

Good for

Fast iteration, changing requirements

Cost

Tokens per request, format drift

Fails when

The contract is long or must hold under adversarial input

  1. The contract is long or must hold under adversarial input
  2. The gap is behaviour, not knowledge
  3. You need new facts or preference trade-offs
  4. You have no reliable comparison signal

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Prompt engineering is good for Fast iteration, changing requirements, costs Tokens per request, format drift, and fails when The contract is long or must hold under adversarial input. Retrieval is good for Facts that change, costs Indexing, latency, context budget, and fails when The gap is behaviour, not knowledge. SFT is good for Format, tone, procedure, distillation, costs Data curation, training run, eval burden, and fails when You need new facts or preference trade-offs. Preference optimisation is good for Ranking between acceptable answers, costs Preference data, more complex training, and fails when You have no reliable comparison signal

Failure modes

  • Format learned, task not. Training loss falls, held-out accuracy does not. Debug by evaluating on paraphrased inputs that keep the task but change surface wording; if accuracy collapses, the model learned a template.
  • Train/serve format mismatch. The model performs in the notebook and misbehaves behind the API. Debug by logging the exact rendered prompt at inference and diffing it against a training example character by character.
  • Label noise amplified. A minority of demonstrations contain wrong answers, and the model reproduces the error confidently. Debug by sampling 50 training targets and auditing them; then check whether failing outputs match a suspicious cluster in the data.
  • Catastrophic forgetting. The fine-tune improves the target task and degrades general instruction following. Debug by running a general capability suite before and after; keep the fine-tune narrow and mix in a small amount of general instruction data if it regresses.

In production

Treat a fine-tune as a versioned artifact with a serving contract: base model, adapter or weights, tokenizer, chat template, decoding defaults. Changing any of them is a new release. Roll out behind a shadow or canary comparison against the prompted baseline, and keep the baseline runnable — you will need it when a regression appears.

Measure three things continuously: task success on a fixed golden set, format validity rate at the parser, and a general capability sample to catch forgetting. Then watch cost from both sides: training is a one-off, but inference may be cheaper because prompts got shorter, or more expensive because you now host weights.

The data flywheel is the real operational asset. Log production failures, have humans correct a sample, and add corrections to the next dataset version. Without that loop, the first fine-tune is also the last useful one.

Common misconceptions

People think: SFT is how you teach a model your company's knowledge.

Actually: It can memorise fragments, but retrieval is more accurate, auditable and updatable. Use SFT for how the model should respond, and retrieval for what it should know.

People think: More epochs will make it learn better.

Actually: Past a small number of epochs, loss keeps falling while held-out quality does not. The model is fitting your examples, including their mistakes.

People think: SFT aligns the model with what users prefer.

Actually: SFT imitates demonstrations. It has no notion of preference between two acceptable answers; that requires comparison data and a preference objective.

Interview angle

"Why fine-tune instead of prompting?" is the entry question; the follow-up is "what would you fine-tune on, and how would you know it worked?" Answer with a behaviour that prompting cannot hold reliably, a dataset built from real traffic with human corrections, an evaluation that separates format from task accuracy, and a regression suite for forgetting. Then say what you would not fine-tune: changing facts. See LoRA and the fine-tuning interview category.

Sources

  • LoRA — training the same objective with a small parameter budget.
  • RAG — the alternative when the gap is knowledge.
  • Structured outputs — the contract SFT is often used to enforce.
  • Prompt engineering — the baseline you must beat.