Why it matters
A pretrained model is a text continuer. Given a question it may answer, restate the question or produce a list, because nothing in pretraining specified that a request should be obeyed. Instruction tuning supplies that contract with many pairs of a natural-language request and the response a careful person would give. The engineering problem is coverage, since the model generalises across descriptions and fails on shapes it never saw.
Mental model
It is like teaching a new employee the house style by showing worked examples from several departments rather than training them for one job. The analogy breaks because the model does not infer intent; it pattern-matches phrasing, so an instruction written in an unfamiliar register can still slip through.
How it works
Step 1 of 6
Hold templates out
The unit of evaluation is a whole template, not a random example. Splitting examples lets near-identical wording sit on both sides and inflates the score.
Active layers
- Task description
Hover a layer to read what it holds.
Next: Write the format layers
- Supervised
- response tokens
- Held out
- whole templates
- Adds
- format, not facts
Figure. The format layers of one instruction example, in template order, and the single layer the loss selects. The system preamble, instruction and optional input are prompt; only the response is supervised. The dashed lane is the supervision signal, not a runtime path. Category balance and template diversity are named in the steps rather than drawn.
What to notice. The vertical order is the template's order, not the order the model computes: the response is assembled after the prompt but the forward pass runs once over the whole sequence. The mask is what turns instructions into context rather than targets. The figure does not show evaluation; the last step names held-out templates as the test, because a category win can hide a failure elsewhere.
Text version
A vertical stack. A task description enters a group for one example, which holds system preamble, instruction, optional input and response in template order. The assembled example feeds a chat template, then a forward pass, then a loss. A dashed lane carries the response directly to the loss labelled supervised; the loss feeds an update and an instruction follower. Six steps cover holding templates out, the prompt layers, the response, assembly and forward, the masked loss, and evaluation.
- Collect tasks across the categories you care about, including extraction, classification, rewriting, refusal and multi-step reasoning.
- Hold out whole task templates rather than random examples, so evaluation measures generalisation to unseen instructions.
- Write responses in the exact register and length you want at inference, since the model copies style more reliably than it follows a description of style.
- Balance categories by the count of distinct templates, or the largest category will dominate the loss.
- Compute loss on response tokens only and mask the instruction, so the model learns to answer rather than to reproduce requests.
- Evaluate on held-out templates and on a sample of the base model's traffic to catch regressions outside the target distribution.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Broad task mix | Generalises to unseen instructions | Annotation cost across many categories | You need one narrow, high-volume behaviour |
| Narrow task mix | High accuracy on the target task | Poor transfer and brittle phrasing | Users phrase requests in many ways |
| Model-written responses | Cheap and scalable | Inherits the writer model's errors | Correctness matters more than style |
| Human-written responses | Faithful style and judgement | Slow and expensive | Volume is the binding constraint |
Good for
Generalises to unseen instructions
Cost
Annotation cost across many categories
Fails when
You need one narrow, high-volume behaviour
- You need one narrow, high-volume behaviour
- Users phrase requests in many ways
- Correctness matters more than style
- Volume is the binding constraint
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Broad task mix is good for Generalises to unseen instructions, costs Annotation cost across many categories, and fails when You need one narrow, high-volume behaviour. Narrow task mix is good for High accuracy on the target task, costs Poor transfer and brittle phrasing, and fails when Users phrase requests in many ways. Model-written responses is good for Cheap and scalable, costs Inherits the writer model's errors, and fails when Correctness matters more than style. Human-written responses is good for Faithful style and judgement, costs Slow and expensive, and fails when Volume is the binding constraint
Failure modes
- Template leakage. Held-out examples reuse the wording of training instructions and scores look excellent. Debug: cluster instruction texts and check that no evaluation cluster appears in the training set.
- Answer-only training. The model learns to emit a plausible answer to any input, including empty or hostile ones. Debug: include refusal and clarification examples and score them as their own category.
- Style override. Responses are correct but verbose because the training answers were verbose. Debug: measure mean response length against the target and inspect the longest deciles.
In production
Version the task mix and re-run the held-out templates whenever it changes. Track refusal rate and response length on live traffic, since both drift before accuracy does.
Common misconceptions
People think: Instruction tuning adds knowledge.
Actually: It shapes how the model responds to requests; facts still come from pretraining or from context you supply.
People think: More examples per task is the lever.
Actually: Distinct task templates generalise; repeating one template mostly teaches that template.
Interview angle
The generalisation claim is the heart of it: the model learns to map descriptions to behaviour, so evaluation must hold out templates. Mention loss masking and category balance. Candidates who describe instruction tuning as teaching facts, or who evaluate with a random split of one template, have missed what the method does.
Sources
- Wei et al., Finetuned Language Models Are Zero-Shot Learners — instruction tuning across task clusters and zero-shot transfer.
- Chung et al., Scaling Instruction-Finetuned Language Models — task diversity and scale in instruction tuning.
- Zhou et al., LIMA: Less Is More for Alignment — a small curated set as an alternative to volume.