Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 15 of 22

Distillation

0 of 22 complete

Why it matters

A model that is good enough but too expensive to serve is a common end state. Distillation converts capability into a smaller artefact by training the student on the teacher's outputs, which are denser than labels: a distribution over tokens says which alternatives were nearly right. The engineering question is what to distil, since matching the teacher everywhere spends capacity on behaviour you do not need.

Mental model

Distillation is an apprentice copying a master's finished work rather than only the final grade: the corrections in the margin teach more than the answer. The analogy breaks because the apprentice also inherits the master's mistakes and cannot exceed what the master demonstrated.

How it works

Teacher targets, student weightsInteractive
signalbypasscrossloopfrozen

Step 1 of 5

Same prompts, two models

The prompt set is fixed first, because it defines what the student can learn. The teacher is frozen and its version recorded, since the student will reproduce its errors as well as its answers.

Active layers

  • Prompt set
  • Teacher

Hover a layer to read what it holds.

Next: Soften the targets

Transfers
behaviour on these prompts
Teacher
frozen, versioned
Ships
student only
Step 1/5 · Same prompts, two models

Figure. One distillation step. The prompt set reaches both models; the teacher is frozen and produces softened targets; the loss compares them with the student's logits and updates only the student. Batching, the temperature schedule and prompt filtering are omitted, and the loop is one update rather than a convergence curve.

What to notice. The dotted cross edge is the transfer: soft targets carry which alternatives were nearly right, which a hard label cannot. The dashed lane is the part of the signal that is not distillation — the same prompts with only labels, which is ordinary supervised fine-tuning. The teacher never enters the optimiser, so the saving is at inference, and the student is bounded by the teacher's behaviour on exactly these prompts.

Text version

A vertical stack. A prompt set feeds a frozen teacher, which produces softened targets. The same prompt set reaches the student along a dashed lane, and the student produces logits. The loss compares the student logits with the teacher's soft targets along a dotted cross lane, then an update band sends gradients back to the student along a dash-dot lane and writes the student checkpoint. Five steps cover the shared prompts, the softened targets, the loss, the update that reaches only the student, and what ships.

  1. Pick the teacher for the behaviour you need, and record its version, since the student will inherit its quirks.
  2. Decide the target: full distributions if you can access logits, sampled outputs if the teacher is only reachable through an API.
  3. Generate a prompt set that covers the deployment distribution rather than the teacher's training set.
  4. Train the student with a temperature-scaled soft-target loss plus a hard-label loss where labels exist.
  5. Evaluate the student against the teacher and against the task baseline, sliced by prompt type.
  6. Measure the served cost and latency of the student, because a smaller model with longer outputs may not be cheaper per request.

Trade-offs

ChoiceGood forCostFails when
Logit distillationDense signal, fast convergenceNeeds teacher internalsThe teacher is API-only
Output distillationWorks with any teacherLess signal per exampleYou need large volumes
Distil one behaviourEfficient use of student capacityNarrow coverageTraffic is broad
Distil everythingBroad fidelityCapacity spent on irrelevant behaviourThe student is much smaller
Trade-off mapInteractive

Good for

Dense signal, fast convergence

Cost

Needs teacher internals

Fails when

The teacher is API-only

  1. The teacher is API-only
  2. You need large volumes
  3. Traffic is broad
  4. The student is much smaller

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Logit distillation is good for Dense signal, fast convergence, costs Needs teacher internals, and fails when The teacher is API-only. Output distillation is good for Works with any teacher, costs Less signal per example, and fails when You need large volumes. Distil one behaviour is good for Efficient use of student capacity, costs Narrow coverage, and fails when Traffic is broad. Distil everything is good for Broad fidelity, costs Capacity spent on irrelevant behaviour, and fails when The student is much smaller

Failure modes

  • Teacher error inheritance. The student reproduces a systematic mistake with higher confidence. Debug: evaluate student and teacher on a labelled set and diff their errors by category.
  • Prompt set mismatch. The student scores well offline and fails on production inputs. Debug: compare the prompt distribution of the distillation set with a sample of real traffic.
  • Cost inversion. The student is cheaper per token but produces longer answers and costs more per request. Debug: measure cost per completed request, not cost per token.

In production

Canary the student against the teacher on live traffic and compare task success, latency and cost per request. Keep the teacher available for the slices where the student regresses.

Common misconceptions

People think: Distillation transfers knowledge the student did not have.

Actually: It transfers behaviour on the prompts you used; outside them the student knows no more than before.

People think: A distilled model is strictly worse.

Actually: It can match the teacher on the distilled distribution and often generalises differently, but it is not uniformly better.

Interview angle

Separate what is transferred, which is behaviour on a prompt distribution, from what is compressed, which is inference cost. Mention soft targets and the teacher's error inheritance. Weak answers claim distillation preserves capability generally and never ask which prompts the student was trained on.

Sources