Why it matters
Full fine-tuning needs gradients and optimiser state for every weight, plus a full checkpoint per task. That cost is paid per variant, which makes per-customer or per-task models impractical. PEFT changes the accounting: freeze the base, train a small structure, and store a fraction of a model per adaptation. The engineering question shifts from can we afford to train to where the update should live and how it will be served.
Mental model
PEFT is a set of sticky notes on a reference book: the book stays the same and each reader adds a thin layer of annotations. The analogy breaks at the spine, because some methods insert new layers into the forward pass, which changes latency and prevents a clean merge.
How it works
Step 1 of 5
Freeze the base
The pretrained matrix keeps its weights and receives no gradient. Nothing in the optimiser refers to it, which is where the memory saving starts.
Active layers
- Input vector x
- Frozen base
- W · original projection
Hover a layer to read what it holds.
Next: Project down
- Trainable
- 2·d·r = 65,536
- Frozen
- d² = 16.8M
- Ratio
- 0.39% · illustrative
Figure. A low-rank adapter beside a frozen projection. The base weight W is not updated; A projects d_model down to rank r and B projects back, and the two paths are summed. B is zero-initialised, so the adapter starts as a no-op.
What to notice. This figure shows the low-rank variant of PEFT; bottleneck, prefix and selective methods place their parameters elsewhere. Only A and B receive gradients, so optimiser state and checkpoint size scale with 2·d·r per adapted matrix instead of d². At d_model = 4096 and r = 8 that is 65,536 trainable parameters against 16.8M frozen ones. Merging removes the adapter's serving latency but also removes the ability to switch it off.
Text version
A vertical stack. The input x enters a frozen original projection in a sunken dashed band, and the same input branches along the right into a trainable low-rank adapter: A projects 4096 features down to rank 8, B projects back to 4096. The frozen W x path and the adapter B A x path meet at a sum, which produces the output. Five steps cover freezing, the down-projection, the zero-initialised up-projection, the sum, and the parameter accounting.
- Decide the binding constraint first: training memory, checkpoint size, inference latency, or the number of variants you must serve.
- Pick the insertion point that matches it, such as low-rank updates on projections, bottleneck layers, or prefix vectors.
- Freeze the base explicitly and assert that only the intended parameters require gradients before the first step.
- Match the training objective to the full fine-tuning recipe, since PEFT changes capacity rather than the loss.
- Compare against a full fine-tuning baseline on the same data at least once, to learn whether the low-rank constraint binds.
- Record the merge and serving plan with the adapter, because the same weights behave differently merged and unmerged.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Low-rank updates | Cheap storage and near-zero serving cost when merged | Rank and target choice need tuning | The task needs a large behavioural shift |
| Bottleneck adapters | Stable training, composable layers | Added latency unless fused | The latency budget is tight |
| Prompt or prefix tuning | Very few parameters | Consumes context and can be brittle | Prompts are long or the task is complex |
| Selective training | Simple to reason about | Sparse gains and awkward checkpoints | You need many variants per base |
Good for
Cheap storage and near-zero serving cost when merged
Cost
Rank and target choice need tuning
Fails when
The task needs a large behavioural shift
- The task needs a large behavioural shift
- The latency budget is tight
- Prompts are long or the task is complex
- You need many variants per base
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Low-rank updates is good for Cheap storage and near-zero serving cost when merged, costs Rank and target choice need tuning, and fails when The task needs a large behavioural shift. Bottleneck adapters is good for Stable training, composable layers, costs Added latency unless fused, and fails when The latency budget is tight. Prompt or prefix tuning is good for Very few parameters, costs Consumes context and can be brittle, and fails when Prompts are long or the task is complex. Selective training is good for Simple to reason about, costs Sparse gains and awkward checkpoints, and fails when You need many variants per base
Failure modes
- Frozen base not actually frozen. Gradients reach the base weights and memory looks like full fine-tuning. Debug: count parameters with requires_grad and print the optimiser param groups before step one.
- Adapter applied to the wrong modules. Training converges slowly and quality plateaus below the baseline. Debug: list adapted module names and compare them against a reference implementation for the architecture.
- Merge and serve mismatch. Offline scores are good and the merged model behaves differently. Debug: evaluate the merged artefact, not the adapter, with the exact serving configuration.
In production
Store each adapter with the base model identifier and revision it was trained against, since an adapter is not portable across base checkpoints. Monitor per-adapter error rates when many share one deployment.
Common misconceptions
People think: PEFT means lower quality by definition.
Actually: On many behaviour and style tasks a low-rank update matches full fine-tuning; the gap appears when the task needs a large shift.
People think: Adapters are always free at inference.
Actually: Unmerged adapters add compute and memory; only merging or fusing removes the overhead.
Interview angle
Separate the memory accounting from the quality question: PEFT saves optimiser state and checkpoint size, and its quality depends on where the update lives and how much capacity the task needs. Name one case where it loses. Weak answers recite that a method is parameter-efficient without saying what it does not save.
Sources
- Houlsby et al., Parameter-Efficient Transfer Learning for NLP — bottleneck adapters inserted into a frozen model.
- Hu et al., LoRA: Low-Rank Adaptation of Large Language Models — low-rank updates as an alternative insertion point.
- Hugging Face, PEFT documentation — the supported methods and their configuration.