Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 6 of 22

PEFT

0 of 22 complete

Why it matters

Full fine-tuning needs gradients and optimiser state for every weight, plus a full checkpoint per task. That cost is paid per variant, which makes per-customer or per-task models impractical. PEFT changes the accounting: freeze the base, train a small structure, and store a fraction of a model per adaptation. The engineering question shifts from can we afford to train to where the update should live and how it will be served.

Mental model

PEFT is a set of sticky notes on a reference book: the book stays the same and each reader adds a thin layer of annotations. The analogy breaks at the spine, because some methods insert new layers into the forward pass, which changes latency and prevents a clean merge.

How it works

A frozen base with a trainable low-rank branchInteractive
signalbypassfrozentrainable

Step 1 of 5

Freeze the base

The pretrained matrix keeps its weights and receives no gradient. Nothing in the optimiser refers to it, which is where the memory saving starts.

Active layers

  • Input vector x
  • Frozen base
  • W · original projection

Hover a layer to read what it holds.

Next: Project down

Trainable
2·d·r = 65,536
Frozen
d² = 16.8M
Ratio
0.39% · illustrative
Step 1/5 · Freeze the base

Figure. A low-rank adapter beside a frozen projection. The base weight W is not updated; A projects d_model down to rank r and B projects back, and the two paths are summed. B is zero-initialised, so the adapter starts as a no-op.

What to notice. This figure shows the low-rank variant of PEFT; bottleneck, prefix and selective methods place their parameters elsewhere. Only A and B receive gradients, so optimiser state and checkpoint size scale with 2·d·r per adapted matrix instead of d². At d_model = 4096 and r = 8 that is 65,536 trainable parameters against 16.8M frozen ones. Merging removes the adapter's serving latency but also removes the ability to switch it off.

Text version

A vertical stack. The input x enters a frozen original projection in a sunken dashed band, and the same input branches along the right into a trainable low-rank adapter: A projects 4096 features down to rank 8, B projects back to 4096. The frozen W x path and the adapter B A x path meet at a sum, which produces the output. Five steps cover freezing, the down-projection, the zero-initialised up-projection, the sum, and the parameter accounting.

  1. Decide the binding constraint first: training memory, checkpoint size, inference latency, or the number of variants you must serve.
  2. Pick the insertion point that matches it, such as low-rank updates on projections, bottleneck layers, or prefix vectors.
  3. Freeze the base explicitly and assert that only the intended parameters require gradients before the first step.
  4. Match the training objective to the full fine-tuning recipe, since PEFT changes capacity rather than the loss.
  5. Compare against a full fine-tuning baseline on the same data at least once, to learn whether the low-rank constraint binds.
  6. Record the merge and serving plan with the adapter, because the same weights behave differently merged and unmerged.

Trade-offs

ChoiceGood forCostFails when
Low-rank updatesCheap storage and near-zero serving cost when mergedRank and target choice need tuningThe task needs a large behavioural shift
Bottleneck adaptersStable training, composable layersAdded latency unless fusedThe latency budget is tight
Prompt or prefix tuningVery few parametersConsumes context and can be brittlePrompts are long or the task is complex
Selective trainingSimple to reason aboutSparse gains and awkward checkpointsYou need many variants per base
Trade-off mapInteractive

Good for

Cheap storage and near-zero serving cost when merged

Cost

Rank and target choice need tuning

Fails when

The task needs a large behavioural shift

  1. The task needs a large behavioural shift
  2. The latency budget is tight
  3. Prompts are long or the task is complex
  4. You need many variants per base

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Low-rank updates is good for Cheap storage and near-zero serving cost when merged, costs Rank and target choice need tuning, and fails when The task needs a large behavioural shift. Bottleneck adapters is good for Stable training, composable layers, costs Added latency unless fused, and fails when The latency budget is tight. Prompt or prefix tuning is good for Very few parameters, costs Consumes context and can be brittle, and fails when Prompts are long or the task is complex. Selective training is good for Simple to reason about, costs Sparse gains and awkward checkpoints, and fails when You need many variants per base

Failure modes

  • Frozen base not actually frozen. Gradients reach the base weights and memory looks like full fine-tuning. Debug: count parameters with requires_grad and print the optimiser param groups before step one.
  • Adapter applied to the wrong modules. Training converges slowly and quality plateaus below the baseline. Debug: list adapted module names and compare them against a reference implementation for the architecture.
  • Merge and serve mismatch. Offline scores are good and the merged model behaves differently. Debug: evaluate the merged artefact, not the adapter, with the exact serving configuration.

In production

Store each adapter with the base model identifier and revision it was trained against, since an adapter is not portable across base checkpoints. Monitor per-adapter error rates when many share one deployment.

Common misconceptions

People think: PEFT means lower quality by definition.

Actually: On many behaviour and style tasks a low-rank update matches full fine-tuning; the gap appears when the task needs a large shift.

People think: Adapters are always free at inference.

Actually: Unmerged adapters add compute and memory; only merging or fusing removes the overhead.

Interview angle

Separate the memory accounting from the quality question: PEFT saves optimiser state and checkpoint size, and its quality depends on where the update lives and how much capacity the task needs. Name one case where it loses. Weak answers recite that a method is parameter-efficient without saying what it does not save.

Sources