Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 10 of 22

Full fine-tuning

0 of 22 complete

Why it matters

Some adaptations are not a nudge. A new language, a new modality, a vocabulary the tokenizer does not represent, or a task needing reasoning the base model never performed can exceed what a low-rank update expresses. Full fine-tuning removes the capacity constraint. The cost is that every parameter carries optimiser state during training and a full checkpoint afterwards, and regressions become global rather than local.

Mental model

Full fine-tuning is renovating the house rather than rearranging furniture: you can change the floor plan, and you also carry the risk of structural damage. The analogy breaks because a renovated house has one state, while a fine-tuned model can be shipped beside its predecessor and compared request by request.

How it works

Every weight trainable, priced per runInteractive
signalbypassloop

Step 1 of 6

Start from the base

The run starts from a pretrained checkpoint and places no frozen mask anywhere. Every parameter is in the optimiser's parameter groups.

Active layers

  • Pretrained checkpoint
  • All weights trainable

Hover a layer to read what it holds.

Next: Embeddings move

Weights · bf16
≈14 GB
Gradients · bf16
≈14 GB
AdamW · fp32
≈56 GB
Total · 7B
≈84 GB minimum
Step 1/6 · Start from the base

Figure. All parameters of the pretrained model receive gradients, so the optimiser carries state for the whole model and each run writes a complete checkpoint. The base stays available for rollback on the right-hand edge; the new checkpoint is a full copy, not a delta. The backward pass is summarised as one gradient edge into the optimiser and one update rail back to the group; per-layer paths are not drawn.

What to notice. Memory is dominated by optimiser state, not weights: for 7B parameters in mixed precision, bf16 weights and gradients are about 14 GB each while AdamW's two fp32 moments are about 56 GB. That is roughly 84 GB before activations and any fp32 master copy. All figures are illustrative. The capacity to change what the model knows is what a low-rank update cannot always match.

Text version

A vertical stack. A pretrained checkpoint feeds a group where embeddings, attention projections, feed-forward matrices and the output head are all marked trainable. The head's gradients feed an optimiser step, which writes a new full checkpoint. A right-hand loop labelled update all weights returns from the optimiser to the trainable group, and a bypass keeps the original checkpoint for rollback. Six steps cover the unfrozen start, embeddings, attention and feed-forward, the head, optimiser memory, and the full copy produced per run.

  1. Confirm the change exceeds what an adapter can express, using a PEFT baseline on the same data as the evidence.
  2. Decide which parameters to update: all of them, or all except the embedding and output layers when vocabulary must stay stable.
  3. Choose a distributed strategy that shards optimiser state, and check that memory per device leaves room for activations.
  4. Use a lower learning rate than pretraining and a short schedule, since the model already starts near a good solution.
  5. Mix in a small share of general instruction data to limit drift on unrelated behaviour.
  6. Evaluate on the target task and on a general slice, then keep the base model deployable for rollback.

Trade-offs

ChoiceGood forCostFails when
Full updateMaximum capacity for new behaviourOptimiser memory and full checkpointsA small adapter would have passed
Frozen embeddingsPreserves vocabulary and tokenizerLimits adaptation to new termsThe domain has genuinely new tokens
Mixed general dataLess regression on general trafficDilutes the target signalThe target task is very narrow
Sharded trainingFits larger modelsCommunication overhead and complexityThe model already fits on one node
Trade-off mapInteractive

Good for

Maximum capacity for new behaviour

Cost

Optimiser memory and full checkpoints

Fails when

A small adapter would have passed

  1. A small adapter would have passed
  2. The domain has genuinely new tokens
  3. The target task is very narrow
  4. The model already fits on one node

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full update is good for Maximum capacity for new behaviour, costs Optimiser memory and full checkpoints, and fails when A small adapter would have passed. Frozen embeddings is good for Preserves vocabulary and tokenizer, costs Limits adaptation to new terms, and fails when The domain has genuinely new tokens. Mixed general data is good for Less regression on general traffic, costs Dilutes the target signal, and fails when The target task is very narrow. Sharded training is good for Fits larger models, costs Communication overhead and complexity, and fails when The model already fits on one node

Failure modes

  • Optimiser memory blow-up. Training fails at step one with an out-of-memory error that looks like a batch-size problem. Debug: compute parameter, gradient and optimiser state bytes per device before choosing a batch size.
  • Global regression. Target scores rise and general traffic degrades. Debug: run a general instruction slice and a refusal set alongside the target evaluation.
  • Learning rate too high. Loss spikes and never recovers, leaving a model that answers in fragments. Debug: replay the first few hundred steps at a lower rate and compare loss curves.

In production

Keep the previous checkpoint deployable and canary the new one on a slice of live traffic. Monitor refusal rate, average length and off-topic rate, because a full update moves all three.

Common misconceptions

People think: Full fine-tuning always beats PEFT.

Actually: It has more capacity, not more signal; with a small dataset it often overfits where an adapter would generalise.

People think: You need a full update to add knowledge.

Actually: Retrieval or continued pretraining may be better routes, and a full update on instruction data mostly changes behaviour.

Interview angle

Justify the capacity argument and price it: optimiser state, checkpoints, rollback and regression surface. Say which baseline would convince you an adapter is insufficient. Weak answers treat full fine-tuning as the default serious option and never mention what it costs to keep several versions alive.

Sources