Skip to content
AI Engineer Guide.
midReportedcurated

Full fine-tuning or LoRA? Walk me through how you decide, including the memory numbers.

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

LoRA plateaus below target. How do you tell underfitting from a data problem?
  • Sweep rank and target modules on a fixed data subset and watch held-out accuracy; if quality rises with capacity, the adapter was underfitting.
  • If held-out accuracy is flat while training loss falls, the problem is data or labels, not capacity.
  • Compare against a full fine-tuning baseline once, because a low-rank constraint that binds on this task will not be fixed by more epochs.
Your LoRA matches full FT on the task. Any reason to still prefer full FT?
  • Matching on the target task does not prove the broad behaviour survives, so run the base model's regression suite before declaring equivalence.
  • Full fine-tuning is justified when you need a large distribution shift or continued pretraining, which a low-rank update cannot express.
  • It also removes adapter routing and version pinning from serving, at the cost of a full model per variant and roughly 112 GB of training memory.

Red flags

  • Says LoRA is always equivalent to full fine-tuning
  • Claims LoRA reduces training compute proportionally to its parameter savings

Concepts this tests

Tags

Source: github.com

More in Fine-tuning & adaptation