midReportedcurated
Full fine-tuning or LoRA? Walk me through how you decide, including the memory numbers.
30-second answer
- Default to LoRA: full fine-tuning a 7B model with Adam in mixed precision needs roughly 112 GB for weights, gradients and optimiser states before activations, while LoRA stores optimiser state only for the adapters.
- QLoRA goes further by keeping the frozen base in a low-bit format, which is how 7B fits on one 24 GB card, at the cost of slower steps and some numeric fidelity.
- Adapters are tens of megabytes, so you can version, A/B and serve many per base model, which is the real advantage when several teams or tenants need variants.
- Full fine-tuning earns its cost for large behavioural shifts, continued pretraining on billions of tokens, or when a lab owns the main post-training run.
- LoRA saves memory, not FLOPs, because the backward pass still flows through the frozen base, so a LoRA run is not proportionally cheaper in compute.
Likely follow-ups
Open a question to rehearse the answer.
LoRA plateaus below target. How do you tell underfitting from a data problem?
- Sweep rank and target modules on a fixed data subset and watch held-out accuracy; if quality rises with capacity, the adapter was underfitting.
- If held-out accuracy is flat while training loss falls, the problem is data or labels, not capacity.
- Compare against a full fine-tuning baseline once, because a low-rank constraint that binds on this task will not be fixed by more epochs.
Your LoRA matches full FT on the task. Any reason to still prefer full FT?
- Matching on the target task does not prove the broad behaviour survives, so run the base model's regression suite before declaring equivalence.
- Full fine-tuning is justified when you need a large distribution shift or continued pretraining, which a low-rank update cannot express.
- It also removes adapter routing and version pinning from serving, at the cost of a full model per variant and roughly 112 GB of training memory.
Red flags
- Says LoRA is always equivalent to full fine-tuning
- Claims LoRA reduces training compute proportionally to its parameter savings
Concepts this tests
Tags
Source: github.com