Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 7 of 22

LoRA

0 of 22 complete

Why not update every weight

Full fine-tuning stores, for every parameter, a gradient and at least two optimiser states. For a model with billions of parameters that is several times the weight memory before activations, and it produces a complete second copy of the model per task. If you want ten customer-specific variants, you need ten models.

LoRA (Low-Rank Adaptation) starts from an observation about the update rather than the model: the change needed to adapt a large pretrained model to a downstream task tends to have low intrinsic rank. So instead of learning a full ΔW, learn two thin matrices whose product approximates it. The base weights stay frozen and shared; the adaptation is small, portable and composable.

The original paper reports, for GPT-3 175B fine-tuned with Adam, a reduction of trainable parameters by up to 10,000× and GPU memory by 3×, with quality on par or better than full fine-tuning on the tasks it evaluated, and no added inference latency once merged.

Mental model

A clip-on lens. The camera — the pretrained model — is unchanged; a thin piece of glass in front of it shifts what you see. Swap lenses for different tasks, keep one camera body for all of them.

The analogy breaks on where the correction applies and on capacity. A lens corrects the whole image at once; LoRA patches specific matrices inside the network, and its effect compounds through every layer it touches. And a lens cannot make a camera see infrared: if the base model has no representation for the target behaviour — a new language, a new modality, a genuinely different task — a low-rank nudge will not create one.

How it works

The adapter sits beside a frozen projection, not after itInteractive
signalbypassfrozentrainable

Step 1 of 5

Freeze the projection

The selected projection keeps its weights and receives no gradient. The forward pass still runs through it, which is why the base's behaviour is the starting point.

Active layers

  • Input x
  • One projection · in place
  • W · frozen matrix

Hover a layer to read what it holds.

Next: Branch the same input

Trainable
2·d·r = 131,072
Frozen
d² = 16.8M
Rank · α
r 16 · α 32 · illustrative
Step 1/5 · Freeze the projection

Figure. One selected projection, drawn in place: the frozen matrix W and the trainable pair A then B sit inside the same band, and their outputs meet at one sum. The input feeds both paths; B is zero-initialised so the branch starts at zero. Widths are illustrative: d = 4096 and rank 16. Which projections are targeted, and how the merge is deployed, are stated in the steps rather than drawn.

What to notice. The adapter is not applied after the block: it is summed beside W inside the same projection, which is why merging leaves the function unchanged, and why an unmerged adapter costs a little on every forward pass. Only A and B appear in the optimiser, so the saving is in gradients and optimiser state rather than in the forward activations. The figure does not claim the update is sufficient: a rank too small caps what the model can learn.

Text version

A vertical stack. An input vector x enters a group for one projection. Inside, the frozen matrix W runs on the main path while a dashed branch from x goes through A, which maps 4096 features to rank 16, then B, which maps back to 4096. The frozen output and the adapter output meet at a sum band labelled W x plus alpha over r times B A x, inside the same projection group, and the block output leaves at 4096. Two right-hand lanes carry the same x into A and W x into the sum. Five steps cover freezing, the down-projection, the up-projection, the sum, and what is stored.

  1. Pick target matrices. Attention projections are the usual starting set; adding the MLP projections increases capacity and cost. The choice matters more than most hyperparameters.
  2. Freeze the base. Every original weight has requires_grad=False. Only the adapter matrices receive gradients.
  3. Factor the update. For a frozen W∈Rdout×dinW \in \mathbb{R}^{d_{out} \times d_{in}}, learn A∈Rr×dinA \in \mathbb{R}^{r \times d_{in}} and B∈Rdout×rB \in \mathbb{R}^{d_{out} \times r} with r≪min⁡(din,dout)r \ll \min(d_{in}, d_{out}), and compute W′=W+αrBAW' = W + \frac{\alpha}{r} BA.
  4. Initialise so training starts at zero. AA is random, BB is zero, so BA=0BA = 0 and the adapted model begins identical to the base.
  5. Train with the usual objective. The loss is unchanged — for instruction tuning, cross-entropy on answer tokens as in SFT.
  6. Choose rank and scaling as a budget. Rank controls capacity; α/r\alpha/r controls how strongly the update is applied. Raising rank without adjusting scaling changes effective learning rate behaviour.
  7. Evaluate before merging. Compare the adapter against the base model on held-out tasks, not on training loss.
  8. Serve merged or separate. Merge for one adapter per deployment; keep separate for many adapters over one base, where requests are routed to an adapter at inference time.

The trainable parameter count is small and easy to compute per matrix:

∣A∣+∣B∣=r din+dout r=r (din+dout)|A| + |B| = r\,d_{in} + d_{out}\,r = r\,(d_{in} + d_{out})

For a square matrix this scales linearly in rr and in width, while the frozen matrix scales as d2d^2. That gap is why the optimiser state, not the weights, is what LoRA mostly saves.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
 
base = AutoModelForCausalLM.from_pretrained("your-base-model")
config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()
 
# ... train on your SFT objective ...
 
merged = model.merge_and_unload()   # one adapter, no inference overhead
merged.save_pretrained("merged-model")

Names of target modules are model-specific. Inspect the module list before copying a config from a tutorial written for a different architecture.

Trade-offs

MethodGood forCostFails when
Full fine-tuningLarge distribution shift, maximum headroomFull weight and optimiser memory per variantYou need many variants of one base
LoRABehaviour and format adaptation, many variantsAdapter plumbing, merge or routing stepThe task needs capacity a low rank cannot express
QLoRAFine-tuning when GPU memory is the constraintQuantised base, slower steps, slight quality riskYou need maximum throughput or exact numerics
Prompting onlyNo training infrastructureTokens per call, driftThe contract must hold across long sessions
Trade-off mapInteractive

Good for

Large distribution shift, maximum headroom

Cost

Full weight and optimiser memory per variant

Fails when

You need many variants of one base

  1. You need many variants of one base
  2. The task needs capacity a low rank cannot express
  3. You need maximum throughput or exact numerics
  4. The contract must hold across long sessions

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full fine-tuning is good for Large distribution shift, maximum headroom, costs Full weight and optimiser memory per variant, and fails when You need many variants of one base. LoRA is good for Behaviour and format adaptation, many variants, costs Adapter plumbing, merge or routing step, and fails when The task needs capacity a low rank cannot express. QLoRA is good for Fine-tuning when GPU memory is the constraint, costs Quantised base, slower steps, slight quality risk, and fails when You need maximum throughput or exact numerics. Prompting only is good for No training infrastructure, costs Tokens per call, drift, and fails when The contract must hold across long sessions

Failure modes

  • Rank too small. The adapter plateaus below target quality while training loss keeps falling. Debug by sweeping rank on a fixed data subset and watching held-out accuracy, not loss; if quality improves with rank, capacity was the limit.
  • Wrong target modules. Only query and value are adapted, and the task needs output-projection changes. Debug by printing the adapted module names and comparing against the architecture's full projection list.
  • Adapter silently ignored. Serving uses the base model or the wrong adapter version. Debug by sending a prompt that the adapter should answer distinctively and diffing against base output before blaming the data.
  • Merge into the wrong base. The adapter is applied to a different base revision or precision and quality degrades. Debug by recording the base model commit hash in the adapter metadata and asserting it at load time.
  • Cross-tenant adapter bleed. A batched server mixes adapters within one request batch. Debug by tagging each request's adapter ID in logs and by testing concurrent requests from two tenants.

In production

The serving decision is the important one. Merged adapters behave like ordinary models: simple, fast, one artifact per variant. Unmerged adapters let one base serve many tenants, but you pay for routing and for memory headroom, and the server must batch requests by adapter or accept padding waste. Every adapter you keep resident is memory that is not available for KV cache.

Version everything together: base model revision, adapter weights, tokenizer, chat template, and the training data hash. Adapters are small enough to store like code, and they should be reviewed like code, because a swapped adapter is a silent behaviour change.

Evaluate each adapter independently and the base model once. A regression in one tenant's adapter is not visible from aggregate metrics; per-adapter golden sets are the only reliable signal.

Common misconceptions

People think: LoRA is only a cheaper way to do the same thing.

Actually: It changes the deployment model too. Small, swappable adapters over one base make per-tenant and per-task variants practical in a way full fine-tunes are not.

People think: Higher rank always means better quality.

Actually: Rank raises capacity and cost. Past the point where the task's update is representable, extra rank mostly fits noise, and scaling interactions can make training less stable.

People think: LoRA adds latency at inference.

Actually: Merged adapters add nothing: the update is folded into the weights. Unmerged multi-adapter serving adds routing and batching overhead, which is a serving choice, not a property of LoRA.

Interview angle

"How would you fine-tune a model for twenty enterprise customers on one GPU pool?" The answer is the interesting part: LoRA or QLoRA adapters over a shared base, a target-module and rank choice justified by evaluation, a merge-or-route decision driven by traffic and isolation requirements, and per-adapter evaluation plus version pinning. Then name the failure you would guard against — an adapter that silently is not applied, or two tenants' requests batched into the wrong adapter. See SFT and the fine-tuning interview category.

Sources

  • SFT — the objective LoRA usually optimises.
  • Inference — how merged versus unmerged adapters affect serving.
  • RAG — when the gap is knowledge, not behaviour.
  • Cost modelling — training versus serving economics of adapters.