Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 8 of 22

QLoRA

0 of 22 complete

Why it matters

A large base model in half precision can exceed the memory of the device you can rent, before any activations. QLoRA changes the storage format of the frozen weights so more of the budget is left for activations and the adapter. The engineering problem is a three-way squeeze between weight memory, activation memory and throughput, and quantisation buys the first by spending some of the third.

Mental model

QLoRA is reading a compressed archive and unpacking one page at a time: the shelf holds more, but every read costs a decompression step. The analogy breaks because the unpacked page is not identical to the original, and the adapter has to learn around that difference.

How it works

Storage is quantised; the arithmetic stays bf16Interactive
signalbypassfrozentrainable

Step 1 of 5

Store the base low

The frozen weights are written in a 4-bit block format with per-block scales. The saving is in weight memory, before any activation is allocated.

Active layers

  • Input activation
  • Frozen base · 4-bit storage
  • Stored weights

Hover a layer to read what it holds.

Next: Unpack per matmul

Base
4-bit NF4 · block 64
Compute
bf16 per matmul
Adapter
bf16 · paged state
Step 1/5 · Store the base low

Figure. What quantisation changes, layer by layer: the frozen weights are stored in 4-bit blocks, dequantised a tile at a time, and multiplied in bf16, while the adapter and its state stay high precision. The dashed lane is the adapter branch, which reads the same activation. Block size and dtype are illustrative. Activations, the adapter's optimiser state and the serving stack's dequantisation point are omitted from the drawing and named in the steps.

What to notice. The storage band is the only band whose contents changed: the arithmetic below it is bf16 and sits outside the 4-bit group, so steps are usually slower than 16-bit LoRA even though the model fits. If the serving stack dequantises the whole model at load, the saving is gone and the figure no longer describes the deployment. The adapter can absorb part of the quantisation error, but the approximation is still there when the adapter is merged into the weights.

Text version

A vertical stack. An input activation in bf16 enters a frozen base group holding stored weights in NF4 blocks of 64, then a dequantise band that unpacks one tile per matmul, then a bf16 matrix multiply. A dashed lane carries the same activation into an adapter group holding A and B in bf16; A and B meet the multiply output at a sum band and the layer output leaves in bf16. Five steps cover storage, per-matmul dequantisation, bf16 compute, the high-precision adapter, and measuring speed and memory.

  1. Quantise the frozen base to a low-bit block format and keep the adapter and its optimiser state in higher precision.
  2. Choose block size and quantisation dtype together, since finer blocks cost metadata and coarser blocks lose accuracy.
  3. Enable paged optimiser state so a memory spike becomes a stall rather than an out-of-memory failure.
  4. Check that dequantisation happens inside the compute path rather than once at load, or the memory saving disappears.
  5. Measure tokens per second and peak memory against 16-bit LoRA at the same batch size before committing.
  6. Evaluate the quantised base with the adapter attached, and again after any merge, since merging into quantised weights changes the arithmetic.

Trade-offs

ChoiceGood forCostFails when
Low-bit base with adapter on topFits a large base on one deviceSlower steps and quantisation errorThe task is sensitive to fine numeric detail
16-bit LoRAFaster and numerically cleanMore weight memoryThe base does not fit the device
Aggressive quantisationMaximum memory headroomAccuracy loss the adapter must absorbEvaluation margins are already thin
Higher-bit quantisationCloser to the 16-bit baselineLess memory savedMemory is the hard constraint
Trade-off mapInteractive

Good for

Fits a large base on one device

Cost

Slower steps and quantisation error

Fails when

The task is sensitive to fine numeric detail

  1. The task is sensitive to fine numeric detail
  2. The base does not fit the device
  3. Evaluation margins are already thin
  4. Memory is the hard constraint

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Low-bit base with adapter on top is good for Fits a large base on one device, costs Slower steps and quantisation error, and fails when The task is sensitive to fine numeric detail. 16-bit LoRA is good for Faster and numerically clean, costs More weight memory, and fails when The base does not fit the device. Aggressive quantisation is good for Maximum memory headroom, costs Accuracy loss the adapter must absorb, and fails when Evaluation margins are already thin. Higher-bit quantisation is good for Closer to the 16-bit baseline, costs Less memory saved, and fails when Memory is the hard constraint

Failure modes

  • Quantised compute path. Weights are stored quantised but dequantised for the whole forward pass, so memory peaks like full precision. Debug: log peak allocated memory during a forward pass and compare it with the theoretical weight footprint.
  • Adapter learning the quantisation error. Loss falls but the merged model is worse than the adapter alone. Debug: evaluate adapter-attached and merged variants on the same held-out set.
  • Block size mismatch at export. Inference uses a different quantisation layout and outputs shift. Debug: compare logits for a fixed prompt between the training and serving stacks.

In production

Pin the quantisation configuration with the adapter and treat a change of block size as a model change. Watch for memory stalls from paging, which appear as latency spikes rather than errors.

Common misconceptions

People think: QLoRA trains a quantised model.

Actually: The base stays frozen and quantised; only the adapter is trained, in higher precision.

People think: Quantisation is free if the loss looks fine.

Actually: Training loss can hide degradation that shows up on tasks needing precise numbers or long context.

Interview angle

Be precise about what is quantised and what is not, then the memory arithmetic: weights down, activations unchanged, throughput usually worse. Mention that the adapter absorbs some quantisation error and that merging changes results. Answers that call it a smaller model, or expect a speed-up, have the mechanism backwards.

Sources