midReportedcurated
Explain LoRA mathematically. What do r and alpha control, and which modules would you target?
30-second answer
- LoRA freezes W and learns a low-rank update, with A of shape r by k and B of shape d by r, so the forward pass is Wx plus alpha over r times BAx.
- B is initialised to zero, so step zero reproduces the base model exactly and training starts from a known-good function rather than a random perturbation.
- Rank is capacity, typically 8 to 16 for style and format and 32 to 64 for harder shifts, while alpha over r decouples update magnitude from rank, which is why alpha at r or 2r is the sane default.
- Target all attention projections plus the MLP gate, up and down matrices; spreading capacity over attention and MLP is the cheaper first move when rank alone plateaus.
- Higher rank adds capacity to memorise a small dataset, so the failure mode is a better training loss and a worse held-out score, not a stronger adapter.
Likely follow-ups
Open a question to rehearse the answer.
Doubling r does not improve quality. What does that tell you?
- Capacity is not the binding constraint, so look at data quality, target modules and whether the task needs a behaviour the base cannot represent.
- Check that alpha was adjusted with r, because changing rank without rescaling changes the effective update size and can mask the real effect.
- Compare held-out accuracy rather than training loss, since a flat held-out curve with falling loss is memorisation.
Why does targeting more modules at lower rank often beat high rank on a few?
- Each adapted matrix adds a small low-rank correction, and adding the MLP increases capacity and cost, so breadth is the first thing to try when rank alone plateaus.
- If attention-only adapters plateau and adding MLP modules lifts held-out accuracy, the missing capacity was outside attention.
- The cost is more adapter parameters and a larger checkpoint, which matters for serving many adapters but not for a single merged model.
Red flags
- Cannot explain why B is initialised to zero
- Says higher rank is always better without mentioning dataset size
Concepts this tests
Tags
Source: github.com