Skip to content
AI Engineer Guide.
midReportedcurated

Explain LoRA mathematically. What do r and alpha control, and which modules would you target?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Doubling r does not improve quality. What does that tell you?
  • Capacity is not the binding constraint, so look at data quality, target modules and whether the task needs a behaviour the base cannot represent.
  • Check that alpha was adjusted with r, because changing rank without rescaling changes the effective update size and can mask the real effect.
  • Compare held-out accuracy rather than training loss, since a flat held-out curve with falling loss is memorisation.
Why does targeting more modules at lower rank often beat high rank on a few?
  • Each adapted matrix adds a small low-rank correction, and adding the MLP increases capacity and cost, so breadth is the first thing to try when rank alone plateaus.
  • If attention-only adapters plateau and adding MLP modules lifts held-out accuracy, the missing capacity was outside attention.
  • The cost is more adapter parameters and a larger checkpoint, which matters for serving many adapters but not for a single merged model.

Red flags

  • Cannot explain why B is initialised to zero
  • Says higher rank is always better without mentioning dataset size

Concepts this tests

Tags

Source: github.com

More in Fine-tuning & adaptation