Skip to content
AI Engineer Guide.
seniorReportedcurated

What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How would you uptrain an MHA model to GQA and what data would you use?
  • Mean-pool the existing key and value projections within each group to initialise the shared heads, then continue training so the queries adapt to coarser keys.
  • Use the original pretraining mix plus a slice of long-context and retrieval-heavy data, because reduced key diversity shows up there first.
  • A small fraction of the original compute is enough, since the model is adapting an existing representation rather than learning one.
  • Evaluate against the MHA baseline on long-context retrieval slices, not just on aggregate benchmarks that hide the regression.
When is multi-head attention still the right choice?
  • When the cache is not the bottleneck: short context, low concurrency, or a model small enough that memory per sequence is not binding.
  • When quality on tasks needing fine-grained discrimination matters more than throughput, since each shared head removes a distinct set of keys.
  • When the serving kernel does not support the head ratio, because an unsupported grouping either errors or silently truncates.
  • When no further training is affordable, since converting a checkpoint to GQA properly requires uptraining.

Red flags

  • Says GQA shares queries rather than keys and values
  • Believes GQA is only a training-speed trick

Concepts this tests

Tags

Source: github.com

More in LLM & transformer fundamentals