seniorReportedcurated
What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?
30-second answer
- Multi-head attention gives every query head its own K and V; multi-query attention shares one K/V across all heads, and grouped-query attention shares one K/V per group of query heads.
- KV cache size scales with KV heads, so a GQA model with 8 groups cuts cache memory and decode bandwidth against MHA by roughly the ratio of query heads to KV heads, while staying close to MHA quality.
- MQA cuts further and decodes faster, but it gives up more head diversity and degrades more on tasks that need fine discrimination.
- Queries keep their full head count, so each query head still has its own projection and attention pattern; only keys and values are shared.
- An MHA checkpoint can be uptrained into GQA with a small fraction of pretraining compute, converting its KV projections by mean-pooling within each group.
Likely follow-ups
Open a question to rehearse the answer.
How would you uptrain an MHA model to GQA and what data would you use?
- Mean-pool the existing key and value projections within each group to initialise the shared heads, then continue training so the queries adapt to coarser keys.
- Use the original pretraining mix plus a slice of long-context and retrieval-heavy data, because reduced key diversity shows up there first.
- A small fraction of the original compute is enough, since the model is adapting an existing representation rather than learning one.
- Evaluate against the MHA baseline on long-context retrieval slices, not just on aggregate benchmarks that hide the regression.
When is multi-head attention still the right choice?
- When the cache is not the bottleneck: short context, low concurrency, or a model small enough that memory per sequence is not binding.
- When quality on tasks needing fine-grained discrimination matters more than throughput, since each shared head removes a distinct set of keys.
- When the serving kernel does not support the head ratio, because an unsupported grouping either errors or silently truncates.
- When no further training is affordable, since converting a checkpoint to GQA properly requires uptraining.
Red flags
- Says GQA shares queries rather than keys and values
- Believes GQA is only a training-speed trick
Concepts this tests
Tags
Source: github.com
More in LLM & transformer fundamentals
- Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?
- Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?
- Explain speculative decoding. Why is it lossless, and when does it stop helping?