LLM & transformer fundamentals
9 questions
- What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?senior
- Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?mid
- Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?mid
- Explain speculative decoding. Why is it lossless, and when does it stop helping?senior
- Your GPU has free KV-cache memory but throughput is poor and requests sit idle waiting for a batch to finish. What is the lever, and how does it differ from PagedAttention?staff
- What does temperature actually do, give the formula, and why is temperature 0 not a determinism guarantee?junior
- Compare greedy, top-k, top-p and min-p sampling. Why truncate the distribution at all?mid
- How can you increase the context length of an LLM, and what actually limits it?staff
- Your transformer's KV cache grows too large during long sequence generation and you run out of memory. Walk me through diagnosis and mitigation.senior