midReportedcurated
Compare greedy, top-k, top-p and min-p sampling. Why truncate the distribution at all?
30-second answer
- Greedy takes the argmax at every step, which is stable and prone to loops and bland continuations because a likely phrase stays likely.
- Top-k keeps a fixed k tokens, which fails because distribution shape varies: a fixed cut admits junk when the model is confident and truncates real diversity when it is not.
- Top-p keeps the smallest set whose cumulative mass reaches p, so the candidate set adapts per step rather than being fixed at k candidates.
- Min-p keeps tokens with probability at least min_p times the top token's probability, which stays robust at high temperature where top-p's set can explode.
- Truncation exists because the tail is poorly calibrated and one bad token compounds autoregressively, so the trade is diversity against derailment.
Likely follow-ups
Open a question to rehearse the answer.
When does top-p fail and how does min-p address it?
- At high temperature the distribution flattens, so a fixed cumulative mass p can admit a very large candidate set including tokens the model would never choose at low temperature.
- Top-p adapts to cumulative mass but not to the ratio between the top token and the rest: with p 0.9 and a top token at 0.6, it still admits the low-probability tokens that fill the remaining 0.3.
- Min-p sets the cut relative to the top token's probability, so at min_p 0.1 nothing below 0.06 survives in that step, and the set shrinks whenever the model is confident.
- When the top token alone exceeds p, top-p keeps a single candidate, which is the opposite regime and is not the failure case.
- Neither fixes a badly calibrated model: both only decide which candidates are allowed to be drawn.
Why does greedy decoding loop mechanistically?
- Argmax commits to the highest-probability token, and after a repeated phrase the prefix makes the repetition the most likely continuation again.
- There is no mechanism to leave the loop, because the decoder never samples an alternative and never scores the sequence as a whole.
- The signature is the same n-gram repeating verbatim; a small top-two margin is the separate diagnostic for near-ties, which explains arbitrary flips rather than a loop.
- Raising temperature slightly or adding a repetition penalty breaks it, but both can cost format compliance.
Red flags
- Treats top-k and top-p as interchangeable
- Cannot explain why sampling the tail is risky
Concepts this tests
Tags
Source: github.com
More in LLM & transformer fundamentals
- What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?
- Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?
- Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?