Skip to content
AI Engineer Guide.
midReportedcurated

Compare greedy, top-k, top-p and min-p sampling. Why truncate the distribution at all?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

When does top-p fail and how does min-p address it?
  • At high temperature the distribution flattens, so a fixed cumulative mass p can admit a very large candidate set including tokens the model would never choose at low temperature.
  • Top-p adapts to cumulative mass but not to the ratio between the top token and the rest: with p 0.9 and a top token at 0.6, it still admits the low-probability tokens that fill the remaining 0.3.
  • Min-p sets the cut relative to the top token's probability, so at min_p 0.1 nothing below 0.06 survives in that step, and the set shrinks whenever the model is confident.
  • When the top token alone exceeds p, top-p keeps a single candidate, which is the opposite regime and is not the failure case.
  • Neither fixes a badly calibrated model: both only decide which candidates are allowed to be drawn.
Why does greedy decoding loop mechanistically?
  • Argmax commits to the highest-probability token, and after a repeated phrase the prefix makes the repetition the most likely continuation again.
  • There is no mechanism to leave the loop, because the decoder never samples an alternative and never scores the sequence as a whole.
  • The signature is the same n-gram repeating verbatim; a small top-two margin is the separate diagnostic for near-ties, which explains arbitrary flips rather than a loop.
  • Raising temperature slightly or adding a repetition penalty breaks it, but both can cost format compliance.

Red flags

  • Treats top-k and top-p as interchangeable
  • Cannot explain why sampling the tail is risky

Concepts this tests

Tags

Source: github.com

More in LLM & transformer fundamentals