juniorReportedcurated
What does temperature actually do, give the formula, and why is temperature 0 not a determinism guarantee?
30-second answer
- Temperature divides the logits before the softmax, p_i = exp(z_i/T) / Σ exp(z_j/T), so it reshapes the distribution without changing the ranking of tokens.
- T below 1 sharpens the distribution toward the argmax, T above 1 flattens it and gives low-probability tokens more chance, and T = 1 leaves the model's estimate unchanged.
- T = 0 is implemented as argmax, and it is not a determinism guarantee: batching, floating-point non-associativity, MoE routing and provider updates can all change outputs.
- Use near-zero for extraction, classification and code, moderate values for chat and drafting, and remember the effect depends on how confident the model already was.
- The trade is diversity against reliability: lower temperature improves consistency and increases repetition, while higher temperature buys variety and invites derailment.
Likely follow-ups
Open a question to rehearse the answer.
What temperature would you use for an LLM-as-judge and why?
- Near zero, because the same input should get the same verdict and the judge is being used as a measurement rather than a generator.
- A judge that varies between runs makes every eval comparison noisy, and the noise can hide a real regression or invent one.
- Expect repetition and brittle phrasing as the cost, and pin the prompt version and schema so verdicts stay comparable across runs.
- Verify it against human labels on a sample, since a consistent judge can be consistently wrong.
How does temperature interact with top-p ordering?
- Temperature and truncation compose in the provider's documented order: if temperature runs first, flattening the distribution makes a fixed top-p keep more candidates; if truncation runs on raw logits, temperature changes the draw but not the candidate set.
- At high temperature more mass sits in the tail, so a fixed top-p keeps a larger candidate set; at low temperature the same p keeps very few tokens.
- Providers apply these in their own documented order and some reject the parameters entirely, so the same numeric pair is not portable across APIs.
- Changing both at once makes a regression impossible to attribute, so move one knob per experiment.
Red flags
- Says temperature changes which tokens are ranked highest
- Asserts T=0 gives byte-identical outputs across calls
Concepts this tests
Tags
Source: github.com
More in LLM & transformer fundamentals
- What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?
- Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?
- Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?