Why it matters
Sampling parameters are the cheapest quality lever an application has, and the easiest to leave undocumented. A prompt that works at temperature zero can ramble at the default. A classification endpoint that silently inherits a creative preset produces label drift that looks like a model regression. When a provider changes a default or a model version changes its calibration, the same configuration produces different behaviour.
The parameters also define the product's contract. If the UI promises the same summary for the same document, sampling must be constrained and the model version pinned. If the product promises variety, a deterministic decoder is a bug. Neither promise can be kept by prompt wording alone.
Mental model
A mixing desk with a volume control and a gate. Temperature turns the volume on the whole distribution up or down; truncation sets a gate that removes quiet candidates entirely. The seed tries to make the performance repeatable, but the room acoustics — batching, hardware, model updates — still change.
The analogy breaks because the controls are not independent and not standardised. Providers implement the order of operations differently, some expose only a subset, and the same numeric value can mean different things across APIs. There is no universal "temperature 0.7".
How it works
In the model's own order: sorted by logit, descending, so a cut always falls on the tail below the kept set.
No truncation: 10 of 10 tokens kept, top probability 0.443, even share of the kept set 0.100, printed total 1.000.
- Tokens kept
- 10 of 10
- Top probability
- 0.443
- Kept mass before
- 1.000
No truncation: keeps 10 of 10 tokens, 100.0% of the original mass, then renormalises the kept set to 1.000. The full distribution.
Values
| category | Probability after the rule |
|---|---|
| token " the" · base probability 0.443 | 0.443 |
| token " a" · base probability 0.243 | 0.243 |
| token " of" · base probability 0.148 | 0.148 |
| token " and" · base probability 0.081 | 0.081 |
| token " to" · base probability 0.036 | 0.036 |
| token " in" · base probability 0.022 | 0.022 |
| token " that" · base probability 0.013 | 0.014 |
| token " with" · base probability 0.007 | 0.007 |
| token " from" · base probability 0.004 | 0.004 |
| token " bec" · base probability 0.002 | 0.002 |
Reference rule: even share of the kept 10 · 0.100 at 0.100.
Ten authored logits and an illustrative token set. Probabilities are the exact softmax at temperature 1.0, renormalised over the ten tokens shown and rounded by largest remainder. Truncation order and renormalisation follow the page's description; providers differ in both, which the figure cannot show.
Figure. Ten candidate tokens under the same logits, with four truncation rules applied before the draw. The dashed rule is an even share across the tokens the current rule keeps — 0.250 when top-p keeps four — so it moves with the control while the axis stays fixed. Removed tokens are drawn as small dashed stubs slashed through, and the readout and values table call them cut; the kept bars are renormalised to sum to 1.000, which is why they grow. Probabilities are renormalised over the ten candidates, not a vocabulary, and temperature is held at 1.0 here.
What to notice. The narration and the control do the same work, so either can be used. Top-k keeps a fixed five; top-p 0.9 keeps four, because the first four already carry 91.5% of the mass; min-p 4% keeps six, because it cuts relative to the top token and so adapts to how peaked the distribution is. The kept bars are taller than in the untruncated frame even though no probability changed: renormalisation redistributes the removed mass. Top-p equal to 1 would remove nothing and sampling would still occur.
Text version
A horizontal bar chart of ten candidate tokens, the, a, of, and, to, in, that, with, from and bec, showing probability after a truncation rule. A dashed rule marks an even share across the tokens the current rule keeps, which is 0.250 when top-p keeps four. The default frame is top-p 0.9: four tokens are kept and renormalise to 0.484, 0.266, 0.161 and 0.089, and the other six are small dashed stubs slashed through. Four buttons choose none, top-p 0.9, top-k 5 or min-p 0.04; none keeps all ten, top-k keeps five, min-p keeps six. The card reports how many tokens are kept, the top probability after renormalisation and the kept mass before renormalisation, all of which change with the control. Four steps walk no truncation, min-p, top-k and top-p in that order, and each step sets the control to its own rule; moving the control afterwards shows an override note.
Temperature is the most used and least understood. At the low end the distribution concentrates on the most likely token; at the high end it flattens and unlikely tokens become reachable. It does not "add creativity" as a property; it changes the entropy of the sampling distribution, and the effect depends on how confident the model already was.
Top-p (nucleus) keeps the smallest set of tokens whose cumulative probability exceeds the threshold. It adapts to confidence: when the model is sure, the set is small; when it is uncertain, the set grows. Top-k keeps a fixed count, which behaves poorly when the distribution is flat or extremely peaked. Min-p keeps tokens above a fraction of the top token's probability, scaling the candidate set with confidence.
Max tokens is a hard cap, not a target. It is the parameter most often responsible for truncated JSON and half-finished sentences. Stop sequences end generation when a string matches; they are useful for chat turn boundaries but dangerous when the string can appear in legitimate content. Penalties discourage repeated tokens or encourage variety, and they are a common cause of broken code, where repeated punctuation and identifiers are normal.
Seeds are best-effort. A seed can make repeated calls more similar on the same provider and model, but it does not override batching nondeterminism, model updates, or infrastructure changes.
Provider differences that bite
OpenAI documents temperature in the range 0 to 2 for chat completions, with a default of 1. Anthropic's older Messages API documented 0 to 1 and recommended adjusting either temperature or top-p rather than both; its 2026-10-21 API version removed temperature, top-p and top-k, so sending them now fails rather than being clamped. Google's Gemini API documents its own ranges and defaults. A configuration object copied between providers without a version check is a silent behaviour change.
The safe pattern is a provider-neutral configuration with explicit translation per adapter: intent ("deterministic extraction", "balanced chat", "diverse drafting") mapped to concrete values for each provider, with the resolved values logged alongside the response.
A configuration table worth versioning
# Illustrative presets. Resolve per provider; do not assume ranges are portable.
PRESETS = {
"extraction": {"temperature": 0.0, "top_p": 1.0, "max_tokens": 800},
"classification": {"temperature": 0.0, "top_p": 1.0, "max_tokens": 16},
"support_chat": {"temperature": 0.3, "top_p": 0.9, "max_tokens": 600},
"drafting": {"temperature": 0.8, "top_p": 0.95, "max_tokens": 1200},
}
def resolve(preset: str, provider: str) -> dict:
cfg = dict(PRESETS[preset])
if provider == "anthropic":
# Older Messages API versions documented 0..1. The 2026-10-21 version
# rejects temperature/top-p/top-k entirely, so check the model version.
cfg["temperature"] = min(cfg["temperature"], 1.0)
return cfgThe clamp is not a quality decision; it prevents an out-of-range error when a preset is reused across adapters. Keep the preset name in the trace so an eval result can be attributed to a configuration, not just a prompt.
Trade-offs
| Setting | Good for | Cost | Fails when |
|---|---|---|---|
| Temperature 0 | Extraction, classification, reproducible answers | Repetition, brittle to prompt changes | The task needs varied phrasing or exploration |
| Low temperature with top-p | Support chat, summarisation | Still varies run to run | The product promises identical output |
| High temperature with top-p | Brainstorming, marketing drafts | Incoherence, factual drift | Accuracy is part of the requirement |
| Frequency/presence penalties | Breaking repetition in long copy | Breaks code and structured formats | Output must be valid JSON or source code |
| Multiple samples (n) | Reasoning with voting, reranking candidates | Linear cost increase | Latency budget allows one call |
Good for
Extraction, classification, reproducible answers
Cost
Repetition, brittle to prompt changes
Fails when
The task needs varied phrasing or exploration
- The task needs varied phrasing or exploration
- The product promises identical output
- Accuracy is part of the requirement
- Output must be valid JSON or source code
- Latency budget allows one call
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Temperature 0 is good for Extraction, classification, reproducible answers, costs Repetition, brittle to prompt changes, and fails when The task needs varied phrasing or exploration. Low temperature with top-p is good for Support chat, summarisation, costs Still varies run to run, and fails when The product promises identical output. High temperature with top-p is good for Brainstorming, marketing drafts, costs Incoherence, factual drift, and fails when Accuracy is part of the requirement. Frequency/presence penalties is good for Breaking repetition in long copy, costs Breaks code and structured formats, and fails when Output must be valid JSON or source code. Multiple samples (n) is good for Reasoning with voting, reranking candidates, costs Linear cost increase, and fails when Latency budget allows one call
Failure modes
Configuration drift between environments. Staging and production use different defaults, and an eval that passed does not reflect the shipped behaviour. Debug by logging the resolved parameters with every response and diffing them across environments.
Two knobs changed at once. Quality moves and nobody knows why. Debug by changing one parameter per experiment against a fixed eval set, starting with temperature because it has the largest effect.
Truncation blamed on the model. The answer stops mid-sentence. Debug by checking the finish reason and the max-token setting. If it is a length stop, the parameter is the bug.
Penalties corrupting structured output. Repeated keys or punctuation get suppressed and the JSON becomes invalid. Debug by disabling penalties for structured endpoints and re-running the same inputs.
Symptom
Staging and production use different defaults, and an eval that passed does not reflect the shipped behaviour.
Debug step
Logging the resolved parameters with every response and diffing them across environments.
Figure. Each failure with its symptom and the check that would confirm it.
What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.
Text version
Failure modes: Configuration drift between environments: Staging and production use different defaults, and an eval that passed does not reflect the shipped behaviour. Debug by logging the resolved parameters with every response and diffing them across environments. | Two knobs changed at once: Quality moves and nobody knows why. Debug by changing one parameter per experiment against a fixed eval set, starting with temperature because it has the largest effect. | Truncation blamed on the model: The answer stops mid-sentence. Debug by checking the finish reason and the max-token setting. If it is a length stop, the parameter is the bug. | Penalties corrupting structured output: Repeated keys or punctuation get suppressed and the JSON becomes invalid. Debug by disabling penalties for structured endpoints and re-running the same inputs.
In production
Store sampling parameters with the prompt version, not in application code scattered across call sites. Give each endpoint a named preset and a documented intent. Log the resolved values, the finish reason and the token counts. When a provider deprecates a model, treat the migration as a behaviour change and re-run the eval set before switching traffic. For user-facing variety, evaluate with multiple samples so that a lucky single run does not hide a bad distribution.
Common misconceptions
People think: Temperature is a creativity dial that works the same everywhere.
Actually: It rescales logits within a provider-specific range and interacts with truncation. The same number can behave differently across providers and models.
People think: Setting a seed makes output reproducible.
Actually: Seeds are best-effort. Batching, hardware, model updates and provider-side changes can still alter the result, so reproducibility needs version pinning and validation.
People think: Top-p equal to one disables sampling.
Actually: It keeps the full distribution, so sampling still occurs according to the probabilities. Only temperature zero with argmax removes the draw.
Interview angle
Expect a question about an inconsistent assistant or a JSON endpoint that breaks intermittently. The strong answer separates the distribution shape from the truncation rule, names the finish reason as evidence, and proposes a versioned configuration with an eval gate. The weak answer turns temperature down and hopes. See the LLM fundamentals bank and the prompting bank.
Sources
- OpenAI, Chat Completions API reference — documented temperature range, top-p, penalties and token limits.
- Anthropic, Messages API reference — current parameter set (version-sensitive).
- Holtzman et al., The Curious Case of Neural Text Degeneration — nucleus sampling and the failure of pure likelihood decoding.
- OpenAI Cookbook, Reproducible outputs with the seed parameter — what seeds do and do not guarantee.
Related
- Decoding — the selection procedure these parameters configure.
- Greedy and beam search — deterministic alternatives.
- Structured outputs — why penalties and truncation break parsers.
- Prompt versioning — configuration belongs in the same release.
- LLM-as-judge — evaluating changes that are stochastic.