Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 14 of 28

Sampling parameters

0 of 28 complete

Why it matters

Sampling parameters are the cheapest quality lever an application has, and the easiest to leave undocumented. A prompt that works at temperature zero can ramble at the default. A classification endpoint that silently inherits a creative preset produces label drift that looks like a model regression. When a provider changes a default or a model version changes its calibration, the same configuration produces different behaviour.

The parameters also define the product's contract. If the UI promises the same summary for the same document, sampling must be constrained and the model version pinned. If the product promises variety, a deterministic decoder is a bug. Neither promise can be kept by prompt wording alone.

Mental model

A mixing desk with a volume control and a gate. Temperature turns the volume on the whole distribution up or down; truncation sets a gate that removes quiet candidates entirely. The seed tries to make the performance repeatable, but the room acoustics — batching, hardware, model updates — still change.

The analogy breaks because the controls are not independent and not standardised. Providers implement the order of operations differently, some expose only a subset, and the same numeric value can mean different things across APIs. There is no universal "temperature 0.7".

How it works

One distribution, four truncation rulesInteractive
Truncation rule
Probability after the rule

In the model's own order: sorted by logit, descending, so a cut always falls on the tail below the kept set.

No truncation: 10 of 10 tokens kept, top probability 0.443, even share of the kept set 0.100, printed total 1.000.

Tokens kept
10 of 10
Top probability
0.443
Kept mass before
1.000
Step 1/4 · No truncation

No truncation: keeps 10 of 10 tokens, 100.0% of the original mass, then renormalises the kept set to 1.000. The full distribution.

Values
Printed values for every category, in the drawn order. The axis runs from 0 to 0.6. Current frame: Truncation rule none.
categoryProbability after the rule
token " the" · base probability 0.4430.443
token " a" · base probability 0.2430.243
token " of" · base probability 0.1480.148
token " and" · base probability 0.0810.081
token " to" · base probability 0.0360.036
token " in" · base probability 0.0220.022
token " that" · base probability 0.0130.014
token " with" · base probability 0.0070.007
token " from" · base probability 0.0040.004
token " bec" · base probability 0.0020.002

Reference rule: even share of the kept 10 · 0.100 at 0.100.

Ten authored logits and an illustrative token set. Probabilities are the exact softmax at temperature 1.0, renormalised over the ten tokens shown and rounded by largest remainder. Truncation order and renormalisation follow the page's description; providers differ in both, which the figure cannot show.

Figure. Ten candidate tokens under the same logits, with four truncation rules applied before the draw. The dashed rule is an even share across the tokens the current rule keeps — 0.250 when top-p keeps four — so it moves with the control while the axis stays fixed. Removed tokens are drawn as small dashed stubs slashed through, and the readout and values table call them cut; the kept bars are renormalised to sum to 1.000, which is why they grow. Probabilities are renormalised over the ten candidates, not a vocabulary, and temperature is held at 1.0 here.

What to notice. The narration and the control do the same work, so either can be used. Top-k keeps a fixed five; top-p 0.9 keeps four, because the first four already carry 91.5% of the mass; min-p 4% keeps six, because it cuts relative to the top token and so adapts to how peaked the distribution is. The kept bars are taller than in the untruncated frame even though no probability changed: renormalisation redistributes the removed mass. Top-p equal to 1 would remove nothing and sampling would still occur.

Text version

A horizontal bar chart of ten candidate tokens, the, a, of, and, to, in, that, with, from and bec, showing probability after a truncation rule. A dashed rule marks an even share across the tokens the current rule keeps, which is 0.250 when top-p keeps four. The default frame is top-p 0.9: four tokens are kept and renormalise to 0.484, 0.266, 0.161 and 0.089, and the other six are small dashed stubs slashed through. Four buttons choose none, top-p 0.9, top-k 5 or min-p 0.04; none keeps all ten, top-k keeps five, min-p keeps six. The card reports how many tokens are kept, the top probability after renormalisation and the kept mass before renormalisation, all of which change with the control. Four steps walk no truncation, min-p, top-k and top-p in that order, and each step sets the control to its own rule; moving the control afterwards shows an override note.

Temperature is the most used and least understood. At the low end the distribution concentrates on the most likely token; at the high end it flattens and unlikely tokens become reachable. It does not "add creativity" as a property; it changes the entropy of the sampling distribution, and the effect depends on how confident the model already was.

Top-p (nucleus) keeps the smallest set of tokens whose cumulative probability exceeds the threshold. It adapts to confidence: when the model is sure, the set is small; when it is uncertain, the set grows. Top-k keeps a fixed count, which behaves poorly when the distribution is flat or extremely peaked. Min-p keeps tokens above a fraction of the top token's probability, scaling the candidate set with confidence.

Max tokens is a hard cap, not a target. It is the parameter most often responsible for truncated JSON and half-finished sentences. Stop sequences end generation when a string matches; they are useful for chat turn boundaries but dangerous when the string can appear in legitimate content. Penalties discourage repeated tokens or encourage variety, and they are a common cause of broken code, where repeated punctuation and identifiers are normal.

Seeds are best-effort. A seed can make repeated calls more similar on the same provider and model, but it does not override batching nondeterminism, model updates, or infrastructure changes.

Provider differences that bite

OpenAI documents temperature in the range 0 to 2 for chat completions, with a default of 1. Anthropic's older Messages API documented 0 to 1 and recommended adjusting either temperature or top-p rather than both; its 2026-10-21 API version removed temperature, top-p and top-k, so sending them now fails rather than being clamped. Google's Gemini API documents its own ranges and defaults. A configuration object copied between providers without a version check is a silent behaviour change.

The safe pattern is a provider-neutral configuration with explicit translation per adapter: intent ("deterministic extraction", "balanced chat", "diverse drafting") mapped to concrete values for each provider, with the resolved values logged alongside the response.

A configuration table worth versioning

# Illustrative presets. Resolve per provider; do not assume ranges are portable.
PRESETS = {
    "extraction": {"temperature": 0.0, "top_p": 1.0, "max_tokens": 800},
    "classification": {"temperature": 0.0, "top_p": 1.0, "max_tokens": 16},
    "support_chat": {"temperature": 0.3, "top_p": 0.9, "max_tokens": 600},
    "drafting": {"temperature": 0.8, "top_p": 0.95, "max_tokens": 1200},
}
 
def resolve(preset: str, provider: str) -> dict:
    cfg = dict(PRESETS[preset])
    if provider == "anthropic":
        # Older Messages API versions documented 0..1. The 2026-10-21 version
        # rejects temperature/top-p/top-k entirely, so check the model version.
        cfg["temperature"] = min(cfg["temperature"], 1.0)
    return cfg

The clamp is not a quality decision; it prevents an out-of-range error when a preset is reused across adapters. Keep the preset name in the trace so an eval result can be attributed to a configuration, not just a prompt.

Trade-offs

SettingGood forCostFails when
Temperature 0Extraction, classification, reproducible answersRepetition, brittle to prompt changesThe task needs varied phrasing or exploration
Low temperature with top-pSupport chat, summarisationStill varies run to runThe product promises identical output
High temperature with top-pBrainstorming, marketing draftsIncoherence, factual driftAccuracy is part of the requirement
Frequency/presence penaltiesBreaking repetition in long copyBreaks code and structured formatsOutput must be valid JSON or source code
Multiple samples (n)Reasoning with voting, reranking candidatesLinear cost increaseLatency budget allows one call
Trade-off mapInteractive

Good for

Extraction, classification, reproducible answers

Cost

Repetition, brittle to prompt changes

Fails when

The task needs varied phrasing or exploration

  1. The task needs varied phrasing or exploration
  2. The product promises identical output
  3. Accuracy is part of the requirement
  4. Output must be valid JSON or source code
  5. Latency budget allows one call

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Temperature 0 is good for Extraction, classification, reproducible answers, costs Repetition, brittle to prompt changes, and fails when The task needs varied phrasing or exploration. Low temperature with top-p is good for Support chat, summarisation, costs Still varies run to run, and fails when The product promises identical output. High temperature with top-p is good for Brainstorming, marketing drafts, costs Incoherence, factual drift, and fails when Accuracy is part of the requirement. Frequency/presence penalties is good for Breaking repetition in long copy, costs Breaks code and structured formats, and fails when Output must be valid JSON or source code. Multiple samples (n) is good for Reasoning with voting, reranking candidates, costs Linear cost increase, and fails when Latency budget allows one call

Failure modes

Configuration drift between environments. Staging and production use different defaults, and an eval that passed does not reflect the shipped behaviour. Debug by logging the resolved parameters with every response and diffing them across environments.

Two knobs changed at once. Quality moves and nobody knows why. Debug by changing one parameter per experiment against a fixed eval set, starting with temperature because it has the largest effect.

Truncation blamed on the model. The answer stops mid-sentence. Debug by checking the finish reason and the max-token setting. If it is a length stop, the parameter is the bug.

Penalties corrupting structured output. Repeated keys or punctuation get suppressed and the JSON becomes invalid. Debug by disabling penalties for structured endpoints and re-running the same inputs.

Failure modes, one at a timeInteractive

Symptom

Staging and production use different defaults, and an eval that passed does not reflect the shipped behaviour.

Debug step

Logging the resolved parameters with every response and diffing them across environments.

Figure. Each failure with its symptom and the check that would confirm it.

What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.

Text version

Failure modes: Configuration drift between environments: Staging and production use different defaults, and an eval that passed does not reflect the shipped behaviour. Debug by logging the resolved parameters with every response and diffing them across environments. | Two knobs changed at once: Quality moves and nobody knows why. Debug by changing one parameter per experiment against a fixed eval set, starting with temperature because it has the largest effect. | Truncation blamed on the model: The answer stops mid-sentence. Debug by checking the finish reason and the max-token setting. If it is a length stop, the parameter is the bug. | Penalties corrupting structured output: Repeated keys or punctuation get suppressed and the JSON becomes invalid. Debug by disabling penalties for structured endpoints and re-running the same inputs.

In production

Store sampling parameters with the prompt version, not in application code scattered across call sites. Give each endpoint a named preset and a documented intent. Log the resolved values, the finish reason and the token counts. When a provider deprecates a model, treat the migration as a behaviour change and re-run the eval set before switching traffic. For user-facing variety, evaluate with multiple samples so that a lucky single run does not hide a bad distribution.

Common misconceptions

People think: Temperature is a creativity dial that works the same everywhere.

Actually: It rescales logits within a provider-specific range and interacts with truncation. The same number can behave differently across providers and models.

People think: Setting a seed makes output reproducible.

Actually: Seeds are best-effort. Batching, hardware, model updates and provider-side changes can still alter the result, so reproducibility needs version pinning and validation.

People think: Top-p equal to one disables sampling.

Actually: It keeps the full distribution, so sampling still occurs according to the probabilities. Only temperature zero with argmax removes the draw.

Interview angle

Expect a question about an inconsistent assistant or a JSON endpoint that breaks intermittently. The strong answer separates the distribution shape from the truncation rule, names the finish reason as evidence, and proposes a versioned configuration with an eval gate. The weak answer turns temperature down and hopes. See the LLM fundamentals bank and the prompting bank.

Sources