Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 6 of 27

Probability

0 of 27 complete

Why it matters

A model that returns a single answer hides its uncertainty. A model that returns a distribution exposes it, and then you have to decide what to do with it: take the most likely token, sample from the tail, reject the request, or escalate to a human.

That decision is where most production incidents live. Greedy decoding on a task that needs diversity produces repetition; high-temperature sampling on a task that needs exact extraction produces plausible nonsense; an uncalibrated confidence score used as a routing threshold sends the wrong queries to the expensive path. None of these are model bugs. They are misuse of probabilities.

Probability also underpins evaluation. "The new prompt won 7 of 10 cases" is not evidence of improvement. Understanding variance and sample size is what stops a team from shipping noise.

Mental model: a budget split across outcomes

Think of a probability distribution as a fixed budget of 1.0 that must be divided among all possible outcomes. A fair coin splits it 0.5 and 0.5. A model's next-token distribution splits it across tens of thousands of tokens, with most of the budget concentrated on a few plausible continuations.

Conditioning is the same budget re-divided after new information arrives: given that the document is about billing, the budget shifts away from travel tokens.

The analogy breaks in a specific way. The budget is not a property of the world; it is a property of the model's beliefs. Two models can assign different budgets to the same event, and both can be wrong. Probability from a model is a claim about its own uncertainty, not a measurement of reality.

How it works

One budget, four outcomes, three temperaturesInteractive
Temperature
Probability

Ordered by logit, descending. The order is fixed at every temperature, so the bars never swap places.

T = 0.5: A 0.862 · B 0.117 · C 0.019 · D 0.002 — total 1.000.

Top share
0.862
Bottom share
0.002
Printed total
1.000
Step 1/3 · T = 0.5

T = 0.5 divides the logits before the softmax and sharpens it: outcome A rises to 0.862 and D falls to 0.002. No ranking changed; the budget was redistributed.

Values
Printed values for every category, in the drawn order. The axis runs from 0 to 1. Current frame: Temperature T 0.5.
categoryProbability
Outcome A · logit 2.00.862
Outcome B · logit 1.00.117
Outcome C · logit 0.10.019
Outcome D · logit -1.00.002

Reference rule: even split of 4 · 0.250 at 0.250.

The logits are the page's own worked example and every bar is the exact softmax at the stated temperature, rounded by largest remainder so the printed total is 1.000. Four outcomes are shown rather than a vocabulary, and conditioning, calibration and Bayes' rule are not in the figure.

Figure. The page's own worked example — logits 2.0, 1.0, 0.1 and −1.0 — turned into probabilities at T = 0.5, 1 and 2. The dashed rule is 0.25, an even split of the budget across four outcomes. Every bar is the exact softmax of those logits at the stated temperature; the page prints the same three rows.

What to notice. The four shares always sum to 1.000; that is the whole claim of a distribution. Temperature moves mass between outcomes without changing which is most likely. The uniform rule is the comparison that matters: at T = 2 outcome D is still below an even share, and at T = 0.5 it is about 400 times smaller than A. This is a distribution over four outcomes, not a vocabulary, and it is not a calibrated probability that any outcome is correct.

Text version

A horizontal bar chart of four outcomes labelled A with logit 2.0, B with 1.0, C with 0.1 and D with minus 1.0, showing each probability. A dashed rule at 0.25 marks an even split across four outcomes. Three buttons set the temperature to 0.5, 1 or 2; the default is 1, where the shares are 0.638, 0.235, 0.095 and 0.032 and sum to 1.000. At 0.5 they run 0.862, 0.117, 0.019 and 0.002. At 2 they flatten to 0.451, 0.274, 0.174 and 0.101. The card reports the top share, bottom share and printed total for the current frame. Steps sharpen, flatten, then return to T = 1 and highlight A, D and the whole chart in turn.

  1. Define outcomes. The sample space is the set of things that can happen: the next token, the document's class, whether a user churns. Events are subsets of that space.
  2. Assign and normalise. Each outcome gets a non-negative number, and all numbers sum to 1. A model produces unnormalised scores (logits); softmax converts them into a valid distribution.
  3. Condition. P(A∣B)P(A \mid B) is the probability of AA given that BB happened. Conditioning restricts the sample space and renormalises.
  4. Check independence before multiplying. P(A,B)=P(A)P(B)P(A, B) = P(A)P(B) only when AA and BB are independent. Retrieval scores, judge votes and repeated model calls are usually correlated, so naive multiplication overstates confidence.
  5. Use random variables to summarise. A random variable maps outcomes to numbers — the number of retries, the latency, the score. Its expectation is the long-run average and its variance is the spread.
  6. Pick a distribution that fits the outcome type. Bernoulli for a binary event, categorical for one of many tokens, Gaussian for continuous noise around a mean.
  7. Update with Bayes' rule. Start from a prior, multiply by the likelihood of the evidence, normalise. This is the formal version of "how much should this new signal change my mind".
  8. Work in log space. Products of many small probabilities underflow to zero in float32; sums of log-probabilities do not.

The math

Bayes' rule, in the form used for updating a belief:

P(A∣B)=P(B∣A) P(A)P(B).P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}.

Expectation and variance for a discrete random variable:

E[X]=∑xx P(x),Var(X)=E[(X−E[X])2].\mathbb{E}[X] = \sum_x x\,P(x), \qquad \mathrm{Var}(X) = \mathbb{E}\left[(X - \mathbb{E}[X])^2\right].

Softmax with a temperature TT turns logits zz into a distribution; T<1T < 1 sharpens it, T>1T > 1 flattens it:

pi=exp⁡(zi/T)∑jexp⁡(zj/T).p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}.

A minimal implementation

import numpy as np
 
def softmax(logits: np.ndarray, temperature: float = 1.0) -> np.ndarray:
    z = np.asarray(logits, dtype=np.float64) / temperature
    z = z - z.max()                       # log-sum-exp shift: same result, no overflow
    e = np.exp(z)
    return e / e.sum()
 
def report_confidence(probs: np.ndarray) -> dict:
    top = int(np.argmax(probs))
    return {"token": top, "p": float(probs[top]),
            "entropy": float(-(probs * np.log(probs + 1e-12)).sum())}
 
logits = np.array([2.0, 1.0, 0.1, -1.0])
for t in (0.5, 1.0, 2.0):
    p = softmax(logits, temperature=t)
    print(t, np.round(p, 3), report_confidence(p)["p"])
 
rng = np.random.default_rng(0)
draws = rng.choice(len(logits), size=10_000, p=softmax(logits))
print(np.bincount(draws, minlength=4) / 10_000)   # converges to the distribution

Two details are load-bearing. The z.max() subtraction prevents exp overflow without changing the result. And p=softmax(logits) is only correct as a sampling distribution — reading p as "the probability the answer is right" is a different claim that needs calibration evidence.

Trade-offs

ChoiceGood forCostFails when
Greedy (argmax)Extraction, classification, deterministic testsNo diversity, can loopThe task accepts several valid answers and variety matters
Temperature samplingBrainstorming, paraphrasingUnstable output, needs seedsOutput feeds a parser or a customer-facing fact
Top-p / top-kTrimming the implausible tailCan cut a valid rare tokenThe distribution is genuinely flat and the tail holds the answer
Raw model probability as confidenceCheap triage signalOften overconfidentUsed as a user-facing risk score without calibration
Temperature scaling (post-hoc)Calibrating an existing modelNeeds held-out labelled dataTraffic distribution shifts after fitting
Trade-off mapInteractive

Good for

Extraction, classification, deterministic tests

Cost

No diversity, can loop

Fails when

The task accepts several valid answers and variety matters

  1. The task accepts several valid answers and variety matters
  2. Output feeds a parser or a customer-facing fact
  3. The distribution is genuinely flat and the tail holds the answer
  4. Used as a user-facing risk score without calibration
  5. Traffic distribution shifts after fitting

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Greedy (argmax) is good for Extraction, classification, deterministic tests, costs No diversity, can loop, and fails when The task accepts several valid answers and variety matters. Temperature sampling is good for Brainstorming, paraphrasing, costs Unstable output, needs seeds, and fails when Output feeds a parser or a customer-facing fact. Top-p / top-k is good for Trimming the implausible tail, costs Can cut a valid rare token, and fails when The distribution is genuinely flat and the tail holds the answer. Raw model probability as confidence is good for Cheap triage signal, costs Often overconfident, and fails when Used as a user-facing risk score without calibration. Temperature scaling (post-hoc) is good for Calibrating an existing model, costs Needs held-out labelled data, and fails when Traffic distribution shifts after fitting

Failure modes

Confusing confidence with correctness. A model reports 0.92 on an answer that is wrong. Debugging step: bucket predictions by reported confidence, compute empirical accuracy per bucket, and plot the two; the gap is the calibration error.

Multiplying correlated evidence. Three retrieved passages from the same document are treated as three independent confirmations, so the combined confidence is far too high. Debugging step: group evidence by source and count independent sources, not passages.

Base-rate neglect. A classifier with strong sensitivity is deployed on a rare condition and floods reviewers with false positives. Debugging step: compute precision at the actual prevalence, not on the balanced test set.

Underflow in a product of probabilities. A sequence score becomes exactly 0.0 after a few hundred tokens. Debugging step: assert probabilities are stored as log-probabilities and that no score is -inf before a comparison.

In production

Log probabilities, not just decisions. For each model call, record the chosen token's log-probability or the sequence log-probability, then monitor the distribution over time. A shift in mean confidence with flat accuracy is an early warning of distribution change.

Calibrate before you gate. If a threshold routes requests to a human, fit the threshold on labelled data from the same distribution, and re-check it after model or prompt changes. Temperature scaling is a cheap post-hoc fix; it needs a held-out set and does not repair a model that is wrong for structural reasons.

Size your evaluations. Comparing two prompts on twenty examples cannot distinguish a real improvement from noise. Compute the uncertainty on the difference before declaring a winner — this is the same probability arithmetic, applied to your own measurements.

Common misconceptions

People think: A model that says 0.95 is right 95% of the time.

Actually: Only if the model is calibrated. Modern neural networks are frequently overconfident, so reported confidence typically exceeds empirical accuracy until it is calibrated on held-out data.

People think: Two events with probabilities can simply be multiplied to combine them.

Actually: Multiplication requires independence. Model calls, retrieved chunks and judge votes are usually correlated, so the product overstates confidence.

A third: that a probability of 0 means impossibility. In a float32 softmax over a large vocabulary, a token can round to zero while remaining a legitimate continuation. That is a numerical artefact, not a statement about the world.

Interview angle

Probability questions probe whether you can separate three things that look similar: a model's score, a calibrated probability, and a decision threshold. Expect to be asked why softmax needs the max-subtraction, what temperature does to a distribution, how to tell whether a confidence score is trustworthy, and how many evaluation examples you need before believing a result. See logits and softmax and the evaluation questions at /interview/evaluation.

Sources