Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 11 of 28

Logits & softmax

0 of 28 complete

Why it matters

Applications rarely see logits, but every sampling parameter, classifier head and confidence estimate acts on them. Understanding that the model produces a score per vocabulary item explains why temperature changes diversity rather than correctness, why top-p and top-k interact, and why a probability of 0.9 on a token does not mean the sentence is 90 percent likely to be right. Log-probabilities are the only calibrated signal a model exposes.

Mental model

Softmax is a room of candidates where each score becomes a share of a fixed budget of attention, and the loudest voice takes most of it. The metaphor breaks because the budget has no fixed meaning: raising every score equally changes nothing, while raising one takes share from all the others.

How it works

One softmax, five temperaturesInteractive
Temperature
Probability

Sorted by logit, descending. The order is identical at every temperature; only the shares move.

T = 0.2: top token 0.981 on " the", bottom under 0.001 on " zy", entropy 0.10 nats.

Top token
0.981
Entropy
0.10 nats
Printed total
1.000
Step 1/3 · T = 0.2: almost an argmax

At T = 0.2 the top token holds 0.981 and the other seven share 0.019. The order is identical to T = 1; only the shares moved.

Values
Printed values for every category, in the drawn order. The axis runs from 0 to 1. Current frame: Temperature T 0.2.
categoryProbability
token " the" · logit 4.20.981
token " a" · logit 3.40.018
token " of" · logit 2.90.001
token " to" · logit 2.10.000
token " and" · logit 1.60.000
token " in" · logit 0.70.000
token " bec" · logit 0.20.000
token " zy" · logit -1.40.000

Reference rule: uniform over these 8 · 0.125 at 0.125.

The eight logits are authored; each probability is the exact softmax of those logits at the stated temperature, rounded by largest remainder so the printed shares sum to 1.000. Values are renormalised over the eight candidates shown, not over the vocabulary; a share below 0.0005 prints as 0.000, which at T = 0.2 is rounding, not impossibility. The figure omits the biases, penalties and truncation that would be applied after this step.

Figure. Eight next-token candidates, sorted by logit, with the softmax probability at five temperatures on a fixed 0-to-1 axis. The steps walk T = 0.2, T = 2 and back to T = 1; the control selects any of the five. The dashed rule is 1/8, a uniform share across the eight candidates shown. Probabilities are the exact softmax of the eight authored logits renormalised over that set, not over a full vocabulary.

What to notice. Only the shares move: the bar order is identical at T = 0.2 and T = 2. At T = 0.2 the top token holds almost everything and the rest is unreachable in practice; at T = 2 the same eight tokens are all plausible. Because the denominator spans every token in the vocabulary, renormalising over eight candidates overstates each share — the figure is a worked example of the arithmetic, which is why the axis and the uniform rule are printed rather than the numbers alone.

Text version

A horizontal bar chart of eight next-token candidates, the, a, of, to, and, in, bec, zy, sorted by logit from 4.2 down to minus 1.4. Every bar prints its probability to three decimals. A dashed rule at 0.125 marks an even share across eight candidates. Five buttons set the temperature to 0.2, 0.7, 1, 1.5 or 2; the default is 1, where the probabilities run 0.508, 0.228, 0.138, 0.062, 0.038, 0.015, 0.009 and 0.002. At 0.2 the top token reaches 0.981; at 2 the eight shares flatten to 0.314 at the top and 0.019 at the bottom. The card reports the top probability, the entropy in nats and the printed total, all of which change with the control. Three steps set the temperature to 0.2, then 2, then 1, highlighting the top three tokens, the three tail tokens, and then the whole chart with the uniform rule.

  1. Take the final hidden state and multiply by the output matrix to get one logit per vocabulary entry.
  2. Subtract the maximum logit before exponentiating, which keeps the computation stable without changing the result.
  3. Exponentiate and divide by the sum to obtain probabilities that add to one.
  4. Apply temperature by dividing the logits before the softmax, then truncate with top-k or top-p if configured.
  5. Read the chosen token's log-probability when you need a confidence signal, remembering it is conditional on the prefix.
  6. Compare distributions across models only after checking their vocabularies, since the same token id means different strings.

Trade-offs

ChoiceGood forCostFails when
Raw logitsNumerically stable for rankingNot comparable across requests or modelsYou need a probability or threshold
Softmax probabilitiesInterpretable per-step distributionConditioned on one prefix, not on truthYou want document-level confidence
Log-probability of the chosen tokenCheap confidence proxyOverconfident on familiar phrasingThe task needs calibrated risk
Temperature scalingSmooth control of diversityDistorts probabilities away from the model's estimateYou report confidence downstream
Trade-off mapInteractive

Good for

Numerically stable for ranking

Cost

Not comparable across requests or models

Fails when

You need a probability or threshold

  1. You need a probability or threshold
  2. You want document-level confidence
  3. The task needs calibrated risk
  4. You report confidence downstream

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Raw logits is good for Numerically stable for ranking, costs Not comparable across requests or models, and fails when You need a probability or threshold. Softmax probabilities is good for Interpretable per-step distribution, costs Conditioned on one prefix, not on truth, and fails when You want document-level confidence. Log-probability of the chosen token is good for Cheap confidence proxy, costs Overconfident on familiar phrasing, and fails when The task needs calibrated risk. Temperature scaling is good for Smooth control of diversity, costs Distorts probabilities away from the model's estimate, and fails when You report confidence downstream

Failure modes

  • Vocabulary mismatch. Token ids from two models are compared and the strings differ. Debug: decode the ids with each model's tokeniser before comparing.
  • Overflow in naive softmax. Large logits produce infinities and NaN probabilities. Debug: check for the max-subtraction step and assert finite outputs.
  • Confidence read as correctness. A high-probability token is treated as a fact. Debug: calibrate log-probabilities against labelled outcomes before using them as a threshold.

In production

If you expose confidence, calibrate it on a labelled set and monitor its distribution per model version. Cap temperature and truncation in configuration so a client cannot widen the distribution by accident.

Common misconceptions

People think: A probability of 0.95 means the answer is 95 percent likely to be correct.

Actually: It means the token is likely given the prefix and the model's training, not that the claim is true.

People think: Temperature changes which token is most likely.

Actually: It rescales the distribution; the ranking is unchanged, though sampling can pick a different token.

Interview angle

Walk from hidden state to logits to probabilities, and keep the distinction between likelihood and truth explicit. A strong answer connects temperature, top-p and log-probability thresholds to that pipeline. A weak one says softmax normalises numbers and cannot explain why a confident token is not evidence.

Sources