Why it matters
Human judgement is the signal you want and the one you cannot afford per training step. A reward model compresses many comparisons into a function that can score thousands of candidate responses. The compression is lossy: it learns the features that separated the pairs it saw, which may be length, formatting or confidence rather than correctness. Every later decision depends on knowing where that approximation holds.
Mental model
A reward model is a food critic who has only eaten in one city: useful there, unreliable elsewhere, and confident either way. The analogy breaks because the critic also trains the chefs, so the menu shifts toward whatever the critic already likes rather than toward what diners actually want.
How it works
Sorted by score, descending — the ranking is what the scorer is trained to produce, so the sort is the output.
Hover a row for its exact values. Every bar prints its number, and the values table lists all of them.
- Candidates
- 7
- Top score
- 2.31
- Training maximum
- 1.85
A ranks first at 2.31 and G last at 0.98. The differences order the candidates; they do not measure quality in any unit, and 2.31 does not mean twice 1.15.
Values
| category | Reward score |
|---|---|
| Candidate A · 412 tokens · two unsupported claims · 412 tokens | 2.31 |
| Candidate B · 388 tokens · padded restatement · 388 tokens | 2.14 |
| Candidate C · 96 tokens · direct answer · 96 tokens | 1.78 |
| Candidate D · 214 tokens · hedged answer · 214 tokens | 1.62 |
| Candidate E · 152 tokens · answer plus caveat · 152 tokens | 1.44 |
| Candidate F · 121 tokens · short answer · 121 tokens | 1.21 |
| Candidate G · 78 tokens · terse answer · 78 tokens | 0.98 |
Reference rule: top of training score range · 1.85 at 1.85.
Illustrative: seven authored candidates and scores, written to show ranking, length exploitation and out-of-range extrapolation. The correlation between length and score is constructed, not measured. Calibration across score bands must come from held-out prompts against human comparisons.
Figure. Reward-model scores for seven candidate responses to the same prompt, sorted by score. The dashed rule is 1.85, the highest score in the scorer's training data; the two bars above it are extrapolations, and they are also the two longest candidates (412 and 388 tokens). The candidate set and its scores are authored to show two failure modes, not measured from a trained scorer.
What to notice. The output is a ranking, and the scale around it has no absolute meaning. The top two candidates are also the two longest — 412 and 388 tokens against 78 for the terse one — which is the length exploitation the page warns about. Both sit above the training range, where the scorer has no pairs to fit and the number is an extrapolation wearing the same format as a prediction. Nothing here shows calibration; that has to be measured on held-out prompts, per score band.
Text version
A horizontal bar chart of reward scores for seven candidates to one prompt, sorted descending. Values print beside each bar and a dashed rule at 1.85 marks the top of the training score range. Candidate A, 412 tokens, scores 2.31; B, 388 tokens, 2.14; C, 96 tokens, 1.78; D, 214 tokens, 1.62; E, 152 tokens, 1.44; F, 121 tokens, 1.21; G, 78 tokens, 0.98. A and B are above the rule. The card reports seven candidates, a top score of 2.31 and a training maximum of 1.85. Steps highlight A for the ranking, A and B for length, then A and B for the extrapolation. The values table lists every candidate with its token count.
- Train on pairs from the policy you will optimise, since a scorer fitted to another model's outputs ranks the wrong region.
- Use a margin or Bradley-Terry style loss on chosen versus rejected, with both sides answering the same prompt.
- Evaluate ranking accuracy on held-out prompts and report it per prompt category, not as one average.
- Probe for spurious features by testing pairs that differ only in length or formatting.
- Report calibration across score ranges, because the policy will push into the high-score tail where data is thin.
- Freeze the scorer and record its revision, so a policy can be traced to the model it was optimised against.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Single reward model | Simple and cheap to serve | Exploitable and overconfident in the tail | Optimisation pressure is high |
| Ensemble | Reduces exploitation of one scorer | Cost and disagreement to resolve | Latency or memory is constrained |
| Human spot checks | Ground truth on the target quality | Slow and costly | You need per-step feedback |
| Rule-based rewards | Deterministic and auditable | Narrow and easy to game | Quality is stylistic |
Good for
Simple and cheap to serve
Cost
Exploitable and overconfident in the tail
Fails when
Optimisation pressure is high
- Optimisation pressure is high
- Latency or memory is constrained
- You need per-step feedback
- Quality is stylistic
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Single reward model is good for Simple and cheap to serve, costs Exploitable and overconfident in the tail, and fails when Optimisation pressure is high. Ensemble is good for Reduces exploitation of one scorer, costs Cost and disagreement to resolve, and fails when Latency or memory is constrained. Human spot checks is good for Ground truth on the target quality, costs Slow and costly, and fails when You need per-step feedback. Rule-based rewards is good for Deterministic and auditable, costs Narrow and easy to game, and fails when Quality is stylistic
Failure modes
- Length exploitation. Reward climbs with response length regardless of content. Debug: correlate reward with token count on a fixed prompt set and inspect the longest outputs.
- Distribution mismatch. The scorer was trained on another model's outputs and ranks the current policy arbitrarily. Debug: measure accuracy on pairs generated by the policy you are training.
- Tail overconfidence. Scores beyond the training range are extrapolations but look authoritative. Debug: plot the score distribution against the training distribution and flag out-of-range values.
In production
Version the reward model with every policy it scores, and re-measure agreement with humans on a fixed sample each cycle. Cap optimisation steps per scorer, since its value decays under pressure.
Common misconceptions
People think: A reward model measures quality.
Actually: It predicts preference within its training distribution; outside it, the score is an extrapolation.
People think: Higher reward means a better model.
Actually: Reward is optimised directly, so it rises even when quality falls.
Interview angle
Describe the loss, the data dependency and the over-optimisation curve. A strong answer says how they would detect a spurious feature such as length. Weak answers treat the reward model as a general quality judge and never mention that the policy is trained against it.
Sources
- Stiennon et al., Learning to Summarize from Human Feedback — reward modelling from pairwise comparisons.
- Gao et al., Scaling Laws for Reward Model Overoptimization — the curve where proxy reward rises and true quality falls.
- Ouyang et al., Training language models to follow instructions with human feedback — a reward model used inside a policy optimisation loop.