Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 11 of 22

Preference data

0 of 22 complete

Why it matters

Some qualities are easy to recognise and hard to specify: a helpful answer, a safe refusal, a well-structured explanation. Asking annotators to write the ideal response produces inconsistent targets; asking which of two they prefer produces a consistent, low-effort signal. The engineering work is in prompt sampling, rater management and preserving the disagreement that shows where the task is genuinely ambiguous.

Mental model

Preference data is a taste test rather than a recipe: tasters say which sample they like, not how to cook it. The analogy breaks because tasters can be inconsistent and their preferences can reflect the order they tasted in, so the comparison needs design.

How it works

Win rate by category · 40 pairs eachInteractive
Chosen response shown firstChosen response shown second

Sorted by overall win rate across both orders. The gap between the two bars in a row is the position effect; the bar height is not the claim.

Hover a row for its exact values. Every bar prints its number, and the values table lists all of them.

Pairs
200
Overall chosen
55%
Widest order gap
25 pts
Step 1/4 · One category, two orders

Tone wins 71% when the chosen response is shown first and 46% when it is shown second. Both numbers are the same judge under the same protocol, so the 25-point gap measures presentation, not preference.

Values
Printed values for every category, in the drawn order. The axis runs from 0 to 80.
categoryChosen response shown firstChosen response shown second
Tone · 40 pairs · 8% ties71%46%
Coding · 40 pairs · 4% ties61%54%
Refusals · 40 pairs · 2% ties58%56%
Formatting · 40 pairs · 10% ties55%52%
Factual · 40 pairs · 22% ties52%49%

Reference rule: no preference (50%) at 50%.

Illustrative win rates, not a measurement: 40 pairs per category shown in both orders, chosen response wins, ties counted separately. At n = 40 the 95% interval on a win rate is about ±15 points, so only the largest gaps here are distinguishable from chance. The 55% overall rate is the mean of the ten order-specific rates over 400 judgements, not a count over 200 pairs.

Figure. Win rate of the chosen response, split by the order the pair was shown in, over five prompt categories. Each of the 40 pairs per category was shown in both orders, so a bar is 40 judgements from that pair set with ties excluded — the two bars are not halves of the same 40. The dashed rule is 50%, where the two responses are indistinguishable. Illustrative counts: 200 pairs, 400 judgements; the intervals are wide at that size.

What to notice. Read the gap inside a row, not just the bar length. Tone wins 71% of the time when the chosen response is shown first and 46% when it is second — a 25-point swing that is evidence of a position effect, not of quality. Factual sits at 52/49 with 22% ties: at 40 pairs that is indistinguishable from chance, so the category should be dropped or re-collected rather than trained on. The categories are sorted by overall win rate, which hides exactly the disagreement the protocol is meant to expose, so the values table prints the tie rate beside each one.

Text version

A grouped horizontal bar chart, win rate of the chosen response in percent, five prompt categories sorted by overall win rate. Each category has two bars, chosen shown first in accent and chosen shown second in teal, and every bar prints its percentage. A dashed rule at 50% marks no preference. Tone: 71% first, 46% second. Coding: 61% and 54%. Refusals: 58% and 56%. Formatting: 55% and 52%. Factual: 52% and 49%. The tie rates run from 2% for refusals to 22% for factual. The card reports 200 pairs, a 55% overall win rate and a 25-point widest order gap. Three step buttons highlight Tone, then Factual, then Coding, and a fourth returns to the whole chart.

  1. Sample prompts from the traffic distribution you care about, including hard and ambiguous cases rather than only clean ones.
  2. Generate candidate pairs from models of similar strength, since a pair where one is obviously better teaches little.
  3. Randomise presentation order and include repeated pairs to measure position bias and rater consistency.
  4. Record ties and rationales, and keep disagreement rather than forcing a winner.
  5. Measure inter-rater agreement per prompt category and drop categories where agreement is near chance.
  6. Hold out prompts rather than pairs, so the same prompt does not appear in training and evaluation with different responses.

Trade-offs

ChoiceGood forCostFails when
Pairwise human labelsReliable and cheap per judgementSlow for large volumesYou need coverage across many domains
Model-judged preferencesScalable and fastInherits judge biases and self-preferenceJudgements must be defensible
Forced choice with no tiesSimple training signalInvents preferences where raters are splitPrompts are genuinely ambiguous
Ties preservedHonest about ambiguityNeeds a loss that handles tiesThe training framework expects pairs only
Trade-off mapInteractive

Good for

Reliable and cheap per judgement

Cost

Slow for large volumes

Fails when

You need coverage across many domains

  1. You need coverage across many domains
  2. Judgements must be defensible
  3. Prompts are genuinely ambiguous
  4. The training framework expects pairs only

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Pairwise human labels is good for Reliable and cheap per judgement, costs Slow for large volumes, and fails when You need coverage across many domains. Model-judged preferences is good for Scalable and fast, costs Inherits judge biases and self-preference, and fails when Judgements must be defensible. Forced choice with no ties is good for Simple training signal, costs Invents preferences where raters are split, and fails when Prompts are genuinely ambiguous. Ties preserved is good for Honest about ambiguity, costs Needs a loss that handles ties, and fails when The training framework expects pairs only

Failure modes

  • Position bias. The preferred response is whichever appeared first. Debug: compare win rates by presentation order and check that they are close to balanced.
  • Prompt leakage across splits. Evaluation pairs come from prompts seen in training. Debug: group by prompt id and assert the sets are disjoint.
  • Rater collapse. One annotator supplies most labels and their style dominates. Debug: compute per-annotator label share and agreement against the majority.

In production

Track agreement and tie rate per batch as leading indicators of data quality. Re-audit when the candidate models change, because preferences are relative to the pair that was shown.

Common misconceptions

People think: Preference data is objective ground truth.

Actually: It records what a particular group preferred under a particular protocol, including their biases.

People think: More pairs always improve the reward model.

Actually: Coverage of prompt types matters more, and near-duplicate pairs add little.

Interview angle

Cover prompt sampling, pair construction and the statistics that expose bias: order randomisation, tie rate, inter-rater agreement. Explain why prompts are the split unit. Weak answers describe collecting thumbs up and down without saying how they would detect position bias or annotator drift.

Sources