Why it matters
Collecting preferences is one problem; turning them into a policy update is another. PPO needs a reward model, a value function and a sampling loop, which is powerful and fragile. DPO showed that a similar preference objective can be written as a classification loss on the policy itself, removing the reward model and the sampling. The engineering choice is between control and operational simplicity.
Mental model
The difference is between tuning a radio by ear while a station plays, and adjusting it from a recording of what you liked. The analogy breaks because the recording goes stale: preferences collected on an old policy may not describe the new one.
How it works
Sorted by sampled win rate, descending. The pair of bars in each row is the same policy under two decoding settings.
Hover a row for its exact values. Every bar prints its number, and the values table lists all of them.
- Held-out prompts
- 200
- Best sampled
- 65%
- Greedy shrink
- 6–7 pts
PPO wins 65% of sampled comparisons. Online methods can sample from the current policy and score those samples, so part of this lead is exploration the offline pair set never saw.
Values
| category | Sampled decoding | Greedy decoding |
|---|---|---|
| PPO · 200 prompts · online | 65% | 58% |
| GRPO · 200 prompts · online | 63% | 57% |
| DPO · 200 prompts · offline | 62% | 56% |
| DPO, stale pairs · 200 prompts · pairs from the old policy | 58% | 51% |
Reference rule: tie with checkpoint at 50%.
Illustrative win rates over the supervised checkpoint, 200 held-out prompts per run, ties counted as half a win. The figure omits the reference-model divergence, the beta sweep and the length and refusal rates the page asks to report beside a win rate.
Figure. Win rate of three methods — four runs — over their own supervised checkpoint, measured once with sampled decoding and once with greedy decoding. The dashed rule is 50%, a tie with the checkpoint. Illustrative rates over 200 held-out prompts; they are authored to show the direction of the effect, not a benchmark.
What to notice. Every method loses 6 to 7 points when decoding becomes greedy, which is why the page says to compare against the baseline with identical decoding settings. DPO's stale-pair run is the failure case: it wins 58% when sampling and 51% when greedy, so the reported gain is mostly decoding variance. The figure cannot show the divergence from the reference or the beta sweep the page asks for, and it does not claim PPO is better in general — only that the online methods sampled more gains here.
Text version
A grouped horizontal bar chart of win rate against a supervised checkpoint, four runs, two decodings each. A dashed rule at 50% marks a tie. PPO wins 65% sampled and 58% greedy. GRPO wins 63% and 57%. DPO wins 62% and 56%. DPO trained on pairs from an old policy wins 58% and 51%, the greedy bar sitting almost on the rule. The accent bar is sampled decoding and the teal bar greedy. The card reports 200 held-out prompts, a best sampled rate of 65% and a greedy shrink of 6 to 7 points. Steps highlight PPO, GRPO, DPO, the stale-pair run, then the whole chart.
- Start from a supervised model that already produces responses in the right format, since preference methods refine rather than teach format.
- Choose online or offline: online if you can afford sampling and scoring during training, offline if the pair set is fixed.
- For DPO, keep the reference model frozen and set the beta term that controls how far the policy may move.
- For PPO, train the value function and reward model first and verify each separately before the loop runs.
- Evaluate on held-out prompts with human or judge preference, and report length and refusal rate beside win rate.
- Compare against the supervised baseline with identical decoding settings, because preference gains often shrink under greedy decoding.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| DPO | Simple, stable, no reward model | Offline only, sensitive to beta and data coverage | The policy must explore beyond the pair set |
| PPO | Online improvement and control | Infrastructure and instability | You cannot operate a sampling loop |
| GRPO | No value model, group-relative baseline | Needs many samples per prompt | The sample budget is small |
| Single-label variants | Works with thumbs up and down | Weaker signal per example | You can collect clean pairs |
Good for
Simple, stable, no reward model
Cost
Offline only, sensitive to beta and data coverage
Fails when
The policy must explore beyond the pair set
- The policy must explore beyond the pair set
- You cannot operate a sampling loop
- The sample budget is small
- You can collect clean pairs
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: DPO is good for Simple, stable, no reward model, costs Offline only, sensitive to beta and data coverage, and fails when The policy must explore beyond the pair set. PPO is good for Online improvement and control, costs Infrastructure and instability, and fails when You cannot operate a sampling loop. GRPO is good for No value model, group-relative baseline, costs Needs many samples per prompt, and fails when The sample budget is small. Single-label variants is good for Works with thumbs up and down, costs Weaker signal per example, and fails when You can collect clean pairs
Failure modes
- Reference model drift. The reference is updated or reloaded differently and the implicit reward changes scale. Debug: hash the reference weights and assert they are frozen across the run.
- Beta set by guesswork. The policy either barely moves or collapses to one style. Debug: sweep beta on a small run and plot win rate against divergence from the reference.
- Pair set from another model. Win rate improves offline and human preference does not. Debug: regenerate pairs with the current policy and re-evaluate.
In production
Log the implicit divergence or reward margin per step and stop on held-out preference, not training loss. Keep the supervised checkpoint deployable, since preference tuning can reduce output diversity.
Common misconceptions
People think: DPO is a different objective from RLHF.
Actually: It is a reparameterisation of a preference objective under a particular model of the reward.
People think: Offline preference tuning cannot overfit.
Actually: It overfits the pair set and the reference, and the symptom is a policy that wins comparisons and reads worse.
Interview angle
Put the objective in terms of the reference model and the beta term, and say what each family needs operationally. A strong answer notes that offline methods cannot explore. Weak answers treat DPO as a drop-in upgrade to PPO without mentioning data coverage or the reference model.
Sources
- Rafailov et al., Direct Preference Optimization — the offline objective that removes the explicit reward model.
- Schulman et al., Proximal Policy Optimization Algorithms — the online optimiser used in PPO-based RLHF.
- Shao et al., DeepSeekMath — group-relative policy optimisation without a value model.