seniorReportedcurated
DPO versus PPO-based RLHF: when would you choose each, and what does DPO give up?
30-second answer
- DPO is offline: it fits a fixed preference set with a classification-style loss against a frozen reference, needs two models instead of PPO's four, and has no reward model, critic or generation loop.
- PPO or GRPO is online: it samples fresh responses and scores them, which matters when a verifier or a strong reward model can rank outputs the pair set never contained.
- DPO's honest limit is that it can only reweight behaviour present in its pairs, and its data drifts off-policy as the model improves.
- Iterative DPO is the practical middle ground: regenerate pairs from the current policy between rounds, keeping DPO's simplicity while refreshing coverage.
- Online methods buy a higher ceiling at real operational cost, including sampling infrastructure, reward-model validation and instability that DPO avoids.
Likely follow-ups
Open a question to rehearse the answer.
Why does off-policy data cap DPO's ceiling?
- DPO raises the probability of chosen responses and lowers rejected ones, but it cannot create a behaviour that never appears in either list.
- As the policy moves, the fixed pairs describe an older model, so the gradient increasingly rewards choices the current policy would not make.
- The ceiling is therefore coverage of the pair set, not the size of the model or the number of epochs.
Where does a programmatic verifier beat both?
- Use a verifier when success is checkable, such as unit tests, exact arithmetic, schema validity or a database end state, because it scores fresh samples without a learned proxy.
- GRPO fits this case well: sample several responses per prompt and use the group's relative score as the baseline, so no value model is needed.
- The verifier must be audited with a known-bad solution, or it becomes a reward model with the same hacking surface.
Red flags
- Describes DPO as simply cheaper RLHF with no offline limitation
- Cannot say what beta does or why the reference model exists
Concepts this tests
Tags
Source: github.com