Skip to content
AI Engineer Guide.
seniorReportedcurated

DPO versus PPO-based RLHF: when would you choose each, and what does DPO give up?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why does off-policy data cap DPO's ceiling?
  • DPO raises the probability of chosen responses and lowers rejected ones, but it cannot create a behaviour that never appears in either list.
  • As the policy moves, the fixed pairs describe an older model, so the gradient increasingly rewards choices the current policy would not make.
  • The ceiling is therefore coverage of the pair set, not the size of the model or the number of epochs.
Where does a programmatic verifier beat both?
  • Use a verifier when success is checkable, such as unit tests, exact arithmetic, schema validity or a database end state, because it scores fresh samples without a learned proxy.
  • GRPO fits this case well: sample several responses per prompt and use the group's relative score as the baseline, so no value model is needed.
  • The verifier must be audited with a known-bad solution, or it becomes a reward model with the same hacking surface.

Red flags

  • Describes DPO as simply cheaper RLHF with no offline limitation
  • Cannot say what beta does or why the reference model exists

Concepts this tests

Tags

Source: github.com

More in Fine-tuning & adaptation