seniorReportedcurated
After a DPO run, reviewers say answers got longer and more flattering but not more accurate. What happened, and what do you do?
30-second answer
- The preference pairs rewarded length and confident tone, and DPO optimised exactly the signal it was given, so the policy learned to be longer and more flattering rather than more correct.
- Diagnose in order: compare mean length of chosen versus rejected responses, audit whether the labelling judge was length-biased, and check beta, since a low beta lets the policy drift far from the SFT reference.
- Rebuild pairs so chosen and rejected differ in correctness but are similar in length, add a rubric that penalises unsupported claims, and raise beta to constrain the drift.
- Move correctness to a verifiable reward where the task allows it, and track response length and refusal rate as guardrail metrics on every run.
- Tighter pairs cost more annotation, and a higher beta limits how far the model can improve, so the fix buys accuracy at the price of headroom.
Likely follow-ups
Open a question to rehearse the answer.
Your judge model produced the labels. How do you audit it for length bias?
- Stratify judged pairs by length difference and check whether the win rate for the longer answer exceeds the human rate in each bucket.
- Swap presentation order on a sample to separate position bias from length bias, and compare the judge against human labels on the same pairs.
- Correlate judge wins with correctness labels, because a judge that prefers longer answers will look consistent while measuring the wrong property.
Preference win rate is up 8 points. Why is that not sufficient evidence?
- Win rate is measured on the same preference distribution the policy was trained on, so it can rise while held-out correctness is flat.
- Report length, refusal rate, format validity and task success beside the win rate, because the failure here was invisible in the headline number.
- Evaluate on held-out prompts with identical decoding settings, since preference gains can shrink or vanish under greedy decoding.
Red flags
- Blames model nondeterminism or the base model
- Only looks at the aggregate win rate and never at dataset statistics
Concepts this tests
Tags
Source: cloudsoftsol.com