Skip to content
AI Engineer Guide.
seniorReportedcurated

After a DPO run, reviewers say answers got longer and more flattering but not more accurate. What happened, and what do you do?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Your judge model produced the labels. How do you audit it for length bias?
  • Stratify judged pairs by length difference and check whether the win rate for the longer answer exceeds the human rate in each bucket.
  • Swap presentation order on a sample to separate position bias from length bias, and compare the judge against human labels on the same pairs.
  • Correlate judge wins with correctness labels, because a judge that prefers longer answers will look consistent while measuring the wrong property.
Preference win rate is up 8 points. Why is that not sufficient evidence?
  • Win rate is measured on the same preference distribution the policy was trained on, so it can rise while held-out correctness is flat.
  • Report length, refusal rate, format validity and task success beside the win rate, because the failure here was invisible in the headline number.
  • Evaluate on held-out prompts with identical decoding settings, since preference gains can shrink or vanish under greedy decoding.

Red flags

  • Blames model nondeterminism or the base model
  • Only looks at the aggregate win rate and never at dataset statistics

Concepts this tests

Tags

Source: cloudsoftsol.com

More in Fine-tuning & adaptation