Why it matters
Supervised tuning needs a target response for every prompt, which is expensive and often impossible for qualities like tone or safety. RLHF replaces the target with a comparison: a reward model scores responses, and the policy is trained to score higher. The engineering difficulty is that the reward is a learned approximation, so the policy will find its weaknesses unless it is held close to the model the reward was fitted on.
Mental model
RLHF is training a dog with a judge who scores each trick: the dog optimises the judge, not the intention behind the rubric. The analogy breaks because you can retrain the judge, and the KL penalty keeps the dog from abandoning everything it already knew.
How it works
Step 1 of 6
Start from SFT
The policy begins at the supervised checkpoint. The reference is a frozen copy of that same checkpoint, so the leash starts at zero distance.
Active layers
- Prompt batch
- Policy · trainable
- Policy πθ
Hover a layer to read what it holds.
Next: Sample from the policy
- Policy
- θ trainable
- Reference
- frozen · KL leash
- Reward
- frozen · proxy
Figure. Where the reward and the reference attach. The policy is the only trainable model; the reward scores a sampled response, and the reference supplies token log-probabilities over the same prompt-response pairs for the KL term. The two frozen models enter the objective from different directions and neither is updated. Preference-data collection and the policy-gradient clipping details are omitted; beta is illustrative.
What to notice. The reference is not a second reward: it anchors the policy to the checkpoint the reward was fitted around, which is what makes exploiting the scorer cost something. Only the policy parameters receive gradient, so reward rising while held-out preference falls is the failure this arrangement exposes. Stop on preference, not on reward.
Text version
A vertical stack. A prompt batch feeds a trainable policy group holding the policy and its sampled responses. The responses enter a frozen reward model, whose score enters the objective; the prompt reaches a frozen reference model along a dashed lane and the sampled response tokens reach it along a dotted cross lane, so its log-probabilities are over the same sequence as the policy's, which also enter the objective. The objective feeds a policy-only update, which writes a checkpoint and returns to the policy along a dash-dot lane. Six steps cover the supervised start, sampling, scoring, the reference anchor, the combined update, and stopping.
- Collect demonstrations and run supervised fine-tuning so the policy starts in the region the reward model will cover.
- Gather pairwise preferences on outputs from that policy, then train a reward model with a margin loss on chosen versus rejected.
- Hold out prompts and check reward-model accuracy on them, since a scorer that cannot rank held-out pairs gives noise.
- Optimise the policy with a policy-gradient method against the reward, with a per-token KL penalty to the frozen reference.
- Tune the KL coefficient and stop on held-out preference evaluation rather than on reward, which rises while quality falls.
- Compare against the supervised baseline on a fixed prompt set, because RLHF can win on reward and lose on readability.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Strong KL constraint | Stable, close to the reference | Limited improvement | The base behaviour is the problem |
| Weak KL constraint | Large behaviour change | Reward hacking and mode collapse | The reward model is imperfect |
| PPO | Well understood and flexible | Many moving parts and memory | You cannot staff the infrastructure |
| Direct preference optimisation | Simpler, no reward model | Less control and no reusable scorer | You need a reward signal at inference |
Good for
Stable, close to the reference
Cost
Limited improvement
Fails when
The base behaviour is the problem
- The base behaviour is the problem
- The reward model is imperfect
- You cannot staff the infrastructure
- You need a reward signal at inference
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Strong KL constraint is good for Stable, close to the reference, costs Limited improvement, and fails when The base behaviour is the problem. Weak KL constraint is good for Large behaviour change, costs Reward hacking and mode collapse, and fails when The reward model is imperfect. PPO is good for Well understood and flexible, costs Many moving parts and memory, and fails when You cannot staff the infrastructure. Direct preference optimisation is good for Simpler, no reward model, costs Less control and no reusable scorer, and fails when You need a reward signal at inference
Failure modes
- Reward hacking. Reward rises while human preference falls, often as verbosity or flattery. Debug: plot reward against held-out human preference and inspect the highest-reward outputs.
- KL collapse. The policy drifts far from the reference and produces degenerate text. Debug: log mean KL per batch and stop when it exceeds the range seen in a healthy run.
- Reward model overfit. The scorer separates training pairs perfectly and held-out pairs poorly. Debug: compare accuracy on train and held-out prompt splits.
In production
Keep the reference model and reward model versioned with the policy, and cap optimisation steps per reward model. Monitor length and refusal rate, which move early when the policy starts gaming the scorer.
Common misconceptions
People think: RLHF makes a model truthful.
Actually: It optimises for the preferences that were collected, which can reward confidence over correctness.
People think: The reward score is a quality metric.
Actually: It is a proxy fitted to comparisons, and it is only meaningful inside the region it was trained on.
Interview angle
Lay out the three stages and the role of the KL term, then explain why reward is not the objective you report. Mention held-out pair accuracy for the reward model. Candidates who describe RLHF as reinforcement learning with people scoring live, rather than from a trained scorer, have the pipeline wrong.
Sources
- Christiano et al., Deep Reinforcement Learning from Human Preferences — the reward-from-comparisons formulation.
- Ouyang et al., Training language models to follow instructions with human feedback — the three-stage SFT, reward model and PPO pipeline.
- Stiennon et al., Learning to Summarize from Human Feedback — KL-constrained optimisation against a learned reward.