Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 12 of 22

RLHF

0 of 22 complete

Why it matters

Supervised tuning needs a target response for every prompt, which is expensive and often impossible for qualities like tone or safety. RLHF replaces the target with a comparison: a reward model scores responses, and the policy is trained to score higher. The engineering difficulty is that the reward is a learned approximation, so the policy will find its weaknesses unless it is held close to the model the reward was fitted on.

Mental model

RLHF is training a dog with a judge who scores each trick: the dog optimises the judge, not the intention behind the rubric. The analogy breaks because you can retrain the judge, and the KL penalty keeps the dog from abandoning everything it already knew.

How it works

Reward scores the sample; the reference holds the policyInteractive
signalbypasscrossloopfrozen

Step 1 of 6

Start from SFT

The policy begins at the supervised checkpoint. The reference is a frozen copy of that same checkpoint, so the leash starts at zero distance.

Active layers

  • Prompt batch
  • Policy · trainable
  • Policy πθ

Hover a layer to read what it holds.

Next: Sample from the policy

Policy
θ trainable
Reference
frozen · KL leash
Reward
frozen · proxy
Step 1/6 · Start from SFT

Figure. Where the reward and the reference attach. The policy is the only trainable model; the reward scores a sampled response, and the reference supplies token log-probabilities over the same prompt-response pairs for the KL term. The two frozen models enter the objective from different directions and neither is updated. Preference-data collection and the policy-gradient clipping details are omitted; beta is illustrative.

What to notice. The reference is not a second reward: it anchors the policy to the checkpoint the reward was fitted around, which is what makes exploiting the scorer cost something. Only the policy parameters receive gradient, so reward rising while held-out preference falls is the failure this arrangement exposes. Stop on preference, not on reward.

Text version

A vertical stack. A prompt batch feeds a trainable policy group holding the policy and its sampled responses. The responses enter a frozen reward model, whose score enters the objective; the prompt reaches a frozen reference model along a dashed lane and the sampled response tokens reach it along a dotted cross lane, so its log-probabilities are over the same sequence as the policy's, which also enter the objective. The objective feeds a policy-only update, which writes a checkpoint and returns to the policy along a dash-dot lane. Six steps cover the supervised start, sampling, scoring, the reference anchor, the combined update, and stopping.

  1. Collect demonstrations and run supervised fine-tuning so the policy starts in the region the reward model will cover.
  2. Gather pairwise preferences on outputs from that policy, then train a reward model with a margin loss on chosen versus rejected.
  3. Hold out prompts and check reward-model accuracy on them, since a scorer that cannot rank held-out pairs gives noise.
  4. Optimise the policy with a policy-gradient method against the reward, with a per-token KL penalty to the frozen reference.
  5. Tune the KL coefficient and stop on held-out preference evaluation rather than on reward, which rises while quality falls.
  6. Compare against the supervised baseline on a fixed prompt set, because RLHF can win on reward and lose on readability.

Trade-offs

ChoiceGood forCostFails when
Strong KL constraintStable, close to the referenceLimited improvementThe base behaviour is the problem
Weak KL constraintLarge behaviour changeReward hacking and mode collapseThe reward model is imperfect
PPOWell understood and flexibleMany moving parts and memoryYou cannot staff the infrastructure
Direct preference optimisationSimpler, no reward modelLess control and no reusable scorerYou need a reward signal at inference
Trade-off mapInteractive

Good for

Stable, close to the reference

Cost

Limited improvement

Fails when

The base behaviour is the problem

  1. The base behaviour is the problem
  2. The reward model is imperfect
  3. You cannot staff the infrastructure
  4. You need a reward signal at inference

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Strong KL constraint is good for Stable, close to the reference, costs Limited improvement, and fails when The base behaviour is the problem. Weak KL constraint is good for Large behaviour change, costs Reward hacking and mode collapse, and fails when The reward model is imperfect. PPO is good for Well understood and flexible, costs Many moving parts and memory, and fails when You cannot staff the infrastructure. Direct preference optimisation is good for Simpler, no reward model, costs Less control and no reusable scorer, and fails when You need a reward signal at inference

Failure modes

  • Reward hacking. Reward rises while human preference falls, often as verbosity or flattery. Debug: plot reward against held-out human preference and inspect the highest-reward outputs.
  • KL collapse. The policy drifts far from the reference and produces degenerate text. Debug: log mean KL per batch and stop when it exceeds the range seen in a healthy run.
  • Reward model overfit. The scorer separates training pairs perfectly and held-out pairs poorly. Debug: compare accuracy on train and held-out prompt splits.

In production

Keep the reference model and reward model versioned with the policy, and cap optimisation steps per reward model. Monitor length and refusal rate, which move early when the policy starts gaming the scorer.

Common misconceptions

People think: RLHF makes a model truthful.

Actually: It optimises for the preferences that were collected, which can reward confidence over correctness.

People think: The reward score is a quality metric.

Actually: It is a proxy fitted to comparisons, and it is only meaningful inside the region it was trained on.

Interview angle

Lay out the three stages and the role of the KL term, then explain why reward is not the objective you report. Mention held-out pair accuracy for the reward model. Candidates who describe RLHF as reinforcement learning with people scoring live, rather than from a trained scorer, have the pipeline wrong.

Sources