Why it matters
Some problems have no labelled correct action, only an outcome: a game score, a conversion, a completed task. Reinforcement learning optimises that outcome directly by trial, which is powerful when a simulator is cheap and dangerous when it is not. The engineering problem is that the reward is a measurement, not the goal, and any gap between them is exactly what the policy will exploit.
Mental model
Reinforcement learning is training a dog with treats: behaviour that earns rewards gets repeated, and behaviour that does not fades. The metaphor breaks because the trainer's treats are a proxy for what the trainer wants, and a clever animal can learn to earn treats without doing the intended task.
How it works
- 1Formulate the problem as states, actions, transitions and a reward, and check that the reward is measurable at each step.
- 2Choose the algorithm from the action space: value-based for discrete actions, policy gradient for continuous or stochastic policies.
- 3Collect trajectories with an exploration policy and store state, action, reward and next state.
- 4Estimate returns with discounting and, for actor-critic methods, a value baseline to reduce variance.
- 5Update the policy toward actions with higher estimated advantage, and keep a target network or trust region for stability.
- 6Evaluate on the true objective, not the training reward, using a held-out set of episodes or human judgement.
Step 1 of 6
Formulate the problem as states, actions, transitions and a reward, and check that the reward is measurable at each step.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Formulate the problem as states, actions, transitions and a reward, and check that the re… 2. Choose the algorithm from the action space: value-based for discrete actions, policy grad… 3. Collect trajectories with an exploration policy and store state, action, reward and next … 4. Estimate returns with discounting and, for actor-critic methods, a value baseline to redu… 5. Update the policy toward actions with higher estimated advantage, and keep a target netwo… 6. Evaluate on the true objective, not the training reward, using a held-out set of episodes…
- Formulate the problem as states, actions, transitions and a reward, and check that the reward is measurable at each step.
- Choose the algorithm from the action space: value-based for discrete actions, policy gradient for continuous or stochastic policies.
- Collect trajectories with an exploration policy and store state, action, reward and next state.
- Estimate returns with discounting and, for actor-critic methods, a value baseline to reduce variance.
- Update the policy toward actions with higher estimated advantage, and keep a target network or trust region for stability.
- Evaluate on the true objective, not the training reward, using a held-out set of episodes or human judgement.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Value-based methods | Sample-efficient on discrete actions | Unstable with function approximation, poor for continuous control | The action space is large or continuous |
| Policy gradient | Handles continuous and stochastic policies | High variance, needs many samples | Interaction is expensive |
| On-policy learning | Stable, follows the current policy | Discards data after each update | Samples are costly |
| Off-policy learning | Reuses past experience | Divergence risk and correction complexity | The behaviour policy differs too much |
Good for
Sample-efficient on discrete actions
Cost
Unstable with function approximation, poor for continuous control
Fails when
The action space is large or continuous
- The action space is large or continuous
- Interaction is expensive
- Samples are costly
- The behaviour policy differs too much
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Value-based methods is good for Sample-efficient on discrete actions, costs Unstable with function approximation, poor for continuous control, and fails when The action space is large or continuous. Policy gradient is good for Handles continuous and stochastic policies, costs High variance, needs many samples, and fails when Interaction is expensive. On-policy learning is good for Stable, follows the current policy, costs Discards data after each update, and fails when Samples are costly. Off-policy learning is good for Reuses past experience, costs Divergence risk and correction complexity, and fails when The behaviour policy differs too much
Failure modes
- Reward hacking. The policy maximises the measured proxy in a way that violates the intent. Debug: inspect top-reward episodes manually and compare with the outcome the reward was meant to represent.
- Sparse reward with no signal. The agent never reaches a rewarded state and the gradient stays near zero. Debug: measure the fraction of episodes with non-zero return and add shaping or demonstrations if it is near zero.
- Policy collapse. Exploration decays too fast and the policy repeats one action. Debug: track action entropy across training; a monotone fall to near zero indicates premature convergence.
In production
Keep the policy behind a shadow mode and compare its actions against the current system before switching traffic. Monitor reward and the true outcome separately, because a rising reward with a flat outcome is the signature of a proxy that has been gamed.
Common misconceptions
People think: Reinforcement learning is unsupervised learning.
Actually: It uses a reward signal, which is supervision of a different kind; the agent is told how good an outcome was, just not which action was correct.
People think: A well-designed reward removes the need for evaluation.
Actually: The reward is a proxy, and policies optimise proxies harder than their designers expect; independent evaluation is the only check.
Interview angle
A strong answer writes the objective as expected discounted return, explains credit assignment and exploration, and treats reward design as specification. It should describe how to detect reward hacking. Weak answers describe an agent trying actions until it succeeds, without saying how credit is assigned across time or why the reward is not the goal.
Sources
- Sutton & Barto, Reinforcement Learning: An Introduction — the MDP formulation, value functions and exploration.
- Schulman et al., Proximal Policy Optimization Algorithms — a stable policy-gradient method and its trust region.
- Mnih et al., Human-level control through deep reinforcement learning — value-based deep RL and experience replay.