Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 27 of 27

Reinforcement learning

0 of 27 complete

Why it matters

Some problems have no labelled correct action, only an outcome: a game score, a conversion, a completed task. Reinforcement learning optimises that outcome directly by trial, which is powerful when a simulator is cheap and dangerous when it is not. The engineering problem is that the reward is a measurement, not the goal, and any gap between them is exactly what the policy will exploit.

Mental model

Reinforcement learning is training a dog with treats: behaviour that earns rewards gets repeated, and behaviour that does not fades. The metaphor breaks because the trainer's treats are a proxy for what the trainer wants, and a clever animal can learn to earn treats without doing the intended task.

How it works

Mechanism, step by stepInteractive
  1. 1Formulate the problem as states, actions, transitions and a reward, and check that the reward is measurable at each step.
  2. 2Choose the algorithm from the action space: value-based for discrete actions, policy gradient for continuous or stochastic policies.
  3. 3Collect trajectories with an exploration policy and store state, action, reward and next state.
  4. 4Estimate returns with discounting and, for actor-critic methods, a value baseline to reduce variance.
  5. 5Update the policy toward actions with higher estimated advantage, and keep a target network or trust region for stability.
  6. 6Evaluate on the true objective, not the training reward, using a held-out set of episodes or human judgement.

Step 1 of 6

Formulate the problem as states, actions, transitions and a reward, and check that the reward is measurable at each step.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Formulate the problem as states, actions, transitions and a reward, and check that the re… 2. Choose the algorithm from the action space: value-based for discrete actions, policy grad… 3. Collect trajectories with an exploration policy and store state, action, reward and next … 4. Estimate returns with discounting and, for actor-critic methods, a value baseline to redu… 5. Update the policy toward actions with higher estimated advantage, and keep a target netwo… 6. Evaluate on the true objective, not the training reward, using a held-out set of episodes…

  1. Formulate the problem as states, actions, transitions and a reward, and check that the reward is measurable at each step.
  2. Choose the algorithm from the action space: value-based for discrete actions, policy gradient for continuous or stochastic policies.
  3. Collect trajectories with an exploration policy and store state, action, reward and next state.
  4. Estimate returns with discounting and, for actor-critic methods, a value baseline to reduce variance.
  5. Update the policy toward actions with higher estimated advantage, and keep a target network or trust region for stability.
  6. Evaluate on the true objective, not the training reward, using a held-out set of episodes or human judgement.

Trade-offs

ChoiceGood forCostFails when
Value-based methodsSample-efficient on discrete actionsUnstable with function approximation, poor for continuous controlThe action space is large or continuous
Policy gradientHandles continuous and stochastic policiesHigh variance, needs many samplesInteraction is expensive
On-policy learningStable, follows the current policyDiscards data after each updateSamples are costly
Off-policy learningReuses past experienceDivergence risk and correction complexityThe behaviour policy differs too much
Trade-off mapInteractive

Good for

Sample-efficient on discrete actions

Cost

Unstable with function approximation, poor for continuous control

Fails when

The action space is large or continuous

  1. The action space is large or continuous
  2. Interaction is expensive
  3. Samples are costly
  4. The behaviour policy differs too much

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Value-based methods is good for Sample-efficient on discrete actions, costs Unstable with function approximation, poor for continuous control, and fails when The action space is large or continuous. Policy gradient is good for Handles continuous and stochastic policies, costs High variance, needs many samples, and fails when Interaction is expensive. On-policy learning is good for Stable, follows the current policy, costs Discards data after each update, and fails when Samples are costly. Off-policy learning is good for Reuses past experience, costs Divergence risk and correction complexity, and fails when The behaviour policy differs too much

Failure modes

  • Reward hacking. The policy maximises the measured proxy in a way that violates the intent. Debug: inspect top-reward episodes manually and compare with the outcome the reward was meant to represent.
  • Sparse reward with no signal. The agent never reaches a rewarded state and the gradient stays near zero. Debug: measure the fraction of episodes with non-zero return and add shaping or demonstrations if it is near zero.
  • Policy collapse. Exploration decays too fast and the policy repeats one action. Debug: track action entropy across training; a monotone fall to near zero indicates premature convergence.

In production

Keep the policy behind a shadow mode and compare its actions against the current system before switching traffic. Monitor reward and the true outcome separately, because a rising reward with a flat outcome is the signature of a proxy that has been gamed.

Common misconceptions

People think: Reinforcement learning is unsupervised learning.

Actually: It uses a reward signal, which is supervision of a different kind; the agent is told how good an outcome was, just not which action was correct.

People think: A well-designed reward removes the need for evaluation.

Actually: The reward is a proxy, and policies optimise proxies harder than their designers expect; independent evaluation is the only check.

Interview angle

A strong answer writes the objective as expected discounted return, explains credit assignment and exploration, and treats reward design as specification. It should describe how to detect reward hacking. Weak answers describe an agent trying actions until it succeeds, without saying how credit is assigned across time or why the reward is not the goal.

Sources