Why it matters
Offline evaluation is repeatable, cheap and safe, which is exactly why it uses a frozen sample that gradually stops resembling production. Online evaluation measures the outcome that matters but cannot isolate a cause without an experiment, and experiments are noisy at small traffic. The engineering problem is to keep the two connected: know which offline metric is supposed to predict which online outcome, and check that the prediction still holds.
Mental model
It is a wind tunnel versus a test track. The metaphor breaks because the track changes shape while you are driving on it and the tunnel never does, so a result from one is evidence about the other only while the mapping holds.
How it works
Each branch names one question and the instrument that can answer it; the last branch is the loop that keeps them connected.
Step 1 of 5
Is it safe to ship?
A fixed dataset answers whether a change regresses cases you already know about. It cannot tell you whether users are better off, because it does not measure users.
Condition · safe to ship?
Offline gate
Choose
Answers: will this change regress known cases? Cannot: whether users are better off.
Repeatable, cheap and safe, on a frozen sample that gradually stops resembling production. Gate releases on it for regressions and safety failures.
Path walked
- Which question are you answering?
- safe to ship?Offline gate
Ruled out: Online experiment, Online read, Resampled offline set
Figure. Offline and online evaluation are not rivals; each answers a question the other cannot. The branch names the question, then states what the instrument can and cannot report. The dashed note is the loop that connects them, not a fifth instrument.
What to notice. The branches are questions, not a release procedure: a real system usually needs several at once, and the final dashed note is the loop that connects them. No traffic threshold or correlation number appears, because the page names none. The figure does not claim that an online lift is causal without an experiment, or that an offline score is user impact.
Text version
A decision map that routes four evaluation questions. The start asks which question you are answering. The branch “safe to ship?” leads to an offline gate: it answers whether a change regresses known cases, and cannot say whether users are better off, on a set that drifts from traffic. The branch “did it help?” leads to an online experiment: it answers the outcome under a pre-registered metric and guardrails, and cannot explain why at low traffic. The branch “why did it move?” leads to an online read: it shows that the metric moved, not that the change caused it, because seasonality, mix and novelty confound it. The branch “known failures?” leads to a resampled offline set: it keeps known failures visible and cannot cover cases never sampled. A final branch, marked always, leads to a distinct dashed note rather than an instrument: gate releases offline, test effects online, compare deltas, and feed confirmed online failures back into the set.
- Name the offline metric and the online outcome it should predict, and write down the causal story linking them.
- Gate releases on offline results for regressions and safety failures.
- Run an online experiment with a pre-registered primary metric and guardrails.
- Compare offline and online deltas for changes that shipped both ways.
- Re-sample the offline set from recent traffic after each experiment cycle.
- Feed confirmed online failures back into the offline set.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Offline only | Fast, cheap, no user risk | Proxy metrics, stale distribution | Change affects user behaviour |
| Online only | Real outcomes | Slow, risky, needs traffic | Traffic is small or release is urgent |
| Offline gate plus online test | Safety and evidence | Two pipelines to maintain | Team cannot maintain both |
| Shadow or interleaved traffic | Realistic without exposure | Infrastructure complexity | Latency or privacy constraints |
Good for
Fast, cheap, no user risk
Cost
Proxy metrics, stale distribution
Fails when
Change affects user behaviour
- Change affects user behaviour
- Traffic is small or release is urgent
- Team cannot maintain both
- Latency or privacy constraints
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Offline only is good for Fast, cheap, no user risk, costs Proxy metrics, stale distribution, and fails when Change affects user behaviour. Online only is good for Real outcomes, costs Slow, risky, needs traffic, and fails when Traffic is small or release is urgent. Offline gate plus online test is good for Safety and evidence, costs Two pipelines to maintain, and fails when Team cannot maintain both. Shadow or interleaved traffic is good for Realistic without exposure, costs Infrastructure complexity, and fails when Latency or privacy constraints
Failure modes
- Proxy divergence. The offline score improves while the online outcome falls. Debug: correlate historical offline deltas with online deltas for shipped changes.
- Stale set. Offline passes because the set no longer resembles traffic. Debug: compare input distributions between the set and last week's logs.
- Novelty confound. The online metric rises then decays as users adjust. Debug: plot the effect by day since launch.
In production
Track the correlation between offline and online deltas over time. Alert when the offline set's distribution drifts past a threshold and refresh it deliberately.
Common misconceptions
People think: Offline evaluation is enough to ship.
Actually: It bounds regression risk on known cases and cannot tell you the effect on users.
People think: Online results are always more trustworthy.
Actually: They are more relevant and often too noisy at small traffic to decide alone.
Interview angle
Name what each mode can and cannot measure, then describe the experiment and the proxy check. A strong answer mentions refreshing the offline set. Treating offline scores as user impact, or an online lift as causal without an experiment, is the weak response.
Sources
- Kohavi et al., Trustworthy Online Controlled Experiments — experiment design, guardrails and common traps.
- Li et al., Unbiased Offline Evaluation of Contextual-Bandit Algorithms — why offline estimates can diverge from live outcomes.
- Liang et al., HELM — a standardised offline evaluation harness.