Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 12 of 28

Offline vs online eval

0 of 28 complete

Why it matters

Offline evaluation is repeatable, cheap and safe, which is exactly why it uses a frozen sample that gradually stops resembling production. Online evaluation measures the outcome that matters but cannot isolate a cause without an experiment, and experiments are noisy at small traffic. The engineering problem is to keep the two connected: know which offline metric is supposed to predict which online outcome, and check that the prediction still holds.

Mental model

It is a wind tunnel versus a test track. The metaphor breaks because the track changes shape while you are driving on it and the tunnel never does, so a result from one is evidence about the other only while the mapping holds.

How it works

One question, one instrumentInteractive
startquestionoutcomepath takenruled outnote, not an option
VerdictChoose· recommended hereConditional· with the named costStandard· the ordinary case

Each branch names one question and the instrument that can answer it; the last branch is the loop that keeps them connected.

Step 1 of 5

Is it safe to ship?

A fixed dataset answers whether a change regresses cases you already know about. It cannot tell you whether users are better off, because it does not measure users.

Condition · safe to ship?

Offline gate

Choose

Answers: will this change regress known cases? Cannot: whether users are better off.

Repeatable, cheap and safe, on a frozen sample that gradually stops resembling production. Gate releases on it for regressions and safety failures.

Path walked

  1. Which question are you answering?
  2. safe to ship?Offline gate

Ruled out: Online experiment, Online read, Resampled offline set

Step 1/5 · Is it safe to ship?

Figure. Offline and online evaluation are not rivals; each answers a question the other cannot. The branch names the question, then states what the instrument can and cannot report. The dashed note is the loop that connects them, not a fifth instrument.

What to notice. The branches are questions, not a release procedure: a real system usually needs several at once, and the final dashed note is the loop that connects them. No traffic threshold or correlation number appears, because the page names none. The figure does not claim that an online lift is causal without an experiment, or that an offline score is user impact.

Text version

A decision map that routes four evaluation questions. The start asks which question you are answering. The branch “safe to ship?” leads to an offline gate: it answers whether a change regresses known cases, and cannot say whether users are better off, on a set that drifts from traffic. The branch “did it help?” leads to an online experiment: it answers the outcome under a pre-registered metric and guardrails, and cannot explain why at low traffic. The branch “why did it move?” leads to an online read: it shows that the metric moved, not that the change caused it, because seasonality, mix and novelty confound it. The branch “known failures?” leads to a resampled offline set: it keeps known failures visible and cannot cover cases never sampled. A final branch, marked always, leads to a distinct dashed note rather than an instrument: gate releases offline, test effects online, compare deltas, and feed confirmed online failures back into the set.

  1. Name the offline metric and the online outcome it should predict, and write down the causal story linking them.
  2. Gate releases on offline results for regressions and safety failures.
  3. Run an online experiment with a pre-registered primary metric and guardrails.
  4. Compare offline and online deltas for changes that shipped both ways.
  5. Re-sample the offline set from recent traffic after each experiment cycle.
  6. Feed confirmed online failures back into the offline set.

Trade-offs

ChoiceGood forCostFails when
Offline onlyFast, cheap, no user riskProxy metrics, stale distributionChange affects user behaviour
Online onlyReal outcomesSlow, risky, needs trafficTraffic is small or release is urgent
Offline gate plus online testSafety and evidenceTwo pipelines to maintainTeam cannot maintain both
Shadow or interleaved trafficRealistic without exposureInfrastructure complexityLatency or privacy constraints
Trade-off mapInteractive

Good for

Fast, cheap, no user risk

Cost

Proxy metrics, stale distribution

Fails when

Change affects user behaviour

  1. Change affects user behaviour
  2. Traffic is small or release is urgent
  3. Team cannot maintain both
  4. Latency or privacy constraints

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Offline only is good for Fast, cheap, no user risk, costs Proxy metrics, stale distribution, and fails when Change affects user behaviour. Online only is good for Real outcomes, costs Slow, risky, needs traffic, and fails when Traffic is small or release is urgent. Offline gate plus online test is good for Safety and evidence, costs Two pipelines to maintain, and fails when Team cannot maintain both. Shadow or interleaved traffic is good for Realistic without exposure, costs Infrastructure complexity, and fails when Latency or privacy constraints

Failure modes

  • Proxy divergence. The offline score improves while the online outcome falls. Debug: correlate historical offline deltas with online deltas for shipped changes.
  • Stale set. Offline passes because the set no longer resembles traffic. Debug: compare input distributions between the set and last week's logs.
  • Novelty confound. The online metric rises then decays as users adjust. Debug: plot the effect by day since launch.

In production

Track the correlation between offline and online deltas over time. Alert when the offline set's distribution drifts past a threshold and refresh it deliberately.

Common misconceptions

People think: Offline evaluation is enough to ship.

Actually: It bounds regression risk on known cases and cannot tell you the effect on users.

People think: Online results are always more trustworthy.

Actually: They are more relevant and often too noisy at small traffic to decide alone.

Interview angle

Name what each mode can and cannot measure, then describe the experiment and the proxy check. A strong answer mentions refreshing the offline set. Treating offline scores as user impact, or an online lift as causal without an experiment, is the weak response.

Sources