Why it matters
Offline sets freeze and stop representing traffic, so some measurement has to happen after release. Online evaluation combines experiment design with sampled review: experiments estimate causal effects, guardrails catch harm, and judge or human review catches quality drift that a click metric cannot see. The difficulty is that live traffic is confounded, and a naive before-and-after comparison will attribute seasonality to the release.
Mental model
It is a test flight with passengers aboard. The metaphor breaks because passengers change behaviour when they notice the new cabin, the crew cannot rerun the flight to check whether the result was real without a control group, and a rerun costs another week.
How it works
- 1Choose a primary outcome and guardrails, and write the decision rule before launch.
- 2Randomise at the unit that matches the product, such as user or session, and check assignment balance.
- 3Run long enough to cover weekly cycles, and avoid peeking without a sequential method.
- 4Sample live conversations for judge or human review with consent and redaction.
- 5Segment results by intent, tenure and locale to find offsetting effects.
- 6Feed confirmed failures back into the offline set.
Step 1 of 6
Choose a primary outcome and guardrails, and write the decision rule before launch.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Choose a primary outcome and guardrails, and write the decision rule before launch. 2. Randomise at the unit that matches the product, such as user or session, and check assign… 3. Run long enough to cover weekly cycles, and avoid peeking without a sequential method. 4. Sample live conversations for judge or human review with consent and redaction. 5. Segment results by intent, tenure and locale to find offsetting effects. 6. Feed confirmed failures back into the offline set.
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. A running estimate of the primary metric with its 95% interval, and the guardrail's estimate with its interval, at three analysis looks in a pre-registered 28-day experiment. The interval narrows as traffic accumulates; the estimate moves inside it. The data are modelled and the decision rule is the claim.
What to notice. At day 10 the primary lift is +2.4 points with an interval of +0.9 to +3.9, which excludes zero. The same test at day 14 reads +0.9 with an interval of −0.4 to +2.2, and at the pre-registered day 28 it settles at +1.4 (+0.5 to +2.3). The guardrail at day 28 is +0.2 points (−0.3 to +0.8): the primary improved while the complaint rate's interval still admits a rise.
Text version
A running estimate of a primary metric from day 1 to day 28, in percentage points, with dashed upper and lower 95% interval bounds and a line at no lift. At day 10 the estimate is +2.4 with bounds +0.9 to +3.9; at day 14 it is +0.9 with bounds −0.4 to +2.2; at day 28 it is +1.4 with bounds +0.5 to +2.3. Below, the complaint-rate estimate sits near +0.2 with bounds that cross zero at every look; at day 28 they run −0.3 to +0.8. A vertical line marks the selected look. Three buttons switch between the day-10, day-14 and day-28 analyses; a panel reports both estimates and their intervals.
- Choose a primary outcome and guardrails, and write the decision rule before launch.
- Randomise at the unit that matches the product, such as user or session, and check assignment balance.
- Run long enough to cover weekly cycles, and avoid peeking without a sequential method.
- Sample live conversations for judge or human review with consent and redaction.
- Segment results by intent, tenure and locale to find offsetting effects.
- Feed confirmed failures back into the offline set.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| A/B test | Causal estimate | Needs traffic and time | Effect is small or traffic is low |
| Shadow or interleaved traffic | Fast, low exposure | Infrastructure work | Latency or privacy constraints |
| Sampled judge review | Covers quality | Judge bias, sampling error | Decisions are high stakes |
| Guardrail dashboards only | Always on | Correlational | You need a causal claim |
Good for
Causal estimate
Cost
Needs traffic and time
Fails when
Effect is small or traffic is low
- Effect is small or traffic is low
- Latency or privacy constraints
- Decisions are high stakes
- You need a causal claim
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: A/B test is good for Causal estimate, costs Needs traffic and time, and fails when Effect is small or traffic is low. Shadow or interleaved traffic is good for Fast, low exposure, costs Infrastructure work, and fails when Latency or privacy constraints. Sampled judge review is good for Covers quality, costs Judge bias, sampling error, and fails when Decisions are high stakes. Guardrail dashboards only is good for Always on, costs Correlational, and fails when You need a causal claim
Failure modes
- Peeking. The test is stopped at a lucky moment and the effect reverses later. Debug: check the analysis plan and the number of interim looks.
- Sample ratio mismatch. Assignment is skewed and the comparison is invalid. Debug: compare observed group sizes against the intended split.
- Metric myopia. The primary metric improves while complaint rate worsens. Debug: review all guardrails, not only the headline.
In production
Monitor assignment balance, guardrails and review-sample coverage. Freeze the decision rule and log every interim analysis.
Common misconceptions
People think: Online metrics are self-explanatory.
Actually: They are confounded by mix, seasonality and novelty until the design controls for them.
People think: Any visible improvement justifies shipping.
Actually: Without guardrails, an improvement in one metric can hide harm in another.
Interview angle
Describe randomisation, pre-registration, guardrails and sampled review, and explain why peeking invalidates a result. A strong answer mentions segmentation. Quoting a dashboard movement as improvement, with no assignment or sample size, is the weak version.
Sources
- Kohavi et al., Trustworthy Online Controlled Experiments — randomisation, guardrails and analysis traps.
- Chiang et al., Chatbot Arena — continuous live preference collection.
- Li et al., Unbiased Offline Evaluation of Contextual-Bandit Algorithms — why live and offline estimates diverge.