Why it matters
Offline eval sets drift from production, and manual review cannot cover volume. An event loop closes the gap by capturing real interactions and scoring them continuously, so a regression appears as a trend rather than a complaint. The engineering problem is making those events complete enough to explain a score and cheap enough to keep, while keeping the judge itself honest.
Mental model
A quality inspector sampling output on the line: continuous, partial, and only as good as the checklist they carry. The metaphor breaks because the inspector here is another model, so the checklist needs auditing and the inspector can be swapped without anyone noticing.
How it works
- 1Define the event schema: request id, prompt version, model, retrieved ids, output, latency and user feedback if any.
- 2Choose a sampling policy per route: full capture for low-volume critical flows, deterministic sampling for high-volume ones.
- 3Score asynchronously with a judge, and validate the judge against a human-labelled slice before trusting it.
- 4Store scores beside the event so a failing slice can be replayed against a candidate prompt.
- 5Separate product metrics from quality metrics, since a drop in satisfaction can be latency rather than answer quality.
- 6Route the worst-scoring events into a review queue and promote confirmed failures into the offline set.
Step 1 of 6
Define the event schema: request id, prompt version, model, retrieved ids, output, latency and user feedback if any.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Define the event schema: request id, prompt version, model, retrieved ids, output, latenc… 2. Choose a sampling policy per route: full capture for low-volume critical flows, determini… 3. Score asynchronously with a judge, and validate the judge against a human-labelled slice … 4. Store scores beside the event so a failing slice can be replayed against a candidate prom… 5. Separate product metrics from quality metrics, since a drop in satisfaction can be latenc… 6. Route the worst-scoring events into a review queue and promote confirmed failures into th…
- Define the event schema: request id, prompt version, model, retrieved ids, output, latency and user feedback if any.
- Choose a sampling policy per route: full capture for low-volume critical flows, deterministic sampling for high-volume ones.
- Score asynchronously with a judge, and validate the judge against a human-labelled slice before trusting it.
- Store scores beside the event so a failing slice can be replayed against a candidate prompt.
- Separate product metrics from quality metrics, since a drop in satisfaction can be latency rather than answer quality.
- Route the worst-scoring events into a review queue and promote confirmed failures into the offline set.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Score every event | Complete picture | Judge cost and a growing backlog | Volume is high |
| Sample by request id | Cheap and unbiased | Misses rare failures | Failures cluster in rare routes |
| Human review only | Trustworthy labels | Low coverage and slow | Traffic is large |
| Judge only | Broad coverage | Judge bias and drift | The domain needs expert judgement |
Good for
Complete picture
Cost
Judge cost and a growing backlog
Fails when
Volume is high
- Volume is high
- Failures cluster in rare routes
- Traffic is large
- The domain needs expert judgement
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Score every event is good for Complete picture, costs Judge cost and a growing backlog, and fails when Volume is high. Sample by request id is good for Cheap and unbiased, costs Misses rare failures, and fails when Failures cluster in rare routes. Human review only is good for Trustworthy labels, costs Low coverage and slow, and fails when Traffic is large. Judge only is good for Broad coverage, costs Judge bias and drift, and fails when The domain needs expert judgement
Failure modes
- Sampling bias. Sampling by success or by easy routes hides the failures that matter. Debug: compare the score distribution of sampled and unsampled traffic on a shared metric.
- Judge drift. A judge model changes and scores shift with no product change. Debug: keep a frozen labelled set and re-score it whenever the judge changes.
- Unattributable score. Events lack the prompt version, so a regression cannot be traced. Debug: assert required fields on ingest and reject events that omit them.
In production
Monitor event volume, scoring lag and judge agreement with human labels. Cap judge spend per day and alert when scoring falls behind the event stream.
Common misconceptions
People think: Online evaluation replaces offline evaluation.
Actually: It complements it: offline sets give stable comparisons, online events reveal distribution and drift.
People think: A judge score is objective.
Actually: It is a model's judgement, with bias and version sensitivity, so it needs calibration against humans.
Interview angle
Describe the event schema, the sampling policy and how the judge is validated. Strong answers connect a score back to a prompt version and a replay. Weak ones add a dashboard without saying what data makes a regression explainable.
Sources
- Azure Architecture Center, Event-Driven Architecture Style — producers, consumers and event routing.
- Breck et al., The ML Test Score — monitoring and production readiness for ML systems.
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — agreement, bias and limits of model judges.