Skip to content
AI Engineer Guide.

pattern · architecture · intermediate · concept 5 of 7

Event-driven eval loop

0 of 7 complete

Why it matters

Offline eval sets drift from production, and manual review cannot cover volume. An event loop closes the gap by capturing real interactions and scoring them continuously, so a regression appears as a trend rather than a complaint. The engineering problem is making those events complete enough to explain a score and cheap enough to keep, while keeping the judge itself honest.

Mental model

A quality inspector sampling output on the line: continuous, partial, and only as good as the checklist they carry. The metaphor breaks because the inspector here is another model, so the checklist needs auditing and the inspector can be swapped without anyone noticing.

How it works

Mechanism, step by stepInteractive
  1. 1Define the event schema: request id, prompt version, model, retrieved ids, output, latency and user feedback if any.
  2. 2Choose a sampling policy per route: full capture for low-volume critical flows, deterministic sampling for high-volume ones.
  3. 3Score asynchronously with a judge, and validate the judge against a human-labelled slice before trusting it.
  4. 4Store scores beside the event so a failing slice can be replayed against a candidate prompt.
  5. 5Separate product metrics from quality metrics, since a drop in satisfaction can be latency rather than answer quality.
  6. 6Route the worst-scoring events into a review queue and promote confirmed failures into the offline set.

Step 1 of 6

Define the event schema: request id, prompt version, model, retrieved ids, output, latency and user feedback if any.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Define the event schema: request id, prompt version, model, retrieved ids, output, latenc… 2. Choose a sampling policy per route: full capture for low-volume critical flows, determini… 3. Score asynchronously with a judge, and validate the judge against a human-labelled slice … 4. Store scores beside the event so a failing slice can be replayed against a candidate prom… 5. Separate product metrics from quality metrics, since a drop in satisfaction can be latenc… 6. Route the worst-scoring events into a review queue and promote confirmed failures into th…

  1. Define the event schema: request id, prompt version, model, retrieved ids, output, latency and user feedback if any.
  2. Choose a sampling policy per route: full capture for low-volume critical flows, deterministic sampling for high-volume ones.
  3. Score asynchronously with a judge, and validate the judge against a human-labelled slice before trusting it.
  4. Store scores beside the event so a failing slice can be replayed against a candidate prompt.
  5. Separate product metrics from quality metrics, since a drop in satisfaction can be latency rather than answer quality.
  6. Route the worst-scoring events into a review queue and promote confirmed failures into the offline set.

Trade-offs

ChoiceGood forCostFails when
Score every eventComplete pictureJudge cost and a growing backlogVolume is high
Sample by request idCheap and unbiasedMisses rare failuresFailures cluster in rare routes
Human review onlyTrustworthy labelsLow coverage and slowTraffic is large
Judge onlyBroad coverageJudge bias and driftThe domain needs expert judgement
Trade-off mapInteractive

Good for

Complete picture

Cost

Judge cost and a growing backlog

Fails when

Volume is high

  1. Volume is high
  2. Failures cluster in rare routes
  3. Traffic is large
  4. The domain needs expert judgement

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Score every event is good for Complete picture, costs Judge cost and a growing backlog, and fails when Volume is high. Sample by request id is good for Cheap and unbiased, costs Misses rare failures, and fails when Failures cluster in rare routes. Human review only is good for Trustworthy labels, costs Low coverage and slow, and fails when Traffic is large. Judge only is good for Broad coverage, costs Judge bias and drift, and fails when The domain needs expert judgement

Failure modes

  • Sampling bias. Sampling by success or by easy routes hides the failures that matter. Debug: compare the score distribution of sampled and unsampled traffic on a shared metric.
  • Judge drift. A judge model changes and scores shift with no product change. Debug: keep a frozen labelled set and re-score it whenever the judge changes.
  • Unattributable score. Events lack the prompt version, so a regression cannot be traced. Debug: assert required fields on ingest and reject events that omit them.

In production

Monitor event volume, scoring lag and judge agreement with human labels. Cap judge spend per day and alert when scoring falls behind the event stream.

Common misconceptions

People think: Online evaluation replaces offline evaluation.

Actually: It complements it: offline sets give stable comparisons, online events reveal distribution and drift.

People think: A judge score is objective.

Actually: It is a model's judgement, with bias and version sensitivity, so it needs calibration against humans.

Interview angle

Describe the event schema, the sampling policy and how the judge is validated. Strong answers connect a score back to a prompt version and a replay. Weak ones add a dashboard without saying what data makes a regression explainable.

Sources