Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 25 of 38

Canary & rollback

0 of 38 complete

Why it matters

A model or prompt change can pass every offline evaluation and still fail on live traffic, because the evaluation set is smaller and cleaner than production. A canary limits the blast radius while producing the only evidence that matters: how the new version behaves on real requests, with real retrieval and real users. The engineering work is making the comparison honest and the reversal fast.

Mental model

It is a test batch in a kitchen: a few plates go out, the chef tastes the result, and the recipe changes for the whole service only if the batch holds. The analogy breaks because diners differ, so a good canary needs a cohort comparable to the population it stands in for.

How it works

The rollback boundary is not the recoveryInteractive
workcanary ramprollbackrestoredriskevent
2 min

Event · step 1 of 7

Baseline, then the canary opens

At 2 min

v41 served all traffic until minute 2, when a deterministic 5% slice moved to v42, sticky by request id so a session cannot flip between versions. The guardrails and rollback threshold were defined before the ramp started.

On screen at this event

  • 100%
  • 95%
  • 5%
  • warm · serving

Next at 4 min: Ramp to 25%

v42 share
5%
Error rate
0.4%
Refusal rate
2.9%
Quality Δ
+0.0
Rollback elapsed
—
Step 1/7 · Baseline, then the canary opens

Figure. Synthetic traffic shares on a twelve-minute clock: v41 ramps down as v42 ramps up to 50%, the quality guardrail breaches at 8 minutes, and v41 takes 1.6 minutes to become warm enough to take all traffic. Percentages are illustrative.

What to notice. The breach is visible in refusal rate and the quality sample while error rate stays flat, which is why an availability-only guardrail misses it. The rollback is triggered at 8 minutes, but traffic is still split until 9.6 because v41 was not warm. Rollback speed is a capacity fact, not a button: keep the previous version ready and rehearse the trigger-to-full-traffic time. Cohorts are assumed comparable; the request mix is not shown.

Text version

Three lanes on a twelve-minute clock. v41 traffic is 100% for 0–2, 95% for 2–4, 75% for 4–6, 50% for 6–8, a dashed red 50% pending from 8 to 9.6, then 100% from 9.6 to 12. v42 traffic is 5% for 2–4, 25% for 4–6, 50% for 6–8 and a red 50% still live from 8 to 9.6, then zero. The v41 process is warm and serving 0–8, loading weights 8–9.6, and ready 9.6–12. A red breach marker sits at 8 and an event marker v41 100% at 9.6. Counters: v42 share 5, 25, 50, 50, 50, 0, 0 percent; error rate stays 0.4–0.6%; refusal rate rises to 5.2%; quality delta falls to −0.6; rollback elapsed 1.6 minutes.

  1. Decide what changes: weights, template, retrieval configuration or decoding parameters, and version each one.
  2. Run the candidate in shadow mode first if the workload allows, comparing outputs without serving them.
  3. Canary a small deterministic slice, sticky by user or request id, so a session does not flip between versions.
  4. Pre-register guardrails: error rate, latency percentiles, refusal rate, a quality proxy and a business metric.
  5. Automate the comparison and the rollback, with a human able to stop the rollout but not required to complete it.
  6. Keep the previous version deployed and warm until the canary completes and the metrics have settled.

Trade-offs

ChoiceGood forCostFails when
Small canary sliceLow blast radiusSlow to reach significanceThe failure affects a rare request type
Large canary sliceFast evidenceMore users exposedThe change is risky and traffic is small
Automatic rollbackFast reversalCan trip on noiseGuardrail metrics are noisy at low volume
Manual approvalHuman judgement on qualitySlower reversal, on-call dependentThe incident happens outside working hours
Trade-off mapInteractive

Good for

Low blast radius

Cost

Slow to reach significance

Fails when

The failure affects a rare request type

  1. The failure affects a rare request type
  2. The change is risky and traffic is small
  3. Guardrail metrics are noisy at low volume
  4. The incident happens outside working hours

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Small canary slice is good for Low blast radius, costs Slow to reach significance, and fails when The failure affects a rare request type. Large canary slice is good for Fast evidence, costs More users exposed, and fails when The change is risky and traffic is small. Automatic rollback is good for Fast reversal, costs Can trip on noise, and fails when Guardrail metrics are noisy at low volume. Manual approval is good for Human judgement on quality, costs Slower reversal, on-call dependent, and fails when The incident happens outside working hours

Failure modes

  • Non-comparable cohorts. The canary gets a different traffic mix and looks better or worse for reasons unrelated to the change. Debug: compare the request mix by route and length between arms before reading the metrics.
  • Guardrail too coarse. A quality regression hides behind a flat error rate. Debug: include a labelled quality sample in the canary comparison, not just availability.
  • Rollback slower than spread. The old version takes minutes to become ready while the new one keeps serving. Debug: time a rollback in staging and compare with the canary ramp schedule.

In production

Emit the version on every response and in every metric label, so the canary is measurable without a separate pipeline. Rehearse a rollback under load each quarter, and record the time from trigger to full traffic on the previous version.

Common misconceptions

People think: Canaries are only for code deployments.

Actually: Prompt and retrieval changes alter behaviour without a binary, so they need the same staged comparison.

People think: A canary that shows no errors is a success.

Actually: Errors are the easiest signal to keep flat; the regression usually appears in quality, refusal rate or cost.

Interview angle

Describe the slice, the sticky assignment and the pre-registered guardrails, then the rollback timing. A strong answer separates quality from availability signals and explains cohort comparability. Weak answers say they send one percent of traffic and watch the error rate.

Sources