Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 35 of 38

Incident response

0 of 38 complete

Why it matters

AI incidents look like ordinary outages until the review, when the team discovers that nobody recorded which prompt version was live or which provider status page changed. Because a model can degrade without an error, the first signal is often a user complaint, and the fastest mitigation is a rollback to a known version rather than a fix. Response practice makes the rollback fast and the evidence complete.

Mental model

It is an emergency room: triage, stabilise, then diagnose, with one person deciding who is treated first. The analogy breaks because the patient is a distributed system with several versions in flight, so stabilising means identifying which version to stop rather than which wound to close.

How it works

A rollback that worked, and two evidence gapsInteractive
workwaitretryversionriskgap · no spanevent
24 min

Event · step 1 of 7

14:22 · alert and ack

At 24 min

The window starts at 13:58 with v17 serving. Prompt v18 ships at 14:12 with no error spike; wrong answers start at 14:16. The SLO burn is detected at 14:18, the page goes out at 14:20 and is acknowledged at 14:22; a commander is assigned at 14:24, separate from the debuggers. Triage has begun; the incident log has no version for v18.

On screen at this event

  • v17
  • v18
  • wrong answers
  • alert
  • ack
  • no version
  • v18
  • triage

Next at 33 min: 14:31 · mitigate before diagnosing

Prompt live
v18
Minutes to mitigate
—
Incident log
no version
Trace evidence
live
Commander
—
Step 1/7 · 14:22 · alert and ack

Figure. A synthetic 60-minute window starting at 13:58. The team rolls back first and restores service in twenty minutes, but the incident log has no version field at onset and un-exported traces are deleted at 14:43. Times and revisions are illustrative.

What to notice. The failures are not the rollback. The incident log has no version between 14:12 and 14:33, so the version lane is reconstructed from response metadata; the trace store drops the un-exported traces at 14:43, so only the export that starts at 14:38 preserves the incident window. Mitigation was fast because the team rolled back first. The review is slow because the evidence was not captured at onset and not exported before retention.

Text version

Six lanes over 60 minutes. Users see wrong answers 18–38. The SLO burn is detected at 20 (14:18), the page goes out at 22 (14:20), it is acknowledged at 24 (14:22) and a commander is assigned at 26 (14:24). Mitigation triages 24–33 and rolls v18 back to v17 at 33–38. Versions are v17 0–14, v18 14–35 in red, and v17 35–60. The incident log has a red gap 14–35 labelled no version, then backfilled 35–40. The trace store retains traces 0–45, shows a dashed deletion 45–60 labelled deleted, and an export at 40–45 on a second row. Markers read v18 deploy at 14, rollback at 33 and retention at 45. Counters: prompt v18 until the rollback, then v17; minutes to mitigate 20 from the first report; incident log no version until backfilled; trace evidence live, exporting, then partial and deleted; commander named from 14:24.

  1. Declare an incident from a user-visible symptom, and assign a commander separate from the people debugging.
  2. Mitigate with the cheapest reversible action: roll back the prompt, shift traffic to the previous model, disable a tool or shed load.
  3. Record a timeline as you go, including model revision, prompt version, provider status and the exact time of each action.
  4. Preserve traces and request samples before retention deletes them, and note the query used.
  5. Communicate on a fixed cadence with what is known, what is not, and when the next update comes.
  6. Run a blameless review that separates trigger, contributing factors and the detection gap, with owned actions.

Trade-offs

ChoiceGood forCostFails when
Roll back firstFast mitigationLoses the evidence in the bad versionThe cause needs live debugging
Diagnose firstUnderstands the failureLonger user impactThe regression is a known change
Broad status communicationTrust and fewer duplicate reportsTime from the respondersThe audience is external and regulated
Narrow internal responseFast coordinationUsers learn from social mediaThe outage is visible
Trade-off mapInteractive

Good for

Fast mitigation

Cost

Loses the evidence in the bad version

Fails when

The cause needs live debugging

  1. The cause needs live debugging
  2. The regression is a known change
  3. The audience is external and regulated
  4. The outage is visible

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Roll back first is good for Fast mitigation, costs Loses the evidence in the bad version, and fails when The cause needs live debugging. Diagnose first is good for Understands the failure, costs Longer user impact, and fails when The regression is a known change. Broad status communication is good for Trust and fewer duplicate reports, costs Time from the responders, and fails when The audience is external and regulated. Narrow internal response is good for Fast coordination, costs Users learn from social media, and fails when The outage is visible

Failure modes

  • Missing version in the timeline. The review cannot tell which prompt was live at the onset. Debug: check that the incident log has a version field populated from response metadata.
  • Evidence deleted by retention. Traces expire before the review and the failing prompts are gone. Debug: confirm an export exists for the incident window with a timestamp before retention.
  • No commander. Two teams apply conflicting mitigations and the system oscillates. Debug: read the incident channel and count distinct people issuing actions without coordination.

In production

Keep a rollback runbook that names the exact commands and the current version pointers, and rehearse it. Time the first mitigation, not just the resolution, and review that number after each incident.

Common misconceptions

People think: Incident response is about finding the root cause.

Actually: It is about limiting impact first; the cause can wait until users are served.

People think: A model regression is not an incident because nothing is down.

Actually: A quality regression that reaches users is an incident with a different signal, and it needs the same command and rollback discipline.

Interview angle

Separate roles, mitigation and diagnosis, and name the version data the timeline needs. Describe preserving evidence before retention and a review with owned actions. Jumping to a root-cause hypothesis while users are still affected is the failure.

Sources