Why it matters
AI incidents look like ordinary outages until the review, when the team discovers that nobody recorded which prompt version was live or which provider status page changed. Because a model can degrade without an error, the first signal is often a user complaint, and the fastest mitigation is a rollback to a known version rather than a fix. Response practice makes the rollback fast and the evidence complete.
Mental model
It is an emergency room: triage, stabilise, then diagnose, with one person deciding who is treated first. The analogy breaks because the patient is a distributed system with several versions in flight, so stabilising means identifying which version to stop rather than which wound to close.
How it works
Event · step 1 of 7
14:22 · alert and ack
At 24 min
The window starts at 13:58 with v17 serving. Prompt v18 ships at 14:12 with no error spike; wrong answers start at 14:16. The SLO burn is detected at 14:18, the page goes out at 14:20 and is acknowledged at 14:22; a commander is assigned at 14:24, separate from the debuggers. Triage has begun; the incident log has no version for v18.
On screen at this event
- v17
- v18
- wrong answers
- alert
- ack
- no version
- v18
- triage
Next at 33 min: 14:31 · mitigate before diagnosing
- Prompt live
- v18
- Minutes to mitigate
- —
- Incident log
- no version
- Trace evidence
- live
- Commander
- —
Figure. A synthetic 60-minute window starting at 13:58. The team rolls back first and restores service in twenty minutes, but the incident log has no version field at onset and un-exported traces are deleted at 14:43. Times and revisions are illustrative.
What to notice. The failures are not the rollback. The incident log has no version between 14:12 and 14:33, so the version lane is reconstructed from response metadata; the trace store drops the un-exported traces at 14:43, so only the export that starts at 14:38 preserves the incident window. Mitigation was fast because the team rolled back first. The review is slow because the evidence was not captured at onset and not exported before retention.
Text version
Six lanes over 60 minutes. Users see wrong answers 18–38. The SLO burn is detected at 20 (14:18), the page goes out at 22 (14:20), it is acknowledged at 24 (14:22) and a commander is assigned at 26 (14:24). Mitigation triages 24–33 and rolls v18 back to v17 at 33–38. Versions are v17 0–14, v18 14–35 in red, and v17 35–60. The incident log has a red gap 14–35 labelled no version, then backfilled 35–40. The trace store retains traces 0–45, shows a dashed deletion 45–60 labelled deleted, and an export at 40–45 on a second row. Markers read v18 deploy at 14, rollback at 33 and retention at 45. Counters: prompt v18 until the rollback, then v17; minutes to mitigate 20 from the first report; incident log no version until backfilled; trace evidence live, exporting, then partial and deleted; commander named from 14:24.
- Declare an incident from a user-visible symptom, and assign a commander separate from the people debugging.
- Mitigate with the cheapest reversible action: roll back the prompt, shift traffic to the previous model, disable a tool or shed load.
- Record a timeline as you go, including model revision, prompt version, provider status and the exact time of each action.
- Preserve traces and request samples before retention deletes them, and note the query used.
- Communicate on a fixed cadence with what is known, what is not, and when the next update comes.
- Run a blameless review that separates trigger, contributing factors and the detection gap, with owned actions.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Roll back first | Fast mitigation | Loses the evidence in the bad version | The cause needs live debugging |
| Diagnose first | Understands the failure | Longer user impact | The regression is a known change |
| Broad status communication | Trust and fewer duplicate reports | Time from the responders | The audience is external and regulated |
| Narrow internal response | Fast coordination | Users learn from social media | The outage is visible |
Good for
Fast mitigation
Cost
Loses the evidence in the bad version
Fails when
The cause needs live debugging
- The cause needs live debugging
- The regression is a known change
- The audience is external and regulated
- The outage is visible
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Roll back first is good for Fast mitigation, costs Loses the evidence in the bad version, and fails when The cause needs live debugging. Diagnose first is good for Understands the failure, costs Longer user impact, and fails when The regression is a known change. Broad status communication is good for Trust and fewer duplicate reports, costs Time from the responders, and fails when The audience is external and regulated. Narrow internal response is good for Fast coordination, costs Users learn from social media, and fails when The outage is visible
Failure modes
- Missing version in the timeline. The review cannot tell which prompt was live at the onset. Debug: check that the incident log has a version field populated from response metadata.
- Evidence deleted by retention. Traces expire before the review and the failing prompts are gone. Debug: confirm an export exists for the incident window with a timestamp before retention.
- No commander. Two teams apply conflicting mitigations and the system oscillates. Debug: read the incident channel and count distinct people issuing actions without coordination.
In production
Keep a rollback runbook that names the exact commands and the current version pointers, and rehearse it. Time the first mitigation, not just the resolution, and review that number after each incident.
Common misconceptions
People think: Incident response is about finding the root cause.
Actually: It is about limiting impact first; the cause can wait until users are served.
People think: A model regression is not an incident because nothing is down.
Actually: A quality regression that reaches users is an incident with a different signal, and it needs the same command and rollback discipline.
Interview angle
Separate roles, mitigation and diagnosis, and name the version data the timeline needs. Describe preserving evidence before retention and a review with owned actions. Jumping to a root-cause hypothesis while users are still affected is the failure.
Sources
- Google, SRE Book: Managing Incidents — roles, communication and mitigation-first practice.
- PagerDuty, Incident Response Documentation — a public incident response process and role definitions.