Why it matters
Teams often choose a metric first and then optimise it, which measures the proxy rather than the product. Error analysis reverses that: read real traces, describe what went wrong in the user's terms, group the descriptions and count them. The largest group becomes the next fix and the next eval cases, and the counts make the priority arguable instead of anecdotal.
Mental model
It is a doctor reading the chart before ordering tests. The metaphor breaks because the chart is a sample, and the disease may be in the patients nobody sampled, so the taxonomy needs counts and a fresh sample later with the same instrument.
How it works
- 34
- 19
- 9
- 6
- 18
- 11
- 7
- 14
- 9
- 5
- 8
- 8
- 6
- 4
- 2
Step 1 of 7
Read the sample
120 traces sampled at random and read end to end, including 40 successes. The successes are the comparison that tells you whether the pipeline had the answer at all.
root · current
Sample · 120 traces
A random sample read end to end, including successes. The successes are what separate 'the pipeline never had the document' from 'the model used it badly'.
Root of the hierarchy
Figure. A taxonomy of 80 sampled failures whose five group counts sum to the sample, with the largest group opened into three slices. Child counts are parts of their parent. The bars are counts, not rates: 34 retrieval misses and 6 format violations are different sizes of problem.
What to notice. Grouping by root cause is what makes the taxonomy actionable; grouping by symptom collapses these five into one bucket called 'answer was wrong'. Counts come from one sample of 120 traces including 40 successes, and carry sampling error the figure does not draw — 8 refusals is a small slice. The numbers are illustrative; the instrument — sample, describe, group, count — is the page's method.
Text version
A failure taxonomy drawn as a tree with counts. The root is a sample of 120 traces: 80 failures and 40 successes read end to end. Five root causes hang from it, and their counts sum to 80. Retrieval missed the answer has 34, split into a prose query at 19, a stale index at 9 and an un-ingested document at 6. Right document, wrong span has 18, split into a chunk boundary splitting a table at 11 and the answer sitting in a later section at 7. Tool call malformed has 14, split into a wrong argument name at 9 and a missing required field at 5. Refused a valid request has 8, all policy over-trigger. Format violation has 6, split into a missing citation at 4 and a wrong unit at 2. Later steps open the retrieval group, convert its dominant cause into eval cases, and re-sample after the fix.
- Collect a random sample of real traces including successes, and read them end to end.
- Write a one-line description of what went wrong for each failure in the user's terms.
- Group the descriptions into root causes and count each group.
- Pick the largest group that is fixable with the current architecture.
- Turn representative examples into eval cases with a check for that cause.
- Re-sample after the fix to see whether the distribution changed.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Small manual sample | Deep understanding | Sampling error | Failure is rare |
| Large automated triage | Coverage | Classifier bias, shallow labels | You need root causes |
| Symptom grouping | Fast | Duplicate fixes | Causes differ under one symptom |
| Root-cause grouping | Actionable | Slower, needs judgement | Labellers lack system knowledge |
Good for
Deep understanding
Cost
Sampling error
Fails when
Failure is rare
- Failure is rare
- You need root causes
- Causes differ under one symptom
- Labellers lack system knowledge
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Small manual sample is good for Deep understanding, costs Sampling error, and fails when Failure is rare. Large automated triage is good for Coverage, costs Classifier bias, shallow labels, and fails when You need root causes. Symptom grouping is good for Fast, costs Duplicate fixes, and fails when Causes differ under one symptom. Root-cause grouping is good for Actionable, costs Slower, needs judgement, and fails when Labellers lack system knowledge
Failure modes
- Sampling only failures. The base rate and successful patterns are invisible. Debug: include successes in the sample and compare their traces.
- Symptom taxonomy. Every group is answer was wrong and no fix follows. Debug: rewrite each group label as a cause that names a pipeline stage.
- Anecdote-driven roadmap. The loudest complaint sets priorities. Debug: compare group counts with the sample size and an interval.
In production
Run error analysis on a fixed cadence and store the taxonomy with counts. Alert when a new group appears after a release.
Common misconceptions
People think: Error analysis is optional once metrics exist.
Actually: Metrics say that quality moved, while analysis says why and what to fix.
People think: More traces always means better analysis.
Actually: Beyond a sample, reading time grows while the dominant causes are already visible.
Interview angle
Sample real traces, group by root cause, count the groups and convert the largest into eval cases. A strong answer includes successes and explains how priority was chosen. Jumping to a metric without reading failures is the weak version.
Sources
- Hamel Husain, A Field Guide to Rapidly Improving AI Products — looking at data before choosing metrics.
- Shankar et al., Who Validates the Validators? — iterative alignment of evaluators with human labels.
- Ribeiro et al., Beyond Accuracy with CheckList — turning observed failures into behavioural tests.