Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 3 of 28

Coverage

0 of 28 complete

Why it matters

A suite of two hundred cases can pass while an entire intent category is untested, and the aggregate score will look healthy. Coverage is the property that says whether the sample resembles the work the system actually does. Without it, a team cannot distinguish a robust system from one that happens to handle the cases someone thought of, and every new feature silently adds behaviour nobody is measuring.

Mental model

It is the mesh size of a net: what you catch depends on the holes. The metaphor breaks because you never see the fish that swam through, only the ones left in the net, so an empty hole looks exactly like a sea with no fish.

How it works

The suite's shape against traffic's shapeInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Share of last month's requests and share of the eval suite, by intent cluster, ordered by traffic. The counts are illustrative; the mismatch between the two bars is what coverage measures.

What to notice. Integrations is 10.8% of traffic and has no case at all; other is 3.8% and also has none, so 14.6% of traffic is untested while the aggregate score stays healthy. Billing is over-sampled at 37.6% of cases against 28.4% of traffic. Adding 33 cases to the two empty clusters covers every cluster and brings the case mix closer to the traffic mix.

Text version

Horizontal bars for seven intent clusters, each with two bars: share of traffic and share of eval cases. Billing is 28.4% and 37.6%, refunds 19.2% and 29.4%, account 17.1% and 18.3%, shipping 13.9% and 11.2%, integrations 10.8% with no cases, policy 6.8% and 3.6%, other 3.8% with no cases. The two zero bars are dashed stubs labelled no cases. A two-button control switches to an after-remediation suite of 230 cases, where integrations is 10.4% and other 3.9%, and a panel reports that five of seven clusters are covered before and seven of seven after.

  1. Cluster production or historical requests by intent and record the share of traffic in each cluster.
  2. Map existing eval cases onto those clusters and mark the clusters with no cases.
  3. Add cases for the highest-volume empty cluster before adding rare edge cases.
  4. Include known failure categories from incidents, support tickets and review comments.
  5. Maintain coverage as a table of clusters with counts and review it when traffic shifts.
  6. Re-sample the production side quarterly, because usage drifts away from the original set.

Trade-offs

ChoiceGood forCostFails when
Sample from production trafficRepresentativeBias toward current behaviour, privacy workNew capability has no traffic yet
Synthetic edge casesCovers rare and hostile inputsMay not resemble real useRealism matters more than breadth
Expert-written casesDeep and targetedSlow and expensiveDomain is broad
Public benchmark as coverageFree comparisonWrong distributionProduct is specialised
Trade-off mapInteractive

Good for

Representative

Cost

Bias toward current behaviour, privacy work

Fails when

New capability has no traffic yet

  1. New capability has no traffic yet
  2. Realism matters more than breadth
  3. Domain is broad
  4. Product is specialised

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Sample from production traffic is good for Representative, costs Bias toward current behaviour, privacy work, and fails when New capability has no traffic yet. Synthetic edge cases is good for Covers rare and hostile inputs, costs May not resemble real use, and fails when Realism matters more than breadth. Expert-written cases is good for Deep and targeted, costs Slow and expensive, and fails when Domain is broad. Public benchmark as coverage is good for Free comparison, costs Wrong distribution, and fails when Product is specialised

Failure modes

  • Cluster blindness. A whole intent category has no cases and fails silently in production. Debug: join production intents against eval case tags and list clusters with zero cases.
  • Coverage by count. Many near-duplicate cases raise the total without adding new behaviour. Debug: cluster the eval set itself and count distinct clusters.
  • Frozen distribution. The suite reflects last year's traffic. Debug: compare the eval cluster mix with the last month of production requests.

In production

Report coverage as a live table against traffic clusters. Alert when a cluster's production share grows past its eval share, which means the suite is falling behind usage.

Common misconceptions

People think: More eval cases means better coverage.

Actually: Duplicates inflate counts, while coverage is about distinct behaviour.

People think: A benchmark covers the general case.

Actually: It covers its own construction, which is usually narrower and cleaner than your traffic.

Interview angle

Define coverage against a sampling frame taken from real usage and count clusters. A strong answer treats a missing cluster as the finding and mentions drift over time. Equating coverage with the number of cases, with no account of how the set was chosen, is the weak response.

Sources