Why it matters
Inference systems fail in ways generic service dashboards do not show: tokens stream slowly while request latency looks fine, a provider starts returning shorter answers, or cache pressure raises wait time without raising errors. A useful dashboard encodes those specific failure shapes and links each panel to the query that can explain it. Without that, an incident review starts by exporting logs and reconstructing what the graphs meant.
Mental model
A dashboard is a cockpit, not an engine bay: a few instruments that tell you whether to land, and a way to open the panel behind each one. The analogy breaks because cockpit instruments are standard, while AI metrics are defined by your product's notion of a good answer.
How it works
- 1List the five questions asked during the last three incidents and design a panel for each.
- 2Put traffic, error rate, time-to-first-token, inter-token latency, queue wait, cache utilisation, cost per unit and a quality proxy on the first screen.
- 3Give every rate a denominator, so a count of failures is paired with the requests it came from.
- 4Annotate deploys, prompt versions and provider incidents on the same time axis.
- 5Link each panel to a trace or log query pre-filtered to the same window.
- 6Review the dashboard after each incident and delete panels nobody opened.
Step 1 of 6
List the five questions asked during the last three incidents and design a panel for each.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. List the five questions asked during the last three incidents and design a panel for each. 2. Put traffic, error rate, time-to-first-token, inter-token latency, queue wait, cache util… 3. Give every rate a denominator, so a count of failures is paired with the requests it came… 4. Annotate deploys, prompt versions and provider incidents on the same time axis. 5. Link each panel to a trace or log query pre-filtered to the same window. 6. Review the dashboard after each incident and delete panels nobody opened.
- List the five questions asked during the last three incidents and design a panel for each.
- Put traffic, error rate, time-to-first-token, inter-token latency, queue wait, cache utilisation, cost per unit and a quality proxy on the first screen.
- Give every rate a denominator, so a count of failures is paired with the requests it came from.
- Annotate deploys, prompt versions and provider incidents on the same time axis.
- Link each panel to a trace or log query pre-filtered to the same window.
- Review the dashboard after each incident and delete panels nobody opened.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| One dense screen | Everything visible at once | Hard to read under pressure | The audience is broad |
| Separate per-audience views | Each team sees what it acts on | Duplication and drift | Nobody owns the shared definitions |
| Quality panel from labels | Trustworthy signal | Labelling cost and lag | Traffic changes faster than labels arrive |
| Quality panel from proxies | Cheap and immediate | Can move for reasons unrelated to quality | The proxy is the only evidence |
Good for
Everything visible at once
Cost
Hard to read under pressure
Fails when
The audience is broad
- The audience is broad
- Nobody owns the shared definitions
- Traffic changes faster than labels arrive
- The proxy is the only evidence
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: One dense screen is good for Everything visible at once, costs Hard to read under pressure, and fails when The audience is broad. Separate per-audience views is good for Each team sees what it acts on, costs Duplication and drift, and fails when Nobody owns the shared definitions. Quality panel from labels is good for Trustworthy signal, costs Labelling cost and lag, and fails when Traffic changes faster than labels arrive. Quality panel from proxies is good for Cheap and immediate, costs Can move for reasons unrelated to quality, and fails when The proxy is the only evidence
Failure modes
- Average latency panel. Averages stay flat while a growing share of users waits far longer. Debug: add p50, p95 and p99 for time-to-first-token on the same chart.
- Counts without denominators. Error counts rise simply because traffic rose. Debug: recompute every panel as a ratio against requests in the same window.
- Quality proxy gaming. The proxy improves while user complaints rise. Debug: correlate the proxy with a labelled sample monthly and replace it if agreement drops.
In production
Treat dashboards as code: version the queries and review changes. Alert from SLOs rather than from panels, and keep a per-provider and per-model breakdown so a regression can be localised without writing a new query.
Common misconceptions
People think: More panels mean more observability.
Actually: Panels compete for attention; a dashboard earns its space by answering a question someone acts on.
People think: A score from a model grader is a quality metric.
Actually: It is a proxy that must be calibrated against human labels, or it measures style rather than correctness.
Interview angle
Derive the panels from the questions asked during incidents, including the split between time-to-first-token and inter-token latency. Insist on denominators and deploy markers. Listing tools or metrics without saying what decision each panel supports is the weak form.
Sources
- Google, SRE Book: Monitoring Distributed Systems — the four golden signals and why averages mislead.
- Google, SRE Workbook: Monitoring — dashboard design from user-facing symptoms.