Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 23 of 38

Dashboards

0 of 38 complete

Why it matters

Inference systems fail in ways generic service dashboards do not show: tokens stream slowly while request latency looks fine, a provider starts returning shorter answers, or cache pressure raises wait time without raising errors. A useful dashboard encodes those specific failure shapes and links each panel to the query that can explain it. Without that, an incident review starts by exporting logs and reconstructing what the graphs meant.

Mental model

A dashboard is a cockpit, not an engine bay: a few instruments that tell you whether to land, and a way to open the panel behind each one. The analogy breaks because cockpit instruments are standard, while AI metrics are defined by your product's notion of a good answer.

How it works

Mechanism, step by stepInteractive
  1. 1List the five questions asked during the last three incidents and design a panel for each.
  2. 2Put traffic, error rate, time-to-first-token, inter-token latency, queue wait, cache utilisation, cost per unit and a quality proxy on the first screen.
  3. 3Give every rate a denominator, so a count of failures is paired with the requests it came from.
  4. 4Annotate deploys, prompt versions and provider incidents on the same time axis.
  5. 5Link each panel to a trace or log query pre-filtered to the same window.
  6. 6Review the dashboard after each incident and delete panels nobody opened.

Step 1 of 6

List the five questions asked during the last three incidents and design a panel for each.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. List the five questions asked during the last three incidents and design a panel for each. 2. Put traffic, error rate, time-to-first-token, inter-token latency, queue wait, cache util… 3. Give every rate a denominator, so a count of failures is paired with the requests it came… 4. Annotate deploys, prompt versions and provider incidents on the same time axis. 5. Link each panel to a trace or log query pre-filtered to the same window. 6. Review the dashboard after each incident and delete panels nobody opened.

  1. List the five questions asked during the last three incidents and design a panel for each.
  2. Put traffic, error rate, time-to-first-token, inter-token latency, queue wait, cache utilisation, cost per unit and a quality proxy on the first screen.
  3. Give every rate a denominator, so a count of failures is paired with the requests it came from.
  4. Annotate deploys, prompt versions and provider incidents on the same time axis.
  5. Link each panel to a trace or log query pre-filtered to the same window.
  6. Review the dashboard after each incident and delete panels nobody opened.

Trade-offs

ChoiceGood forCostFails when
One dense screenEverything visible at onceHard to read under pressureThe audience is broad
Separate per-audience viewsEach team sees what it acts onDuplication and driftNobody owns the shared definitions
Quality panel from labelsTrustworthy signalLabelling cost and lagTraffic changes faster than labels arrive
Quality panel from proxiesCheap and immediateCan move for reasons unrelated to qualityThe proxy is the only evidence
Trade-off mapInteractive

Good for

Everything visible at once

Cost

Hard to read under pressure

Fails when

The audience is broad

  1. The audience is broad
  2. Nobody owns the shared definitions
  3. Traffic changes faster than labels arrive
  4. The proxy is the only evidence

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: One dense screen is good for Everything visible at once, costs Hard to read under pressure, and fails when The audience is broad. Separate per-audience views is good for Each team sees what it acts on, costs Duplication and drift, and fails when Nobody owns the shared definitions. Quality panel from labels is good for Trustworthy signal, costs Labelling cost and lag, and fails when Traffic changes faster than labels arrive. Quality panel from proxies is good for Cheap and immediate, costs Can move for reasons unrelated to quality, and fails when The proxy is the only evidence

Failure modes

  • Average latency panel. Averages stay flat while a growing share of users waits far longer. Debug: add p50, p95 and p99 for time-to-first-token on the same chart.
  • Counts without denominators. Error counts rise simply because traffic rose. Debug: recompute every panel as a ratio against requests in the same window.
  • Quality proxy gaming. The proxy improves while user complaints rise. Debug: correlate the proxy with a labelled sample monthly and replace it if agreement drops.

In production

Treat dashboards as code: version the queries and review changes. Alert from SLOs rather than from panels, and keep a per-provider and per-model breakdown so a regression can be localised without writing a new query.

Common misconceptions

People think: More panels mean more observability.

Actually: Panels compete for attention; a dashboard earns its space by answering a question someone acts on.

People think: A score from a model grader is a quality metric.

Actually: It is a proxy that must be calibrated against human labels, or it measures style rather than correctness.

Interview angle

Derive the panels from the questions asked during incidents, including the split between time-to-first-token and inter-token latency. Insist on denominators and deploy markers. Listing tools or metrics without saying what decision each panel supports is the weak form.

Sources