Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 34 of 38

SLOs

0 of 38 complete

Why it matters

Availability targets alone cannot describe an assistant that answers every request quickly and incorrectly. A quality indicator requires a labelled sample, a judge with known agreement, or a deterministic check such as citation validity, and each has a cost and a lag. The objective exists to make the trade explicit: how much wrongness is acceptable, measured over what window, and what the team does when the budget is spent.

Mental model

An objective is a thermostat setting: it does not heat the room, it decides when the heating turns on. The analogy breaks because a thermostat measures one number, while a good AI service needs several targets that can conflict, such as speed against depth of reasoning.

How it works

The tail spends the budgetInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. A modelled latency distribution cut at a 2 s objective, with the requests past the cut consumed from a 30-day error budget. The distribution is log-normal and every number is generated by the stated model; the shape — a thin tail that widens with load — is the claim.

What to notice. At ×1 traffic 1.7% of requests miss the 2 s objective and a 2% budget survives the month. At ×2 traffic 8.0% miss it, p95 has crossed 2 s, and the budget is spent by day 8. The density's peak barely moves while its tail crosses the line: the objective fails before the typical request feels slow.

Text version

A density curve of request latency from 0 to 6,000 ms with a vertical 2,000 ms SLO line. The area to its right is filled red, labelled requests over the SLO, and annotated 1.7% of requests past this line; the rest of the curve is teal. Both axes are labelled: latency in milliseconds across the bottom, share of requests per 200 ms bucket up the left. Below, a second panel plots the error budget remaining from 100% at day 0 to 17% at day 30 on its own day axis. A traffic slider from ×1 to ×2.5 redraws both panels: at ×2 the p95 is 2,320 ms, the annotation reads 8.0% of requests past this line, and the budget reaches zero on day 8. Three statistics under the legend report p95, the over-SLO share and how long the 2% budget lasts.

  1. Pick the user journey and the moment of pain, then define the event that counts as good for it.
  2. Write the indicator as good events divided by valid events, and specify what excludes a request from the denominator.
  3. Set the target from measured behaviour and user tolerance, not from an aspiration.
  4. Add a quality indicator with a stated sampling rate, grader and agreement check against human labels.
  5. Attach an error-budget policy that says what freezes when the budget is exhausted.
  6. Alert on burn rate over a short and a long window, so both sudden outages and slow degradation page someone.

Trade-offs

ChoiceGood forCostFails when
Latency target onlyEasy to measure and alertMisses wrong but fast answersThe product's value is answer correctness
Quality target from human labelsTrustworthyExpensive and delayedTraffic changes daily
Quality target from a model judgeCheap and fastJudge drift and biasThe judge was never validated on your domain
Strict targetProtects usersBudget exhausts and blocks releasesThe system is still stabilising
Trade-off mapInteractive

Good for

Easy to measure and alert

Cost

Misses wrong but fast answers

Fails when

The product's value is answer correctness

  1. The product's value is answer correctness
  2. Traffic changes daily
  3. The judge was never validated on your domain
  4. The system is still stabilising

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Latency target only is good for Easy to measure and alert, costs Misses wrong but fast answers, and fails when The product's value is answer correctness. Quality target from human labels is good for Trustworthy, costs Expensive and delayed, and fails when Traffic changes daily. Quality target from a model judge is good for Cheap and fast, costs Judge drift and bias, and fails when The judge was never validated on your domain. Strict target is good for Protects users, costs Budget exhausts and blocks releases, and fails when The system is still stabilising

Failure modes

  • Unvalidated judge. The quality indicator reports success while users complain. Debug: compare judge verdicts with a human-labelled sample and report agreement per category.
  • Denominator drift. A change in what counts as a valid request makes the ratio jump without any behaviour change. Debug: chart numerator and denominator separately and diff the exclusion rule in version control.
  • Budget without policy. The budget is exhausted and nothing changes, so the target becomes decorative. Debug: check that a burn-rate alert has an owner and a documented action.

In production

Review objectives quarterly against actual user complaints, and keep the sampling cost of quality measurement visible in the budget. Page on burn rate, report on budget remaining, and record every exception to the policy.

Common misconceptions

People think: An objective is a promise to users.

Actually: It is an internal operating target; the external promise is usually looser and stated in a contract.

People think: Quality cannot be a target because it is subjective.

Actually: It can be measured as a rate over a defined judgement, provided the judgement is written down and its agreement is checked.

Interview angle

Define the good-event ratio, add a quality indicator with a stated grader, and attach a budget policy to the number. Explain multi-window burn alerts. An availability percentage with no numerator, denominator or consequence is not an objective.

Sources