Why it matters
A conventional service is defined by its code. Ship the same commit to the same image and you get the same behaviour. An LLM application has at least four inputs that can change independently: the model behind an API alias, the prompt, the retrieval index, and the tool schemas the model is allowed to call. Any one of them can change the output distribution without a code diff.
That breaks the assumptions CI was built on. A test suite that asserts exact strings will fail constantly and teach the team to ignore it. A deploy that reverts the container does not revert a provider-side model update or an index rebuild that ran this morning. And unlike a crash, a quality regression is silent: the service returns 200s, latency looks normal, and only the answers are worse.
LLMOps is the set of practices that make those changes observable, comparable and reversible. It is closer to release engineering for a data product than to training infrastructure, which is why it borrows more from SLOs and tracing than from experiment tracking.
Mental model
A restaurant kitchen supplied by a mill that silently changes the grind, with a recipe the head chef adjusts verbally each morning. You cannot taste every plate, so you keep three things: a written spec for what the dish should be, a batch record of exactly which flour and which recipe version went out on which night, and a tasting before service.
The analogy breaks at the feedback signal. A bad dinner produces a complaint you can count. A bad answer often looks fluent, plausible and correctly formatted, so complaint volume is a lagging and biased measure. You need labelled evaluation, not customer anger, as the primary sensor.
How it works
- 1Write the task contract. One paragraph on what the system is for, plus the quality metric that decides success, plus the failure that must never happen. This becomes the anchor for every later decision.
- 2Build the eval set before the first release. A few hundred labelled cases drawn from real traffic, including the boring ones. It is the only artifact that lets you compare two releases honestly. See datasets and golden sets.
- 3Assemble the release bundle. Model id and resolved version, prompt file with a content hash, index alias and index commit, embedding model, tool schema version, decoding parameters, and the eval set version used for the gate.
- 4Gate offline. Run the eval set against the candidate bundle and the current bundle, on the same inputs, and compare per-case rather than only in aggregate. Aggregates hide a win on easy cases and a loss on the hard tail.
- 5Shadow, then canary. Send a copy of live traffic to the candidate without showing the result, score both offline where labels exist, then expose a small traffic share.
- 6Watch online signals. Task success proxies (escalation rate, edit rate, retry rate), latency percentiles, cost per request, refusal rate, and retrieval hit rate. These catch what the eval set does not contain.
Step 1 of 6
Write the task contract. One paragraph on what the system is for, plus the quality metric that decides success, plus the failure that must never happen. This becomes the anchor for every later decision.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Write the task contract. One paragraph on what the system is for, plus the quality metric… 2. Build the eval set before the first release. A few hundred labelled cases drawn from real… 3. Assemble the release bundle. Model id and resolved version, prompt file with a content ha… 4. Gate offline. Run the eval set against the candidate bundle and the current bundle, on th… 5. Shadow, then canary. Send a copy of live traffic to the candidate without showing the res… 6. Watch online signals. Task success proxies (escalation rate, edit rate, retry rate), late…
- Write the task contract. One paragraph on what the system is for, plus the quality metric that decides success, plus the failure that must never happen. This becomes the anchor for every later decision.
- Build the eval set before the first release. A few hundred labelled cases drawn from real traffic, including the boring ones. It is the only artifact that lets you compare two releases honestly. See datasets and golden sets.
- Assemble the release bundle. Model id and resolved version, prompt file with a content hash, index alias and index commit, embedding model, tool schema version, decoding parameters, and the eval set version used for the gate.
- Gate offline. Run the eval set against the candidate bundle and the current bundle, on the same inputs, and compare per-case rather than only in aggregate. Aggregates hide a win on easy cases and a loss on the hard tail.
- Shadow, then canary. Send a copy of live traffic to the candidate without showing the result, score both offline where labels exist, then expose a small traffic share.
- Watch online signals. Task success proxies (escalation rate, edit rate, retry rate), latency percentiles, cost per request, refusal rate, and retrieval hit rate. These catch what the eval set does not contain.
- Close the loop. Route thumbs-down, human edits and escalations back into the eval set after review. A feedback loop with no curation just accumulates noise.
- Keep rollback complete. Rolling back the prompt while the new index stays live is a half-rollback. Every artifact in the bundle needs a previous value that is still queryable.
- Release unit
- Model + prompt + index + tools
- Primary gate
- Labelled eval set
- Second gate
- Shadow traffic
- Third gate
- Canary share
- Rollback scope
- All four artifacts
- Hardest signal
- Silent quality drift
The release bundle in practice
The manifest below is the artifact a reviewer reads during an incident. It answers "what was live at 14:05" without guessing.
release: support-copilot-2026-11-03
model:
provider: <provider>
id: <model-id>
resolved_version: <version string returned by the API>
prompt:
path: prompts/answer.v7.md
sha256: <hash of the file contents>
retrieval:
index_alias: kb-live
index_commit: <ingest run id>
embedding_model: <model-id>@<version>
decoding:
temperature: 0.0
max_output_tokens: 800
tools:
schema_version: 12
gate:
eval_set: support-golden-v4
min_task_success: 0.90 # your agreed floor, not a universal numberTwo fields do most of the work. resolved_version records what the provider actually served, because aliases move. index_commit records which corpus the answers were grounded in, because a re-index is a behaviour change.
Sizing a canary
Canary detection is a sample-size problem, and the arithmetic is unforgiving at low volume. At 200,000 requests per day, a 5% canary receives 10,000 requests per day, or about 0.12 requests per second. Accumulating 1,000 canary observations takes roughly 1,000 / 0.12 ≈ 8,300 seconds, or 2.3 hours.
If the baseline defect rate is 2% and the candidate is 4%, those 1,000 observations contain about 20 versus 40 defects. That is a signal. If the candidate is 2.2%, the difference is 20 versus 22 defects, which is noise. Below a few thousand canary requests, prefer shadow evaluation on replayed traffic, where you can score every request cheaply, over a live canary you cannot read.
Trade-offs
| Release path | Buys | Costs | Fails when |
|---|---|---|---|
| Prompt-only change, eval gate | Fast, cheap, easy rollback | Hours of eval per change | Prompt is compensating for missing retrieval |
| Model version upgrade | Quality or cost step change | Full re-eval, new failure shapes | Eval set is too small to resolve the difference |
| Index rebuild | Fresh knowledge, better chunks | Hours of embedding, alias swap | Embedding model changed without re-embedding everything |
| Fine-tune | Consistent format and tone at scale | Training pipeline, frozen knowledge | You needed facts, not behaviour |
| No gate, ship on vibes | Maximum speed | Undetected regressions | Traffic distribution differs from the demo |
Good for
Fast, cheap, easy rollback
Cost
Hours of eval per change
Fails when
Prompt is compensating for missing retrieval
- Prompt is compensating for missing retrieval
- Eval set is too small to resolve the difference
- Embedding model changed without re-embedding everything
- You needed facts, not behaviour
- Traffic distribution differs from the demo
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Prompt-only change, eval gate is good for Fast, cheap, easy rollback, costs Hours of eval per change, and fails when Prompt is compensating for missing retrieval. Model version upgrade is good for Quality or cost step change, costs Full re-eval, new failure shapes, and fails when Eval set is too small to resolve the difference. Index rebuild is good for Fresh knowledge, better chunks, costs Hours of embedding, alias swap, and fails when Embedding model changed without re-embedding everything. Fine-tune is good for Consistent format and tone at scale, costs Training pipeline, frozen knowledge, and fails when You needed facts, not behaviour. No gate, ship on vibes is good for Maximum speed, costs Undetected regressions, and fails when Traffic distribution differs from the demo
Failure modes
- The floating alias. The config says
latestand the provider moves it. Debug by logging the resolved model version on every response, then diffing that field across a week; if it changed without a release, you have drift, not a bug. - Prompt edited outside the repository. Someone tunes wording in a vendor console or an admin UI and the file hash no longer matches production. Debug by emitting the prompt hash as a span attribute at request time and alerting when it differs from the deployed manifest.
- Eval against the wrong index. The gate passes because it ran against a snapshot from last week. Debug by recording the index commit in the eval report and refusing to compare reports with different commits.
- Half rollback. The prompt reverts, the index does not, and quality stays bad while the team argues. Debug by checking that the previous index alias still resolves and still has vectors from the matching embedding model before declaring the rollback complete.
In production
On-call needs four panels, not forty: release manifest diffs, eval score trend per release, index freshness (age of the newest document and lag of the ingest queue), and spend against budget. See AI dashboards for the shape.
Enforce a change budget: one artifact per release where possible. Changing the model and the chunker together produces a quality jump you cannot attribute, and an incident you cannot bisect.
Keep the eval suite fast enough to run on every change (minutes, not hours) and a slower, larger suite for weekly runs. The fast suite is the gate; the slow suite is the map.
Common misconceptions
People think: LLMOps is MLOps with a bigger model.
Actually: There is usually no training loop. The recurring work is evaluation, prompt and index versioning, tracing and cost control. MLOps tooling for experiments and feature stores rarely maps onto it.
People think: A good offline eval means you can ship.
Actually: Offline eval measures the distribution you sampled. It cannot see a new user segment, a shifted query mix or a provider change that arrived after the eval ran. Shadow and canary traffic exist for that gap.
Interview angle
Expect "how do you ship a prompt change safely?" A weak answer describes a code review. A strong answer names the bundle, the offline gate with a fixed eval set, shadow traffic, a canary with a measurable decision rule, and a rollback that includes the index. Then it names the cost: two hours of eval for a one-line change, and why that is still cheaper than a silent regression. See system design process and the production interview category.
Sources
- Sculley et al., Hidden Technical Debt in Machine Learning Systems — why entangled, multi-artifact systems decay.
- Google Cloud, MLOps: Continuous delivery and automation pipelines — pipeline and gate vocabulary.
- OpenTelemetry GenAI semantic conventions — the fields worth recording per model call.
- NIST, AI Risk Management Framework — governance vocabulary for release sign-off.
Related
- AI SLOs — what to promise and how to measure it.
- Observability — traces and metrics that make drift visible.
- Versioning — pinning artifacts that live outside your repository.
- Canary and rollback — the mechanics of a reversible release.
- Evaluation datasets — the gate's raw material.