Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 32 of 38

Drift

0 of 38 complete

Why it matters

A model is fixed at training time but the world it answers about is not. Query mix shifts, a new product changes what users ask, a provider updates the served checkpoint, and a retrieval index accumulates documents that change the context. Each can degrade quality while every infrastructure metric stays green, which is why drift detection compares distributions and sampled quality against a baseline instead of waiting for an error.

Mental model

It is a scale that drifts after being moved: the readings are consistent with each other but no longer with the standard weight. The analogy breaks because you cannot recalibrate a model by zeroing it; the reference has to be rebuilt from fresh labelled data.

How it works

Mechanism, step by stepInteractive
  1. 1Define a reference window and store the input and output distributions you will compare against it.
  2. 2Monitor cheap input signals first: prompt length, language, topic mix, retrieval score distribution and refusal rate.
  3. 3Track output signals such as length, format validity and tool-call rate, which move before quality metrics do.
  4. 4Run a sampled quality evaluation on a labelled slice at a fixed cadence, and chart it against the baseline.
  5. 5Set thresholds with hysteresis and a minimum duration, so a single busy hour does not alert.
  6. 6Attach each alert to a decision: refresh the index, re-tune the prompt, roll back the model or collect new labels.

Step 1 of 6

Define a reference window and store the input and output distributions you will compare against it.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Define a reference window and store the input and output distributions you will compare a… 2. Monitor cheap input signals first: prompt length, language, topic mix, retrieval score di… 3. Track output signals such as length, format validity and tool-call rate, which move befor… 4. Run a sampled quality evaluation on a labelled slice at a fixed cadence, and chart it aga… 5. Set thresholds with hysteresis and a minimum duration, so a single busy hour does not ale… 6. Attach each alert to a decision: refresh the index, re-tune the prompt, roll back the mod…

  1. Define a reference window and store the input and output distributions you will compare against it.
  2. Monitor cheap input signals first: prompt length, language, topic mix, retrieval score distribution and refusal rate.
  3. Track output signals such as length, format validity and tool-call rate, which move before quality metrics do.
  4. Run a sampled quality evaluation on a labelled slice at a fixed cadence, and chart it against the baseline.
  5. Set thresholds with hysteresis and a minimum duration, so a single busy hour does not alert.
  6. Attach each alert to a decision: refresh the index, re-tune the prompt, roll back the model or collect new labels.

Trade-offs

ChoiceGood forCostFails when
Input distribution monitoringCheap, immediate, no labelsCannot tell you whether answers got worseThe input mix is stable but quality fell
Sampled human evaluationTrustworthy quality signalSlow and expensiveTraffic changes faster than the label cycle
Model-judge scoringFast and cheap coverageJudge drift and self-preferenceThe judge shares the failure it is scoring
Provider version pinningRemoves one drift sourceMisses upstream improvementsThe provider deprecates the pinned version
Trade-off mapInteractive

Good for

Cheap, immediate, no labels

Cost

Cannot tell you whether answers got worse

Fails when

The input mix is stable but quality fell

  1. The input mix is stable but quality fell
  2. Traffic changes faster than the label cycle
  3. The judge shares the failure it is scoring
  4. The provider deprecates the pinned version

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Input distribution monitoring is good for Cheap, immediate, no labels, costs Cannot tell you whether answers got worse, and fails when The input mix is stable but quality fell. Sampled human evaluation is good for Trustworthy quality signal, costs Slow and expensive, and fails when Traffic changes faster than the label cycle. Model-judge scoring is good for Fast and cheap coverage, costs Judge drift and self-preference, and fails when The judge shares the failure it is scoring. Provider version pinning is good for Removes one drift source, costs Misses upstream improvements, and fails when The provider deprecates the pinned version

Failure modes

  • Alert without an owner. Drift fires weekly and nobody acts, so the signal is ignored during a real regression. Debug: check that every drift alert has a runbook link and a named responder.
  • Seasonality false positive. A weekly traffic pattern trips the threshold every Monday. Debug: compare against the same weekday in the reference window rather than the previous day.
  • Proxy divergence. Output length is stable while correctness falls. Debug: correlate the proxy with labelled quality monthly and retire it if agreement drops.

In production

Keep the reference window versioned alongside the evaluation set, and re-baseline deliberately after an intended change. Report drift next to quality, because a distribution shift with stable quality is information, not an incident.

Common misconceptions

People think: Drift means the model is broken.

Actually: It means the data or the world moved; the model is doing what it was trained to do on inputs it no longer sees.

People think: Monitoring inputs is enough.

Actually: Input shift tells you where to look; only outcome measurement tells you whether the product got worse.

Interview angle

Separate input, output and outcome drift and say what evidence each needs. Name a reference window, a threshold with hysteresis and a decision attached to the alert. Saying you retrain when accuracy drops, without a way to notice, is the gap.

Sources