Why it matters
An AI feature can change behaviour without a code change: the provider updates an alias, an index is rebuilt, a prompt is edited in a database, a decoding parameter is tuned. When quality drops, the first question is what differed, and that is unanswerable unless the exact combination was recorded at request time. Versioning makes behaviour reproducible, which is also the precondition for a canary and for a rollback.
Mental model
It is a recipe card with a revision number: the dish is reproducible only if the card, the ingredients and the oven setting are all identified. The analogy breaks because a model can be updated in place by someone else, so the ingredient changes without a new card.
How it works
Event · step 1 of 6
Bundle A, then the alias moves
At 15 d
Bundle A (v41, prompt v7, idx-104) served until the pinned deploy on day 15 and the bundle id went out on every response. On day 14 the floating alias moved to v42 with no deploy; the deliberate pinned deploy followed on day 15, starting bundle B (v42 · p7 · idx-104). If responses recorded the alias rather than the revision, that one-day gap would be invisible.
On screen at this event
- → v41
- → v42
- v41 pinned
- v42 pinned
- v7
- idx-104
- A
- B
- alias moves
Next at 21 d: Prompt v8 ships
- Bundle
- B
- Prompt
- v7
- Index
- idx-104
- Skew
- no
- Sample errors
- 0.3%
Figure. A synthetic six-week release history: the provider alias moves on day 14, pinned v42 follows on day 15, prompt v8 ships on day 21 and index idx-105 catches up on day 33. Bundle C runs for twelve days against an index that has no field its prompt expects.
What to notice. Pinning the model alone would not prevent this. The break is prompt v8 reading idx-104, and the recorded bundle is the only artefact that shows both at once. Record model, prompt, index, tool schema and decoding parameters as one bundle per response, keep old versions readable through an overlap window, and gate prompt–index compatibility at release. The floating alias is a separate unversioned dependency with its own lane. Tool schema and decoding parameters belong in the same bundle but are not separate lanes here.
Text version
Five lanes over 42 days. The provider alias points at v41 for 0–14, v42 for 14–28 and v43 for 28–42. The pinned model is v41 for 0–15 and v42 for 15–42. Prompt v7 runs 0–21 and v8 from 21. Index idx-104 runs 0–33 and idx-105 from 33. The serving bundle lane is labelled A–D; the step detail gives each bundle as model · prompt · index: A v41·p7·104 for 0–15, B v42·p7·104 for 15–21, C v42·p8·104 in red from 21 to 33, and D v42·p8·105 from 33 to 42. A red phase band labelled version skew spans 21–33. Markers read alias moves at 14, prompt v8 at 21 and index catches up at 33. Counters: bundles B, C, C, D, D, D; skew no, yes, yes, no, no, no; sampled errors 0.3, 0.4, 2.6, 0.4, 0.4, 0.4 percent.
- Enumerate the artefacts that affect output: weights or provider model id, template, index snapshot, tool definitions, decoding parameters.
- Give each artefact an immutable identifier, and store the combination as one bundle identifier per release.
- Pin provider model ids explicitly rather than using an alias that moves.
- Return the bundle identifier in response metadata and attach it to traces, metrics and evaluation runs.
- Keep old prompt and index versions readable during an overlap window so rollback does not require reconstruction.
- Publish a deprecation schedule for each artefact and test the migration before the deadline.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Pin exact model ids | Reproducible behaviour | Misses provider improvements and can break at deprecation | The provider retires the pinned version |
| Follow a floating alias | Always current | Behaviour changes without a deploy | Quality regressions must be explainable |
| Version the prompt in code | Reviewed and deployed like code | Slow iteration for content changes | Non-engineers need to edit prompts |
| Version the prompt in a database | Fast iteration and experimentation | Drift from review and testing | Changes bypass the evaluation gate |
Good for
Reproducible behaviour
Cost
Misses provider improvements and can break at deprecation
Fails when
The provider retires the pinned version
- The provider retires the pinned version
- Quality regressions must be explainable
- Non-engineers need to edit prompts
- Changes bypass the evaluation gate
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Pin exact model ids is good for Reproducible behaviour, costs Misses provider improvements and can break at deprecation, and fails when The provider retires the pinned version. Follow a floating alias is good for Always current, costs Behaviour changes without a deploy, and fails when Quality regressions must be explainable. Version the prompt in code is good for Reviewed and deployed like code, costs Slow iteration for content changes, and fails when Non-engineers need to edit prompts. Version the prompt in a database is good for Fast iteration and experimentation, costs Drift from review and testing, and fails when Changes bypass the evaluation gate
Failure modes
- Floating alias regression. Answers change overnight with no deploy and no diff. Debug: compare the provider-reported model revision in responses before and after the change.
- Index and prompt mismatch. A new prompt assumes fields the current index does not contain. Debug: store the index snapshot id with each request and assert compatibility at startup.
- Unrecorded prompt edit. A production prompt is edited directly and no trace shows the previous text. Debug: require prompt writes to go through a versioned store and diff the active version against the last deployed one.
In production
Report the bundle identifier on every response and in every metric, and alert when more than one bundle is serving outside a canary. Keep a rollback target for each artefact, not just for the code.
Common misconceptions
People think: Versioning the model is enough.
Actually: Prompt, retrieval and decoding parameters change behaviour as much as weights do, and often more often.
People think: A provider model alias is stable because the name is stable.
Actually: Aliases commonly point at a moving target, which is why exact revisions are recorded for reproducibility.
Interview angle
Enumerate the artefacts, define a bundle identifier and put it on every response. Explain why an alias is an unversioned dependency. Saying the code is in git, and considering that sufficient, is the miss.
Sources
- Semantic Versioning — a shared vocabulary for version identifiers and compatibility.
- MLflow, Model Registry — versioned model artefacts and stage transitions.
- OpenAI, Deprecations — how provider model retirements are announced.