Why it matters
Prompt edits are often made directly in a dashboard or a string constant, with no history and no link to the requests they served. When quality drops, the team cannot tell whether the prompt, the model, the retrieval index or the traffic changed. Versioning is the minimum instrumentation for attribution: an immutable identifier per prompt state, attached to every request and every evaluation run.
Mental model
Prompt versioning is a lab notebook: each experiment has a dated entry and a result. The metaphor breaks because the notebook is also the product, so an old entry may still be running in production for some users, and two entries can be live at the same time.
How it works
Event · step 1 of 7
9 d · v8 passed the suite
At 9 d
Prompt v8 is an immutable registry entry, and its evaluation finished on day 9 against the same suite that scored v7. No traffic has moved yet; the pass is a gate, not a release.
On screen at this event
- v7
- eval
- v7 100%
Next at 10 d: 10 d · v8 takes 10%
- Live
- v7
- Logged with id
- 100%
- Evaluation
- v8 pass
- Model
- m-41
Figure. A synthetic 24-day history: v8 passes evaluation, canaries at 10% on day 10, reaches 50% on day 12, is rolled back on day 13, and v9 ships on day 18 after the regression is reproduced. Shares, dates and scores are illustrative.
What to notice. The canary is only attributable because every request records the prompt id. Between days 11 and 12 the log has no prompt id, so exactly the window where quality moved cannot be reconstructed from traffic. Prompt and model revisions change independently, which is why both belong on the request record; versioning without the evaluation matrix only tells you what changed.
Text version
Five lanes over 24 days: registry, traffic, model, request log and evaluation. Registry versions are v7 0 to 13, v8 10 to 13 stacked on its own row beneath v7, and v9 18 to 24. Traffic is v7 at 100% to day 10; a second-row canary bar for v8 runs 10 to 13 while v7 keeps serving, then v7 alone to 18 and v9 after. The step details give the shares: 10% on day 10 and 50% on day 12. The model is m-41 to day 16 and m-42 after. The request log records a prompt id except for a red gap from day 11 to 12 labelled no id. Evaluation runs are v8 8.4 to 9, v8 again 13.2 to 14.5 and v9 16.4 to 17.8. Markers read v8 10% at 10, v8 50% at 12, rollback at 13 and v9 live at 18. Counters: live v7, v7 90% with v8 10%, a 50/50 split, v7, v7, v9, v9; logged with id 100% except 0% on the full gap day at 12; evaluation v8 pass, pending, v8 fail, v9 pass; model m-41 until day 16.
- Store prompts as files or registry entries with an immutable id, not as strings edited in place.
- Record the prompt id and model id on every request and every evaluation result.
- Diff versions by section so reviewers can see which instruction changed.
- Gate a version change on the evaluation suite and a canary share of traffic.
- Keep the previous version deployable, with a rollback that does not require a code release.
- Link each version to the incident or request that motivated it, so the history explains itself.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Prompts in code | Reviewed with the application | Deploy required for every wording change | Non-engineers need to iterate |
| Prompt registry | Fast iteration and rollback | Another service and access control surface | The team is small and deploys often |
| Version per request | Full attribution | Storage and cardinality | Traffic is very high |
| Single live prompt | Simple | No history and no rollback | More than one person edits it |
Good for
Reviewed with the application
Cost
Deploy required for every wording change
Fails when
Non-engineers need to iterate
- Non-engineers need to iterate
- The team is small and deploys often
- Traffic is very high
- More than one person edits it
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Prompts in code is good for Reviewed with the application, costs Deploy required for every wording change, and fails when Non-engineers need to iterate. Prompt registry is good for Fast iteration and rollback, costs Another service and access control surface, and fails when The team is small and deploys often. Version per request is good for Full attribution, costs Storage and cardinality, and fails when Traffic is very high. Single live prompt is good for Simple, costs No history and no rollback, and fails when More than one person edits it
Failure modes
- Unrecorded edit. A dashboard change alters behaviour and no request log shows it. Debug: compare the prompt hash in logs with the registry version for the same period.
- Version without content. Two different prompts share a version label after a partial deploy. Debug: hash the rendered prompt and store the hash with the version.
- Rollback that is not tested. The old version no longer works with the new model. Debug: keep the previous version in the evaluation matrix and run it on every model upgrade.
In production
Attach prompt and model versions to traces, and alert when the live version differs from the reviewed one. Canary a new version on a small traffic share before full rollout.
Common misconceptions
People think: Prompt changes are too small to version.
Actually: Small wording changes move accuracy; without a version they are indistinguishable from noise.
People think: Git already versions prompts.
Actually: Git records the file, not which version served a given request or whether it was overridden at runtime.
Interview angle
Argue that a prompt is a deployable artifact with an id, a diff and a rollback, then say what you record per request. Strong answers pair versioning with evaluation and canary rollout. Weak answers say prompts live in git and cannot explain how they would reconstruct last week's behaviour.
Sources
- Khattab et al., DSPy — prompts as compiled artifacts with versioned pipelines.
- Pryzant et al., ProTeGi — iterative prompt edits that need to be tracked and scored.
- Semantic Versioning — a convention for communicating breaking changes.