Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 13 of 26

Prompt versioning

0 of 26 complete

Why it matters

Prompt edits are often made directly in a dashboard or a string constant, with no history and no link to the requests they served. When quality drops, the team cannot tell whether the prompt, the model, the retrieval index or the traffic changed. Versioning is the minimum instrumentation for attribution: an immutable identifier per prompt state, attached to every request and every evaluation run.

Mental model

Prompt versioning is a lab notebook: each experiment has a dated entry and a result. The metaphor breaks because the notebook is also the product, so an old entry may still be running in production for some users, and two entries can be live at the same time.

How it works

Two prompt versions live at onceInteractive
workversiongap · no spanevent
9 d

Event · step 1 of 7

9 d · v8 passed the suite

At 9 d

Prompt v8 is an immutable registry entry, and its evaluation finished on day 9 against the same suite that scored v7. No traffic has moved yet; the pass is a gate, not a release.

On screen at this event

  • v7
  • eval
  • v7 100%

Next at 10 d: 10 d · v8 takes 10%

Live
v7
Logged with id
100%
Evaluation
v8 pass
Model
m-41
Step 1/7 · 9 d · v8 passed the suite

Figure. A synthetic 24-day history: v8 passes evaluation, canaries at 10% on day 10, reaches 50% on day 12, is rolled back on day 13, and v9 ships on day 18 after the regression is reproduced. Shares, dates and scores are illustrative.

What to notice. The canary is only attributable because every request records the prompt id. Between days 11 and 12 the log has no prompt id, so exactly the window where quality moved cannot be reconstructed from traffic. Prompt and model revisions change independently, which is why both belong on the request record; versioning without the evaluation matrix only tells you what changed.

Text version

Five lanes over 24 days: registry, traffic, model, request log and evaluation. Registry versions are v7 0 to 13, v8 10 to 13 stacked on its own row beneath v7, and v9 18 to 24. Traffic is v7 at 100% to day 10; a second-row canary bar for v8 runs 10 to 13 while v7 keeps serving, then v7 alone to 18 and v9 after. The step details give the shares: 10% on day 10 and 50% on day 12. The model is m-41 to day 16 and m-42 after. The request log records a prompt id except for a red gap from day 11 to 12 labelled no id. Evaluation runs are v8 8.4 to 9, v8 again 13.2 to 14.5 and v9 16.4 to 17.8. Markers read v8 10% at 10, v8 50% at 12, rollback at 13 and v9 live at 18. Counters: live v7, v7 90% with v8 10%, a 50/50 split, v7, v7, v9, v9; logged with id 100% except 0% on the full gap day at 12; evaluation v8 pass, pending, v8 fail, v9 pass; model m-41 until day 16.

  1. Store prompts as files or registry entries with an immutable id, not as strings edited in place.
  2. Record the prompt id and model id on every request and every evaluation result.
  3. Diff versions by section so reviewers can see which instruction changed.
  4. Gate a version change on the evaluation suite and a canary share of traffic.
  5. Keep the previous version deployable, with a rollback that does not require a code release.
  6. Link each version to the incident or request that motivated it, so the history explains itself.

Trade-offs

ChoiceGood forCostFails when
Prompts in codeReviewed with the applicationDeploy required for every wording changeNon-engineers need to iterate
Prompt registryFast iteration and rollbackAnother service and access control surfaceThe team is small and deploys often
Version per requestFull attributionStorage and cardinalityTraffic is very high
Single live promptSimpleNo history and no rollbackMore than one person edits it
Trade-off mapInteractive

Good for

Reviewed with the application

Cost

Deploy required for every wording change

Fails when

Non-engineers need to iterate

  1. Non-engineers need to iterate
  2. The team is small and deploys often
  3. Traffic is very high
  4. More than one person edits it

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Prompts in code is good for Reviewed with the application, costs Deploy required for every wording change, and fails when Non-engineers need to iterate. Prompt registry is good for Fast iteration and rollback, costs Another service and access control surface, and fails when The team is small and deploys often. Version per request is good for Full attribution, costs Storage and cardinality, and fails when Traffic is very high. Single live prompt is good for Simple, costs No history and no rollback, and fails when More than one person edits it

Failure modes

  • Unrecorded edit. A dashboard change alters behaviour and no request log shows it. Debug: compare the prompt hash in logs with the registry version for the same period.
  • Version without content. Two different prompts share a version label after a partial deploy. Debug: hash the rendered prompt and store the hash with the version.
  • Rollback that is not tested. The old version no longer works with the new model. Debug: keep the previous version in the evaluation matrix and run it on every model upgrade.

In production

Attach prompt and model versions to traces, and alert when the live version differs from the reviewed one. Canary a new version on a small traffic share before full rollout.

Common misconceptions

People think: Prompt changes are too small to version.

Actually: Small wording changes move accuracy; without a version they are indistinguishable from noise.

People think: Git already versions prompts.

Actually: Git records the file, not which version served a given request or whether it was overridden at runtime.

Interview angle

Argue that a prompt is a deployable artifact with an id, a diff and a rollback, then say what you record per request. Strong answers pair versioning with evaluation and canary rollout. Weak answers say prompts live in git and cannot explain how they would reconstruct last week's behaviour.

Sources