Why it matters
Traditional software's lifecycle assumes a specification that holds still while the implementation changes. An AI application has a second moving part: behaviour that comes from data and weights, and changes when either changes. The lifecycle exists to keep those versions aligned, so a regression can be attributed and a rollback can name exactly what it is rolling back.
Mental model
The lifecycle is a control loop with a slow sensor: ship, observe, relabel, then retrain or re-prompt. The metaphor breaks because a control loop converges on a setpoint you defined, while here the setpoint is a human judgement that drifts as users learn what the system can do.
How it works
- 1Frame the problem as a decision with an owner, a cost of error and an existing baseline.
- 2Assemble a labelled evaluation set from real inputs before building, and hold back a slice.
- 3Build the thinnest pipeline that can be measured end to end, including retrieval and prompts.
- 4Ship behind a flag with tracing, and compare online behaviour with the offline evaluation.
- 5Feed corrected and failed cases back into the evaluation set on a schedule.
- 6Version prompt, model, data and index together so a rollback is one operation.
Step 1 of 6
Frame the problem as a decision with an owner, a cost of error and an existing baseline.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Frame the problem as a decision with an owner, a cost of error and an existing baseline. 2. Assemble a labelled evaluation set from real inputs before building, and hold back a slic… 3. Build the thinnest pipeline that can be measured end to end, including retrieval and prom… 4. Ship behind a flag with tracing, and compare online behaviour with the offline evaluation. 5. Feed corrected and failed cases back into the evaluation set on a schedule. 6. Version prompt, model, data and index together so a rollback is one operation.
- Frame the problem as a decision with an owner, a cost of error and an existing baseline.
- Assemble a labelled evaluation set from real inputs before building, and hold back a slice.
- Build the thinnest pipeline that can be measured end to end, including retrieval and prompts.
- Ship behind a flag with tracing, and compare online behaviour with the offline evaluation.
- Feed corrected and failed cases back into the evaluation set on a schedule.
- Version prompt, model, data and index together so a rollback is one operation.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Prototype first | Fast learning about feasibility | Evaluation debt and rework | The problem turns out to need reliability |
| Evaluation first | Clear target and safer launches | Slower to first signal | The team cannot label data yet |
| Continuous retraining | Tracks drift closely | Operational burden and regression risk | Labels arrive slowly or noisily |
| Frozen model, prompt iteration | Simple operations | Plateaus on hard cases | The gap is in the data, not the prompt |
Good for
Fast learning about feasibility
Cost
Evaluation debt and rework
Fails when
The problem turns out to need reliability
- The problem turns out to need reliability
- The team cannot label data yet
- Labels arrive slowly or noisily
- The gap is in the data, not the prompt
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Prototype first is good for Fast learning about feasibility, costs Evaluation debt and rework, and fails when The problem turns out to need reliability. Evaluation first is good for Clear target and safer launches, costs Slower to first signal, and fails when The team cannot label data yet. Continuous retraining is good for Tracks drift closely, costs Operational burden and regression risk, and fails when Labels arrive slowly or noisily. Frozen model, prompt iteration is good for Simple operations, costs Plateaus on hard cases, and fails when The gap is in the data, not the prompt
Failure modes
- Unversioned change. Quality drops and nobody can name what changed. Debug: require a manifest of prompt, model and index versions per deploy and diff it.
- Evaluation set rot. Offline scores rise while user complaints grow. Debug: compare the age distribution of evaluation examples with production inputs.
- Orphaned feature. A shipped feature has no owner and no monitoring. Debug: list deployed model-backed endpoints and check each has an alert and a named owner.
In production
Monitor input distribution, output quality proxies and cost per request, and review a sample of real interactions weekly. Keep a rollback path that restores the previous prompt, model and index together.
Common misconceptions
People think: The lifecycle ends at launch.
Actually: Launch starts the phase where real inputs expose what the evaluation set missed.
People think: A better model removes lifecycle work.
Actually: It changes the baseline; monitoring, labelling and versioning remain, because the inputs keep moving.
Interview angle
Strong answers describe the loop and name what is versioned together, then explain how a production failure becomes an evaluation case. Weak answers present a linear pipeline from data to deployment and stop at launch.
Sources
- Sculley et al., Hidden Technical Debt in Machine Learning Systems — why data dependencies and configuration create ongoing lifecycle cost.
- Breck et al., The ML Test Score — a rubric covering data, model, infrastructure and monitoring readiness.
- Google, Site Reliability Engineering — operating a service against explicit objectives and change management.