Skip to content
AI Engineer Guide.

concept · concepts · beginner · concept 5 of 12

AI application lifecycle

0 of 12 complete

Why it matters

Traditional software's lifecycle assumes a specification that holds still while the implementation changes. An AI application has a second moving part: behaviour that comes from data and weights, and changes when either changes. The lifecycle exists to keep those versions aligned, so a regression can be attributed and a rollback can name exactly what it is rolling back.

Mental model

The lifecycle is a control loop with a slow sensor: ship, observe, relabel, then retrain or re-prompt. The metaphor breaks because a control loop converges on a setpoint you defined, while here the setpoint is a human judgement that drifts as users learn what the system can do.

How it works

Mechanism, step by stepInteractive
  1. 1Frame the problem as a decision with an owner, a cost of error and an existing baseline.
  2. 2Assemble a labelled evaluation set from real inputs before building, and hold back a slice.
  3. 3Build the thinnest pipeline that can be measured end to end, including retrieval and prompts.
  4. 4Ship behind a flag with tracing, and compare online behaviour with the offline evaluation.
  5. 5Feed corrected and failed cases back into the evaluation set on a schedule.
  6. 6Version prompt, model, data and index together so a rollback is one operation.

Step 1 of 6

Frame the problem as a decision with an owner, a cost of error and an existing baseline.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Frame the problem as a decision with an owner, a cost of error and an existing baseline. 2. Assemble a labelled evaluation set from real inputs before building, and hold back a slic… 3. Build the thinnest pipeline that can be measured end to end, including retrieval and prom… 4. Ship behind a flag with tracing, and compare online behaviour with the offline evaluation. 5. Feed corrected and failed cases back into the evaluation set on a schedule. 6. Version prompt, model, data and index together so a rollback is one operation.

  1. Frame the problem as a decision with an owner, a cost of error and an existing baseline.
  2. Assemble a labelled evaluation set from real inputs before building, and hold back a slice.
  3. Build the thinnest pipeline that can be measured end to end, including retrieval and prompts.
  4. Ship behind a flag with tracing, and compare online behaviour with the offline evaluation.
  5. Feed corrected and failed cases back into the evaluation set on a schedule.
  6. Version prompt, model, data and index together so a rollback is one operation.

Trade-offs

ChoiceGood forCostFails when
Prototype firstFast learning about feasibilityEvaluation debt and reworkThe problem turns out to need reliability
Evaluation firstClear target and safer launchesSlower to first signalThe team cannot label data yet
Continuous retrainingTracks drift closelyOperational burden and regression riskLabels arrive slowly or noisily
Frozen model, prompt iterationSimple operationsPlateaus on hard casesThe gap is in the data, not the prompt
Trade-off mapInteractive

Good for

Fast learning about feasibility

Cost

Evaluation debt and rework

Fails when

The problem turns out to need reliability

  1. The problem turns out to need reliability
  2. The team cannot label data yet
  3. Labels arrive slowly or noisily
  4. The gap is in the data, not the prompt

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Prototype first is good for Fast learning about feasibility, costs Evaluation debt and rework, and fails when The problem turns out to need reliability. Evaluation first is good for Clear target and safer launches, costs Slower to first signal, and fails when The team cannot label data yet. Continuous retraining is good for Tracks drift closely, costs Operational burden and regression risk, and fails when Labels arrive slowly or noisily. Frozen model, prompt iteration is good for Simple operations, costs Plateaus on hard cases, and fails when The gap is in the data, not the prompt

Failure modes

  • Unversioned change. Quality drops and nobody can name what changed. Debug: require a manifest of prompt, model and index versions per deploy and diff it.
  • Evaluation set rot. Offline scores rise while user complaints grow. Debug: compare the age distribution of evaluation examples with production inputs.
  • Orphaned feature. A shipped feature has no owner and no monitoring. Debug: list deployed model-backed endpoints and check each has an alert and a named owner.

In production

Monitor input distribution, output quality proxies and cost per request, and review a sample of real interactions weekly. Keep a rollback path that restores the previous prompt, model and index together.

Common misconceptions

People think: The lifecycle ends at launch.

Actually: Launch starts the phase where real inputs expose what the evaluation set missed.

People think: A better model removes lifecycle work.

Actually: It changes the baseline; monitoring, labelling and versioning remain, because the inputs keep moving.

Interview angle

Strong answers describe the loop and name what is versioned together, then explain how a production failure becomes an evaluation case. Weak answers present a linear pipeline from data to deployment and stop at launch.

Sources