Why it matters
A prototype optimises for the shortest path to a convincing demonstration: a notebook, a hardcoded prompt, a manual step. Production optimises for the failure that happens at three in the morning. The distance between them is not model quality but engineering: evaluation, observability, error handling, cost control and the ability to change one thing without breaking another.
Mental model
A prototype is a wind-tunnel model; production is the aircraft carrying passengers. The metaphor breaks because a wind-tunnel model scales by known physics, while an AI prototype's behaviour does not scale predictably with traffic and must be measured instead.
How it works
- 1List the shortcuts in the prototype: hardcoded inputs, manual review, no retries, unbounded context.
- 2For each, decide whether it must be removed for launch or can remain an explicit manual step.
- 3Build the evaluation set and tracing before optimising quality, so changes can be compared.
- 4Set budgets for p95 latency, cost per request and failure rate, and test against them under load.
- 5Add permissions, secret handling and retention rules around the model calls.
- 6Rehearse the failure paths: provider outage, tool timeout, malformed output, rollback.
Step 1 of 6
List the shortcuts in the prototype: hardcoded inputs, manual review, no retries, unbounded context.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. List the shortcuts in the prototype: hardcoded inputs, manual review, no retries, unbound… 2. For each, decide whether it must be removed for launch or can remain an explicit manual s… 3. Build the evaluation set and tracing before optimising quality, so changes can be compare… 4. Set budgets for p95 latency, cost per request and failure rate, and test against them und… 5. Add permissions, secret handling and retention rules around the model calls. 6. Rehearse the failure paths: provider outage, tool timeout, malformed output, rollback.
- List the shortcuts in the prototype: hardcoded inputs, manual review, no retries, unbounded context.
- For each, decide whether it must be removed for launch or can remain an explicit manual step.
- Build the evaluation set and tracing before optimising quality, so changes can be compared.
- Set budgets for p95 latency, cost per request and failure rate, and test against them under load.
- Add permissions, secret handling and retention rules around the model calls.
- Rehearse the failure paths: provider outage, tool timeout, malformed output, rollback.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Ship the prototype with guardrails | Fastest learning from real users | Operational debt and incident risk | The action is consequential or data is sensitive |
| Rebuild for production first | Clean architecture, fewer surprises | Delay and no user evidence | The market window is short |
| Manual steps at launch | Avoids building automation prematurely | Does not scale and hides cost | Volume grows quickly |
| Feature flag and canary | Limits blast radius | Extra infrastructure and complexity | The feature cannot be partially enabled |
Good for
Fastest learning from real users
Cost
Operational debt and incident risk
Fails when
The action is consequential or data is sensitive
- The action is consequential or data is sensitive
- The market window is short
- Volume grows quickly
- The feature cannot be partially enabled
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Ship the prototype with guardrails is good for Fastest learning from real users, costs Operational debt and incident risk, and fails when The action is consequential or data is sensitive. Rebuild for production first is good for Clean architecture, fewer surprises, costs Delay and no user evidence, and fails when The market window is short. Manual steps at launch is good for Avoids building automation prematurely, costs Does not scale and hides cost, and fails when Volume grows quickly. Feature flag and canary is good for Limits blast radius, costs Extra infrastructure and complexity, and fails when The feature cannot be partially enabled
Failure modes
- Demo-only evaluation. Quality looks fine until real inputs arrive. Debug: compare the prototype's test inputs with a sample of production inputs.
- Cost surprise. Spend per user exceeds the business model. Debug: measure cost per request at expected volume before launch, including retries.
- No failure path. A dependency error surfaces as a broken page. Debug: force each dependency to fail in staging and confirm a defined user-visible outcome.
In production
Track latency, cost, error rate and sampled quality from the first day, with alerts tied to the budgets set before launch. Keep the prototype's manual steps documented so they can be automated deliberately.
Common misconceptions
People think: A working prototype is most of the work.
Actually: It demonstrates feasibility; the remaining work is evaluation, operations and failure paths.
People think: Production means a bigger model.
Actually: Most of the gap is engineering around the model, not model capability.
Interview angle
Strong answers enumerate the shortcuts a demo takes, say which are unacceptable in production and why, and describe the operational evidence that replaces them. Weak answers equate production readiness with a better model.
Sources
- Sculley et al., Hidden Technical Debt in Machine Learning Systems — the engineering surface that appears once a model is in production.
- Breck et al., The ML Test Score — readiness checks across data, model, infrastructure and monitoring.
- The Twelve-Factor App — configuration, dependencies and operational discipline for a deployable service.