Skip to content
AI Engineer Guide.

concept · concepts · beginner · concept 10 of 12

Prototype vs production

0 of 12 complete

Why it matters

A prototype optimises for the shortest path to a convincing demonstration: a notebook, a hardcoded prompt, a manual step. Production optimises for the failure that happens at three in the morning. The distance between them is not model quality but engineering: evaluation, observability, error handling, cost control and the ability to change one thing without breaking another.

Mental model

A prototype is a wind-tunnel model; production is the aircraft carrying passengers. The metaphor breaks because a wind-tunnel model scales by known physics, while an AI prototype's behaviour does not scale predictably with traffic and must be measured instead.

How it works

Mechanism, step by stepInteractive
  1. 1List the shortcuts in the prototype: hardcoded inputs, manual review, no retries, unbounded context.
  2. 2For each, decide whether it must be removed for launch or can remain an explicit manual step.
  3. 3Build the evaluation set and tracing before optimising quality, so changes can be compared.
  4. 4Set budgets for p95 latency, cost per request and failure rate, and test against them under load.
  5. 5Add permissions, secret handling and retention rules around the model calls.
  6. 6Rehearse the failure paths: provider outage, tool timeout, malformed output, rollback.

Step 1 of 6

List the shortcuts in the prototype: hardcoded inputs, manual review, no retries, unbounded context.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. List the shortcuts in the prototype: hardcoded inputs, manual review, no retries, unbound… 2. For each, decide whether it must be removed for launch or can remain an explicit manual s… 3. Build the evaluation set and tracing before optimising quality, so changes can be compare… 4. Set budgets for p95 latency, cost per request and failure rate, and test against them und… 5. Add permissions, secret handling and retention rules around the model calls. 6. Rehearse the failure paths: provider outage, tool timeout, malformed output, rollback.

  1. List the shortcuts in the prototype: hardcoded inputs, manual review, no retries, unbounded context.
  2. For each, decide whether it must be removed for launch or can remain an explicit manual step.
  3. Build the evaluation set and tracing before optimising quality, so changes can be compared.
  4. Set budgets for p95 latency, cost per request and failure rate, and test against them under load.
  5. Add permissions, secret handling and retention rules around the model calls.
  6. Rehearse the failure paths: provider outage, tool timeout, malformed output, rollback.

Trade-offs

ChoiceGood forCostFails when
Ship the prototype with guardrailsFastest learning from real usersOperational debt and incident riskThe action is consequential or data is sensitive
Rebuild for production firstClean architecture, fewer surprisesDelay and no user evidenceThe market window is short
Manual steps at launchAvoids building automation prematurelyDoes not scale and hides costVolume grows quickly
Feature flag and canaryLimits blast radiusExtra infrastructure and complexityThe feature cannot be partially enabled
Trade-off mapInteractive

Good for

Fastest learning from real users

Cost

Operational debt and incident risk

Fails when

The action is consequential or data is sensitive

  1. The action is consequential or data is sensitive
  2. The market window is short
  3. Volume grows quickly
  4. The feature cannot be partially enabled

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Ship the prototype with guardrails is good for Fastest learning from real users, costs Operational debt and incident risk, and fails when The action is consequential or data is sensitive. Rebuild for production first is good for Clean architecture, fewer surprises, costs Delay and no user evidence, and fails when The market window is short. Manual steps at launch is good for Avoids building automation prematurely, costs Does not scale and hides cost, and fails when Volume grows quickly. Feature flag and canary is good for Limits blast radius, costs Extra infrastructure and complexity, and fails when The feature cannot be partially enabled

Failure modes

  • Demo-only evaluation. Quality looks fine until real inputs arrive. Debug: compare the prototype's test inputs with a sample of production inputs.
  • Cost surprise. Spend per user exceeds the business model. Debug: measure cost per request at expected volume before launch, including retries.
  • No failure path. A dependency error surfaces as a broken page. Debug: force each dependency to fail in staging and confirm a defined user-visible outcome.

In production

Track latency, cost, error rate and sampled quality from the first day, with alerts tied to the budgets set before launch. Keep the prototype's manual steps documented so they can be automated deliberately.

Common misconceptions

People think: A working prototype is most of the work.

Actually: It demonstrates feasibility; the remaining work is evaluation, operations and failure paths.

People think: Production means a bigger model.

Actually: Most of the gap is engineering around the model, not model capability.

Interview angle

Strong answers enumerate the shortcuts a demo takes, say which are unacceptable in production and why, and describe the operational evidence that replaces them. Weak answers equate production readiness with a better model.

Sources