Skip to content
AI Engineer Guide.

concept · concepts · beginner · concept 1 of 12

What is AI engineering

0 of 12 complete

Why it matters

A foundation model is a dependency that returns a different answer to the same input, costs money per token, and changes behaviour when the provider ships a new version. Every conventional engineering assumption — deterministic output, cheap retries, stable interfaces, testable units — needs an explicit replacement before you can put the thing in front of a customer.

That is the engineering problem AI engineers exist to solve. The demo is the easy part: one prompt, one happy path, one reviewer. The product needs a quality bar per user segment, an answer when the model is wrong, a latency budget, a cost ceiling, and a way to notice a regression before a customer does. None of those come from the model, which is why the discipline is mostly ordinary software engineering applied to an unusual dependency: context, evaluation and failure semantics do the real work.

Mental model

Think of a power grid. The generator is someone else's; you build the transmission lines, the breakers, the meters and the contract that says what voltage arrives at the socket. Your job is not to make electricity — it is to make electricity usable and safe at the point of consumption.

The analogy breaks in one important way: a power plant's output is stable and measurable in volts, while a model's output is semantic and varies with phrasing. You cannot put a multimeter on "did this answer resolve the customer's question". That is why AI engineering leans so heavily on evaluation sets and traces: they are the multimeter you had to invent.

How it works

Mechanism, step by stepInteractive
  1. 1Frame the decision, not the feature. Write down what the user does with the output and what a wrong output costs. A summariser that is wrong on a marketing page and one that is wrong on a contract clause need different bars.
  2. 2Decide build versus rent. Choose the model, hosting mode and whether adaptation is needed at all. Most products should start with a hosted model and no fine-tuning; the burden of proof sits with the more complex option.
  3. 3Design the context. Specify what the model receives: instructions, retrieved evidence, conversation state, tool results. Context assembly is where most quality is won or lost, and it is the part you can version and test.
  4. 4Define the output contract. Decide whether the response is prose, a schema, or a tool call. Constrain it structurally rather than asking politely, and decide what happens on a malformed result.
  5. 5Build the evaluation harness before tuning. A small labelled set, a scoring method, and a regression run on every prompt or model change. Without it you cannot tell improvement from noise.
  6. 6Set the latency and cost budget. Fix a target for time-to-first-token and total completion, and a ceiling for cost per request. These constraints eliminate whole designs early.

Step 1 of 6

Frame the decision, not the feature. Write down what the user does with the output and what a wrong output costs. A summariser that is wrong on a marketing page and one that is wrong on a contract clause need different bars.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Frame the decision, not the feature. Write down what the user does with the output and wh… 2. Decide build versus rent. Choose the model, hosting mode and whether adaptation is needed… 3. Design the context. Specify what the model receives: instructions, retrieved evidence, co… 4. Define the output contract. Decide whether the response is prose, a schema, or a tool cal… 5. Build the evaluation harness before tuning. A small labelled set, a scoring method, and a… 6. Set the latency and cost budget. Fix a target for time-to-first-token and total completio…

The work repeats in the same order on every product, and each step produces an artefact you can review.

  1. Frame the decision, not the feature. Write down what the user does with the output and what a wrong output costs. A summariser that is wrong on a marketing page and one that is wrong on a contract clause need different bars.
  2. Decide build versus rent. Choose the model, hosting mode and whether adaptation is needed at all. Most products should start with a hosted model and no fine-tuning; the burden of proof sits with the more complex option.
  3. Design the context. Specify what the model receives: instructions, retrieved evidence, conversation state, tool results. Context assembly is where most quality is won or lost, and it is the part you can version and test.
  4. Define the output contract. Decide whether the response is prose, a schema, or a tool call. Constrain it structurally rather than asking politely, and decide what happens on a malformed result.
  5. Build the evaluation harness before tuning. A small labelled set, a scoring method, and a regression run on every prompt or model change. Without it you cannot tell improvement from noise.
  6. Set the latency and cost budget. Fix a target for time-to-first-token and total completion, and a ceiling for cost per request. These constraints eliminate whole designs early.
  7. Instrument and operate. Log inputs, retrieved context, outputs, token counts and outcome signals. Watch the distribution, not the average, and route failures to a human path.

Steps 3 to 5 are iterative and consume most of the calendar time. Step 1 is the one teams skip, and it is why "the model is not good enough" is usually a misdiagnosis.

Core artefacts
Prompt and context assembly, eval set, traces, fallback path
Owned by you
Context, schema, evaluation, retries, cost, UX of failure
Rented
Weights, tokeniser, serving infrastructure, safety filters
Typical first blocker
No labelled examples of a correct answer
Usual hidden cost
Iteration speed, not inference price

A concrete example

A support assistant over a policy corpus looks like one feature. Decomposed, it is five systems: ingestion that keeps documents current, retrieval that finds the right clause, assembly that fits it into the context budget, generation that cites what it used, and an escalation path. A team that ships it as "call the model with the user's question" spends the next quarter discovering those five systems one incident at a time.

The tell is the bug report. "It made something up" is almost never a generation problem. Tracing usually shows the right clause was never retrieved, was truncated in assembly, or was contradicted by an older cached copy. The fix lives in the pipeline, not the prompt.

Trade-offs

LeverBuys youCosts youFails when
Bigger or newer modelQuality on hard, open-ended tasksPrice, latency, less predictable behaviourThe task needs private or current facts the model never saw
Retrieval and context workGrounding, freshness, citationsPipeline complexity, retrieval latencyThe corpus is thin or the question needs judgement, not lookup
Fine-tuningConsistent format, tone, narrow task accuracyDataset construction, retraining, versioningThe requirement is knowledge rather than behaviour
Aggressive guardrailsFewer harmful or off-contract outputsCoverage, refusal rate, user trustThe guardrail blocks legitimate edge cases and nobody measures it
Human review in the loopA correct answer on the hard tailThroughput, cost per resolved requestVolume grows and the queue becomes the product
Trade-off mapInteractive

Good for

Quality on hard, open-ended tasks

Cost

Price, latency, less predictable behaviour

Fails when

The task needs private or current facts the model never saw

  1. The task needs private or current facts the model never saw
  2. The corpus is thin or the question needs judgement, not lookup
  3. The requirement is knowledge rather than behaviour
  4. The guardrail blocks legitimate edge cases and nobody measures it
  5. Volume grows and the queue becomes the product

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Bigger or newer model is good for Quality on hard, open-ended tasks, costs Price, latency, less predictable behaviour, and fails when The task needs private or current facts the model never saw. Retrieval and context work is good for Grounding, freshness, citations, costs Pipeline complexity, retrieval latency, and fails when The corpus is thin or the question needs judgement, not lookup. Fine-tuning is good for Consistent format, tone, narrow task accuracy, costs Dataset construction, retraining, versioning, and fails when The requirement is knowledge rather than behaviour. Aggressive guardrails is good for Fewer harmful or off-contract outputs, costs Coverage, refusal rate, user trust, and fails when The guardrail blocks legitimate edge cases and nobody measures it. Human review in the loop is good for A correct answer on the hard tail, costs Throughput, cost per resolved request, and fails when Volume grows and the queue becomes the product

Failure modes

Shipping on aggregate accuracy. A single score hides the slice that matters. Debug: split the eval by intent and consequence, then look for any slice below the bar even when the mean is fine.

No answer for "model unavailable". The provider returns errors or elevated latency and the feature fails closed with a stack trace. Debug: inject failures in staging — timeout, 429, malformed JSON — and verify the user sees a defined state rather than a spinner.

Measuring quality by reading outputs manually. This works at ten requests a day and collapses at ten thousand. Debug: sample traces into a labelled set weekly, and track the abstain and escalation rates as first-class metrics.

In production

The operating numbers are time-to-first-token, p95 end-to-end latency, tokens per request, cost per resolved request, schema-validity rate, and escalation rate. None are model benchmarks; all are properties of your system, which is why they belong on your dashboard. A model upgrade is a deployment: pin the version, run the regression suite, canary a slice of traffic, and keep the previous version warm long enough to roll back.

Common misconceptions

People think: AI engineering is prompt engineering with a new job title.

Actually: Prompting is one deliverable. The role also owns retrieval, output contracts, evaluation, latency and cost budgets, observability and failure handling — and those consume more time than wording.

People think: A better model will fix the quality problem.

Actually: A better model raises the ceiling on reasoning. It does not supply facts it was never given, enforce your schema, or tell you which user segment is failing. Those are system properties.

Interview angle

Expect the opening question in almost every AI engineering loop: "walk me through an AI feature you shipped end to end." The strong answer separates the model from the system — what you controlled, what you measured, how you handled what you could not. Name a concrete failure you found in tracing, the fix, and the metric that moved, then be explicit about the trade-off you accepted.

Practice with walk me through an AI feature you shipped end to end, and browse behavioural questions.

Sources