Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 12 of 26

Output contracts

0 of 26 complete

Why it matters

Model output is probabilistic text, and the code that consumes it usually assumes a shape. When the assumption is implicit, violations surface as exceptions deep in a pipeline or, worse, as plausible wrong values. An explicit contract moves the assumption into a schema and a validation step, so a malformed response becomes a handled case with a defined outcome rather than an incident.

Mental model

An output contract is a delivery specification: dimensions, packaging, and what the receiver does with a damaged box. The comparison breaks because the sender can produce a box that meets every dimension and still contains the wrong item. Inspection has to check contents, not just the crate.

How it works

The contract and the repair callInteractive

Contract budget · illustrative tokens

2,000 input+ 700 reserve= 2,700 of 3,000

300 free

Step 1/5 · Contract in prose

Step 1Contract in prose

The four fields cost 480 of the 2,000 input tokens: nearly a quarter of the prompt describes the shape of an answer that has not been generated yet.

reserve 7002,000 tokens

Output reserve

700

480 of 2,000 input tokens are the contract; 1,220 are evidence and request.

output reservefixedBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. A contract is paid in input tokens before generation and in reserve if it is enforced. Writing the four fields as a schema costs 160 tokens instead of 480; a repair call that re-sends the prompt, the invalid object and the error costs 2,730 input tokens, 430 past the window.

What to notice. Counts are authored for a 3,000-token window with a 700-token reserve; no response was generated and no validator was run. The observed output length the reserve should be sized to is a p99 in real systems and is only named here. Provider-enforced schema modes are not drawn; the schema block stands for whatever the request carries to constrain the shape.

Text version

A 3,000-token window with a 700-token answer reserve, so 2,300 tokens are available. The default first step describes the contract in prose: instructions 300, field descriptions 480, document 1,100 and question 120, for 2,000 input tokens and 300 free. The second step replaces the prose with a 160-token schema: 1,680 tokens and 620 free. The third step spends the freed 320 tokens on a longer document, returning to 2,000 tokens and 300 free. The fourth step cuts the reserve from 700 to 300 tokens: the available prompt space rises to 2,700 and 700 tokens look free, while the longest valid object needs 640 and is now truncated. The fifth step is a repair call: instructions, schema, the longer document, the question, a 640-token invalid object and a 90-token validator error, for 2,730 tokens — 430 over the window. Selecting a section shows its count and role.

  1. Write the contract from the consumer's needs: required fields, types, allowed values and nullability.
  2. Express it as a schema the provider can enforce, or as a validation step after generation.
  3. Define the failure path for each violation class: retry, repair with a second call, degrade or reject.
  4. Add semantic checks the schema cannot express, such as totals reconciling or ids existing.
  5. Test with adversarial outputs: truncated JSON, extra fields, wrong enum, empty arrays.
  6. Version the contract with the prompt, since a field change is a breaking interface change.

Trade-offs

ChoiceGood forCostFails when
Schema-enforced generationParseable by constructionModel and provider constraints, less flexibilityThe task needs free-form prose
Prompt-only format requestWorks anywhereOccasional malformed outputDownstream code cannot tolerate repair
Retry on violationSimple recoveryExtra latency and costViolations are systematic, not random
Repair passKeeps most of the contentSecond call and possible driftThe violation is a missing fact
Trade-off mapInteractive

Good for

Parseable by construction

Cost

Model and provider constraints, less flexibility

Fails when

The task needs free-form prose

  1. The task needs free-form prose
  2. Downstream code cannot tolerate repair
  3. Violations are systematic, not random
  4. The violation is a missing fact

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Schema-enforced generation is good for Parseable by construction, costs Model and provider constraints, less flexibility, and fails when The task needs free-form prose. Prompt-only format request is good for Works anywhere, costs Occasional malformed output, and fails when Downstream code cannot tolerate repair. Retry on violation is good for Simple recovery, costs Extra latency and cost, and fails when Violations are systematic, not random. Repair pass is good for Keeps most of the content, costs Second call and possible drift, and fails when The violation is a missing fact

Failure modes

  • Truncated object. The response hits the token cap and the JSON is incomplete. Debug: check the finish reason and set the cap above the observed p99 output length.
  • Schema-valid emptiness. Required fields are present but null or empty. Debug: assert non-empty values and cross-check against the input.
  • Enum drift. The model returns a label outside the allowed set. Debug: log invalid values with the input that produced them and tighten the enum description.

In production

Track contract violation rate per route and model version, and alert when the repair path becomes the common path. Keep a fixture suite of malformed outputs to test the failure handling.

Common misconceptions

People think: A JSON schema makes the output correct.

Actually: It constrains structure; the values can still be wrong or unsupported.

People think: Validation belongs in the consumer.

Actually: Putting it at the boundary, with a defined failure path, keeps one consumer's assumption from becoming every consumer's bug.

Interview angle

Define the contract from the consumer side, then separate structural validation from semantic checks and give each a failure path. Strong answers mention truncation and versioning. Weak answers equate a schema with correctness and leave the retry or repair decision implicit.

Sources