Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 18 of 28

Structured outputs

0 of 28 complete

Why it matters

An LLM inside a larger system is a function whose return type is text. The moment another service consumes that text, the application needs a contract. Prompt-only JSON is a request, not a contract: the model usually complies, occasionally adds a preamble, and fails at the worst possible moment — a batch job at 2 a.m., or a payment flow.

Structured outputs move the guarantee from the prompt to the decoder or the validator. That reduces retries, simplifies downstream code, and makes failures explicit. It does not remove the need to validate, because the schema describes shape, not meaning.

Mental model

A keyboard where only legal next characters are enabled. If the schema says the next field is an integer, the keys for letters simply do not respond. The output is guaranteed to be a syntactically valid document because invalid continuations were never available.

The analogy breaks at semantics. The keyboard can force the field to contain a number; it cannot force the number to be the invoice total rather than the tax. Structure is enforced at the token level, truth is not.

How it works

  1. 1Define the output contract as a JSON Schema, a Pydantic model, or a grammar.
  2. 2Send the schema to the provider, or compile it locally into a decoding constraint.
  3. 3At each step, mask tokens that would make the partial output invalid.
  4. 4Sample from the remaining distribution and append the token.
  5. 5Validate the completed output against the schema at the application boundary.
  6. 6Handle refusals, truncation and validation failures explicitly rather than assuming success.

There are three levels of enforcement, and they are often conflated:

Prompt-only formatting. The instruction says "respond with JSON matching this schema" and maybe includes an example. The model usually complies. Failures are frequent enough to need a parser, a retry, and a repair path, and they cluster on long outputs and unusual inputs.

Provider-enforced schemas. The API accepts a schema and applies constrained decoding server-side. OpenAI's structured outputs take a JSON Schema subset with strict mode; Gemini exposes a response schema; both document restrictions on the supported subset. In OpenAI's strict mode, every property must be listed as required and optionality is expressed with a nullable type, which is a common source of confusion when a schema is copied from a general JSON Schema.

Local constrained decoding. The schema or grammar is compiled into a finite-state machine and applied to the logits during generation. Libraries such as Outlines and grammar support in local runtimes implement this. It works with open-weights models and gives byte-level guarantees for regexes and context-free grammars, at the cost of compilation time and integration work.

Whichever level you use, the application still needs a validator. Providers can return a refusal instead of an object, outputs can be cut off by a token cap, and schema versions drift between services. Validation at the boundary is what turns a probabilistic component into a reliable one.

Why the guarantee is strong

Under unconstrained generation, the probability that a sequence of nn tokens forms a valid document is the product of per-step probabilities over the valid prefixes:

P(valid)=∏t=1n∑y∈Vvalid(y<t)pθ(y∣y<t)P(\text{valid}) = \prod_{t=1}^{n} \sum_{y \in \mathcal{V}_{\text{valid}}(y_{<t})} p_\theta(y \mid y_{<t})

Each factor is below one, so validity decays roughly exponentially with length. Constrained decoding replaces the full vocabulary with the allowed set At\mathcal{A}_t and renormalises:

pt′(y)=pθ(y∣y<t)∑y′∈Atpθ(y′∣y<t),y∈Atp'_t(y) = \frac{p_\theta(y \mid y_{<t})}{\sum_{y' \in \mathcal{A}_t} p_\theta(y' \mid y_{<t})}, \quad y \in \mathcal{A}_t

Now every sampled token keeps the prefix valid, so the completed output parses by construction. The distribution over content is still the model's, which is why semantics remain unguaranteed.

Validating at the boundary

from pydantic import BaseModel, ValidationError
 
class LineItem(BaseModel):
    sku: str
    quantity: int
 
class Order(BaseModel):
    order_id: str
    items: list[LineItem]
    total_cents: int
 
def parse_order(raw: str, finish_reason: str) -> Order | None:
    if finish_reason == "length":
        log_event("structured_output_truncated", raw=raw)
        return None
    try:
        return Order.model_validate_json(raw)
    except ValidationError as exc:
        log_event("structured_output_invalid", errors=exc.errors(), raw=raw)
        return None

The finish-reason check comes before parsing on purpose. A truncated object produces a confusing validation error that looks like a schema problem; classifying it as truncation points at the real fix — a larger output budget or a shorter prompt.

validinvalidSchemaPromptModelParseValidateTyped object
Step 1/4 · Define the schema
Figure. The schema is a contract; validation is what makes it true.
Text version

A schema shapes the prompt or constrains decoding; the model emits a candidate; the parser and validator check it; valid output becomes a typed object, invalid output retries.

Trade-offs

ApproachGood forCostFails when
Prompt-only JSONPrototypes, non-critical pathsFrequent parse errorsOutput feeds a strict downstream contract
Provider structured outputsHosted models, typed extractionSchema-subset restrictions, first-use compile latencyThe schema uses unsupported keywords
Tool calling as a schemaExtraction plus actionsTool semantics leak into promptsThe result is data, not an action
Local grammar decodingOpen-weights, regex or DSL constraintsIntegration and compile costThe constraint is too complex for the runtime
Post-hoc repair passLegacy prompts, occasional malformed outputExtra call, can hide the causeFailures are systematic rather than rare
Trade-off mapInteractive

Good for

Prototypes, non-critical paths

Cost

Frequent parse errors

Fails when

Output feeds a strict downstream contract

  1. Output feeds a strict downstream contract
  2. The schema uses unsupported keywords
  3. The result is data, not an action
  4. The constraint is too complex for the runtime
  5. Failures are systematic rather than rare

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Prompt-only JSON is good for Prototypes, non-critical paths, costs Frequent parse errors, and fails when Output feeds a strict downstream contract. Provider structured outputs is good for Hosted models, typed extraction, costs Schema-subset restrictions, first-use compile latency, and fails when The schema uses unsupported keywords. Tool calling as a schema is good for Extraction plus actions, costs Tool semantics leak into prompts, and fails when The result is data, not an action. Local grammar decoding is good for Open-weights, regex or DSL constraints, costs Integration and compile cost, and fails when The constraint is too complex for the runtime. Post-hoc repair pass is good for Legacy prompts, occasional malformed output, costs Extra call, can hide the cause, and fails when Failures are systematic rather than rare

Failure modes

Truncated output. The token cap fires mid-object and the JSON is invalid. Debug by reading the finish reason first, then the output budget. Increasing max tokens without checking prompt size just moves the failure.

Schema accepted but unsupported keywords ignored. A constraint such as a pattern or a conditional is silently not enforced. Debug by testing a value that should violate the constraint; if it passes, the keyword is outside the supported subset.

Semantically wrong but valid. Every field is present and typed, and the total is the subtotal. Debug with field-level evals and cross-field consistency checks, not with a JSON parser. Constrained decoding did exactly what it promised.

Refusal or empty object. The model declines, or returns an object with empty strings. Debug by checking the provider's refusal signal and the prompt; safety behaviour can override a schema.

In production

Version schemas alongside prompts and treat a schema change as an API change for every consumer. Validate at the boundary, count validation failures by endpoint, and alert on the truncation rate separately. Keep one retry with the validation error fed back, and cap it: repeated repair attempts are a symptom, not a fix. For hosted schemas, measure the first request after a schema change, because compilation can add latency that does not appear in steady state.

Common misconceptions

People think: JSON mode guarantees the schema.

Actually: JSON mode usually guarantees syntactic JSON only. Schema enforcement depends on strict-schema support or constrained decoding, and must be verified per provider.

People think: Constrained decoding makes the answer correct.

Actually: It makes the answer parseable. Field values, cross-field consistency and factual claims still come from the model and still need validation.

People think: A stricter schema is always safer.

Actually: Over-constrained formats can hurt reasoning quality and make refusals more likely. Constrain what the consumer needs, not everything the domain allows.

Interview angle

A common prompt is "the parser breaks once a week — fix it". The strong answer distinguishes syntax from semantics, chooses an enforcement level, checks the finish reason before the parser, and adds validation, metrics and a bounded retry. The weak answer adds "return only JSON" to the prompt. See the LLM fundamentals bank and the agent tooling questions.

Sources