Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 11 of 26

Structured prompting

0 of 26 complete

Why it matters

A prompt is a flat sequence of tokens, so the only thing separating a rule from a document is the text around it. When a prompt mixes instructions, retrieved passages and examples without markers, the model has to infer boundaries, and it does so inconsistently. Explicit structure reduces that inference: consistent headings, delimiters around untrusted content and a named output shape give the model fewer plausible readings.

Mental model

Formatting is punctuation in a legal document: it does not change the words, but it decides which clause a comma belongs to. The comparison breaks because a model can ignore punctuation that a court would enforce, so structure is a strong hint rather than a rule.

How it works

What a delimiter costsInteractive

Rendered prompt · illustrative tokens

1,170 input+ 300 reserve= 1,470 of 1,600

130 free

Step 1/4 · No markers

Step 1No markers

1,170 tokens with 130 free. Nothing separates the rules from the document or the question: the model infers the boundaries, and it does so inconsistently when the document reads like instructions.

reserve 3001,170 tokens

Output reserve

300

No marker tokens at all. That is the cheapest render and the one with the weakest boundary.

output reservefixedBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. The same instructions, document and question, formatted four ways. Markers are tokens: 30 for headings, 90 for XML-style tags, 240 for a JSON envelope — which puts the request 110 tokens over a window the unformatted prompt filled with room to spare.

What to notice. Counts are authored for a 1,600-token window with a 300-token answer reserve; no prompt was rendered and no tokenizer was run. The marker blocks are whole-section averages, not the byte cost of one tag. Delimiter collision inside the document, escaping rules and the model's own chat markers are omitted.

Text version

A 1,600-token window with a 300-token answer reserve, so 1,300 tokens are available. The default first step has no markers: instructions 180, document 900 and question 90, for 1,170 input tokens and 130 free. The second step adds 30 tokens of Markdown headings for 1,200 tokens and 100 free. The third step replaces them with 90 tokens of XML-style tags, marks the document as isolated data, and reads 1,260 tokens with 40 free. The fourth step wraps the same content in a 240-token JSON envelope: 1,410 input tokens, 110 over the window, with the envelope cells crossing the cap rule into the reserve. Selecting a section shows its count and role.

  1. Choose a delimiter style the model has seen in training, such as XML-like tags or Markdown headings, and use it consistently.
  2. Wrap every variable block, especially retrieved text and user input, in a named delimiter.
  3. Put the output format at the end, with a literal example of the expected shape.
  4. Keep the same structure across prompt versions so diffs are readable and caching still matches.
  5. Test with content that contains your delimiters, quotes and newlines, and check the rendered prompt.
  6. Measure format validity separately from answer correctness, since they fail for different reasons.

Trade-offs

ChoiceGood forCostFails when
XML-style tagsClear boundaries, familiar to many modelsMore tokens than plain textThe content itself contains tags
Markdown headingsReadable, cheapWeaker boundary for untrusted textUser input includes headings
JSON envelopeMachine-checkable structureEscaping burden, harder for proseThe model must write long free text
No delimitersFewest tokensBoundaries inferred inconsistentlyAny untrusted content is present
Trade-off mapInteractive

Good for

Clear boundaries, familiar to many models

Cost

More tokens than plain text

Fails when

The content itself contains tags

  1. The content itself contains tags
  2. User input includes headings
  3. The model must write long free text
  4. Any untrusted content is present

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: XML-style tags is good for Clear boundaries, familiar to many models, costs More tokens than plain text, and fails when The content itself contains tags. Markdown headings is good for Readable, cheap, costs Weaker boundary for untrusted text, and fails when User input includes headings. JSON envelope is good for Machine-checkable structure, costs Escaping burden, harder for prose, and fails when The model must write long free text. No delimiters is good for Fewest tokens, costs Boundaries inferred inconsistently, and fails when Any untrusted content is present

Failure modes

  • Delimiter collision. A document contains the closing tag and the model treats the rest as instructions. Debug: search rendered input for the delimiter and escape or strip it.
  • Format without content. Output is valid JSON with empty or wrong fields. Debug: score field-level correctness, not just parse success.
  • Structure drift. Sections move between prompt versions and cached prefixes stop matching. Debug: diff the rendered prompt against the previous version and check the cache hash.

In production

Validate output format on every response and track the invalid rate per model version. Snapshot rendered prompts in tests so structural changes are reviewed.

Common misconceptions

People think: Formatting is cosmetic.

Actually: Delimiters and headings are the model's only signal for where instructions end and data begins.

People think: Valid JSON means the prompt worked.

Actually: Format validity and semantic correctness are independent checks.

Interview angle

Explain format as the boundary signal between instruction and data, then choose delimiters for the content you actually receive. Strong answers test collision cases and separate parse validity from correctness. Weak answers decorate the prompt with headings and assume the model will respect them.

Sources