Why it matters
A prompt is a flat sequence of tokens, so the only thing separating a rule from a document is the text around it. When a prompt mixes instructions, retrieved passages and examples without markers, the model has to infer boundaries, and it does so inconsistently. Explicit structure reduces that inference: consistent headings, delimiters around untrusted content and a named output shape give the model fewer plausible readings.
Mental model
Formatting is punctuation in a legal document: it does not change the words, but it decides which clause a comma belongs to. The comparison breaks because a model can ignore punctuation that a court would enforce, so structure is a strong hint rather than a rule.
How it works
Rendered prompt · illustrative tokens
1,170 input+ 300 reserve= 1,470 of 1,600
130 free
Step 1No markers
1,170 tokens with 130 free. Nothing separates the rules from the document or the question: the model infers the boundaries, and it does so inconsistently when the document reads like instructions.
Output reserve
300No marker tokens at all. That is the cheapest render and the one with the weakest boundary.
Pick a section to read its token count, its role and its state.
Figure. The same instructions, document and question, formatted four ways. Markers are tokens: 30 for headings, 90 for XML-style tags, 240 for a JSON envelope — which puts the request 110 tokens over a window the unformatted prompt filled with room to spare.
What to notice. Counts are authored for a 1,600-token window with a 300-token answer reserve; no prompt was rendered and no tokenizer was run. The marker blocks are whole-section averages, not the byte cost of one tag. Delimiter collision inside the document, escaping rules and the model's own chat markers are omitted.
Text version
A 1,600-token window with a 300-token answer reserve, so 1,300 tokens are available. The default first step has no markers: instructions 180, document 900 and question 90, for 1,170 input tokens and 130 free. The second step adds 30 tokens of Markdown headings for 1,200 tokens and 100 free. The third step replaces them with 90 tokens of XML-style tags, marks the document as isolated data, and reads 1,260 tokens with 40 free. The fourth step wraps the same content in a 240-token JSON envelope: 1,410 input tokens, 110 over the window, with the envelope cells crossing the cap rule into the reserve. Selecting a section shows its count and role.
- Choose a delimiter style the model has seen in training, such as XML-like tags or Markdown headings, and use it consistently.
- Wrap every variable block, especially retrieved text and user input, in a named delimiter.
- Put the output format at the end, with a literal example of the expected shape.
- Keep the same structure across prompt versions so diffs are readable and caching still matches.
- Test with content that contains your delimiters, quotes and newlines, and check the rendered prompt.
- Measure format validity separately from answer correctness, since they fail for different reasons.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| XML-style tags | Clear boundaries, familiar to many models | More tokens than plain text | The content itself contains tags |
| Markdown headings | Readable, cheap | Weaker boundary for untrusted text | User input includes headings |
| JSON envelope | Machine-checkable structure | Escaping burden, harder for prose | The model must write long free text |
| No delimiters | Fewest tokens | Boundaries inferred inconsistently | Any untrusted content is present |
Good for
Clear boundaries, familiar to many models
Cost
More tokens than plain text
Fails when
The content itself contains tags
- The content itself contains tags
- User input includes headings
- The model must write long free text
- Any untrusted content is present
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: XML-style tags is good for Clear boundaries, familiar to many models, costs More tokens than plain text, and fails when The content itself contains tags. Markdown headings is good for Readable, cheap, costs Weaker boundary for untrusted text, and fails when User input includes headings. JSON envelope is good for Machine-checkable structure, costs Escaping burden, harder for prose, and fails when The model must write long free text. No delimiters is good for Fewest tokens, costs Boundaries inferred inconsistently, and fails when Any untrusted content is present
Failure modes
- Delimiter collision. A document contains the closing tag and the model treats the rest as instructions. Debug: search rendered input for the delimiter and escape or strip it.
- Format without content. Output is valid JSON with empty or wrong fields. Debug: score field-level correctness, not just parse success.
- Structure drift. Sections move between prompt versions and cached prefixes stop matching. Debug: diff the rendered prompt against the previous version and check the cache hash.
In production
Validate output format on every response and track the invalid rate per model version. Snapshot rendered prompts in tests so structural changes are reviewed.
Common misconceptions
People think: Formatting is cosmetic.
Actually: Delimiters and headings are the model's only signal for where instructions end and data begins.
People think: Valid JSON means the prompt worked.
Actually: Format validity and semantic correctness are independent checks.
Interview angle
Explain format as the boundary signal between instruction and data, then choose delimiters for the content you actually receive. Strong answers test collision cases and separate parse validity from correctness. Weak answers decorate the prompt with headings and assume the model will respect them.
Sources
- Anthropic, Use XML tags to structure prompts — tag-delimited sections and why they help.
- Schulhoff et al., The Prompt Report — formatting choices across prompting techniques.
- OpenAI, Prompt engineering guide — separating instructions from input with delimiters.