Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 10 of 28

Chat templates

0 of 28 complete

Why it matters

Instruction-tuned models learn to follow roles through a particular serialisation: markers before each turn, a separator after it, sometimes a generation prompt at the end. The weights are conditioned on those exact tokens. An application that invents its own format, or that upgrades a model without its template, sends a prompt the model has never seen and gets degraded instruction following that looks like a quality problem.

Mental model

A chat template is the envelope format a postal system expects: address block here, stamp there. Put the address in the wrong corner and the letter still travels, just not where you meant. The image breaks because a misformatted prompt is not returned to sender; it is answered, badly, with no error.

How it works

Messages to one token streamInteractive

Rendered prompt · illustrative tokens

405 input+ 120 reserve= 525 of 600

75 free

Step 1/5 · Messages as objects

Step 1Messages as objects

The application holds four messages with roles. They are laid side by side here only to compare token cost — no separators exist yet, and nothing here is a token stream.

Message objects · widths compare token cost onlyreserve 120405 tokens

Output reserve

120
output reserveBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. A message list is not what the model receives. The template adds role markers, turn separators and a generation prompt around the authored text, and every one of those tokens is charged to the same budget.

What to notice. The marker shapes are illustrative and do not reproduce any named model's template. What carries over is that the exact markers are model-specific, part of the model interface, and counted in the prompt.

Text version

A 600-token window with a 120-token output reserve. The default first step lays out four messages as separate objects — system 180, user 60, assistant 120 and user 45 — with a row label saying the widths compare token cost only; they are not yet a stream. The second step inserts a BOS token, a system header, three end-of-turn separators, and highlights them as markers. The third step adds the generation prompt and renders one stream of 428 input tokens. The fourth step highlights the markers and counts 23 overhead tokens, about 5% of the prompt. The fifth step highlights the generation prompt at the tail and notes what a missing one causes. Selecting any segment shows its count and role.

  1. Load the template that ships with the model rather than writing role markers by hand.
  2. Render a two-turn conversation and read the raw string, checking the system marker, the turn separators and the trailing generation prompt.
  3. Tokenise the rendered string and count tokens per role marker, since those are overhead on every request.
  4. Compare the template against the one used during fine-tuning or by the provider, especially after any model upgrade.
  5. Store the template id and version alongside the prompt version, so a regression can be attributed.
  6. Run the model's own evaluation prompts through the rendered output before serving traffic.

Trade-offs

ChoiceGood forCostFails when
Model-native templateMatches training, best instruction followingCouples your code to one model familyYou need to switch models per request
Provider-managed templateNo rendering code, upgrades handledYou cannot see or pin the exact formatYou self-host the same model elsewhere
Hand-written formatPortable across modelsSilently degrades models trained on another formatQuality matters and the model is instruction-tuned
Custom template for fine-tuningControl over special tokensMust be reproduced exactly at inferenceAnother team owns the serving path
Trade-off mapInteractive

Good for

Matches training, best instruction following

Cost

Couples your code to one model family

Fails when

You need to switch models per request

  1. You need to switch models per request
  2. You self-host the same model elsewhere
  3. Quality matters and the model is instruction-tuned
  4. Another team owns the serving path

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Model-native template is good for Matches training, best instruction following, costs Couples your code to one model family, and fails when You need to switch models per request. Provider-managed template is good for No rendering code, upgrades handled, costs You cannot see or pin the exact format, and fails when You self-host the same model elsewhere. Hand-written format is good for Portable across models, costs Silently degrades models trained on another format, and fails when Quality matters and the model is instruction-tuned. Custom template for fine-tuning is good for Control over special tokens, costs Must be reproduced exactly at inference, and fails when Another team owns the serving path

Failure modes

  • Template mismatch. The model answers but ignores role boundaries or leaks formatting. Debug: print the rendered prompt and compare it byte for byte with the template shipped with the model.
  • Missing generation prompt. The model continues the last turn instead of starting a new one, or emits an empty response. Debug: check the tail of the rendered string for the assistant header.
  • Double-escaped content. User text containing template markers is re-rendered and breaks the structure. Debug: render input that contains the marker strings and inspect where they land.

In production

Pin the template with the model version and assert the rendered prefix in a test. When a provider changes the template, treat it as a model change and re-run the evaluation.

Common misconceptions

People think: Any clear role format works equally well.

Actually: Instruction-tuned models are sensitive to the exact markers they saw in training; a clear but unfamiliar format can underperform.

People think: The template is the provider's problem.

Actually: If you self-host or fine-tune, you own the rendering and every token it adds.

Interview angle

Say that the template is part of the model interface, show how it is rendered and tokenised, and name the failure it causes when wrong. A strong answer also mentions versioning the template with the model. A weak one treats it as string formatting and has no way to detect a mismatch.

Sources