Why it matters
Instruction-tuned models learn to follow roles through a particular serialisation: markers before each turn, a separator after it, sometimes a generation prompt at the end. The weights are conditioned on those exact tokens. An application that invents its own format, or that upgrades a model without its template, sends a prompt the model has never seen and gets degraded instruction following that looks like a quality problem.
Mental model
A chat template is the envelope format a postal system expects: address block here, stamp there. Put the address in the wrong corner and the letter still travels, just not where you meant. The image breaks because a misformatted prompt is not returned to sender; it is answered, badly, with no error.
How it works
Rendered prompt · illustrative tokens
405 input+ 120 reserve= 525 of 600
75 free
Step 1Messages as objects
The application holds four messages with roles. They are laid side by side here only to compare token cost — no separators exist yet, and nothing here is a token stream.
Output reserve
120Pick a section to read its token count, its role and its state.
Figure. A message list is not what the model receives. The template adds role markers, turn separators and a generation prompt around the authored text, and every one of those tokens is charged to the same budget.
What to notice. The marker shapes are illustrative and do not reproduce any named model's template. What carries over is that the exact markers are model-specific, part of the model interface, and counted in the prompt.
Text version
A 600-token window with a 120-token output reserve. The default first step lays out four messages as separate objects — system 180, user 60, assistant 120 and user 45 — with a row label saying the widths compare token cost only; they are not yet a stream. The second step inserts a BOS token, a system header, three end-of-turn separators, and highlights them as markers. The third step adds the generation prompt and renders one stream of 428 input tokens. The fourth step highlights the markers and counts 23 overhead tokens, about 5% of the prompt. The fifth step highlights the generation prompt at the tail and notes what a missing one causes. Selecting any segment shows its count and role.
- Load the template that ships with the model rather than writing role markers by hand.
- Render a two-turn conversation and read the raw string, checking the system marker, the turn separators and the trailing generation prompt.
- Tokenise the rendered string and count tokens per role marker, since those are overhead on every request.
- Compare the template against the one used during fine-tuning or by the provider, especially after any model upgrade.
- Store the template id and version alongside the prompt version, so a regression can be attributed.
- Run the model's own evaluation prompts through the rendered output before serving traffic.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Model-native template | Matches training, best instruction following | Couples your code to one model family | You need to switch models per request |
| Provider-managed template | No rendering code, upgrades handled | You cannot see or pin the exact format | You self-host the same model elsewhere |
| Hand-written format | Portable across models | Silently degrades models trained on another format | Quality matters and the model is instruction-tuned |
| Custom template for fine-tuning | Control over special tokens | Must be reproduced exactly at inference | Another team owns the serving path |
Good for
Matches training, best instruction following
Cost
Couples your code to one model family
Fails when
You need to switch models per request
- You need to switch models per request
- You self-host the same model elsewhere
- Quality matters and the model is instruction-tuned
- Another team owns the serving path
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Model-native template is good for Matches training, best instruction following, costs Couples your code to one model family, and fails when You need to switch models per request. Provider-managed template is good for No rendering code, upgrades handled, costs You cannot see or pin the exact format, and fails when You self-host the same model elsewhere. Hand-written format is good for Portable across models, costs Silently degrades models trained on another format, and fails when Quality matters and the model is instruction-tuned. Custom template for fine-tuning is good for Control over special tokens, costs Must be reproduced exactly at inference, and fails when Another team owns the serving path
Failure modes
- Template mismatch. The model answers but ignores role boundaries or leaks formatting. Debug: print the rendered prompt and compare it byte for byte with the template shipped with the model.
- Missing generation prompt. The model continues the last turn instead of starting a new one, or emits an empty response. Debug: check the tail of the rendered string for the assistant header.
- Double-escaped content. User text containing template markers is re-rendered and breaks the structure. Debug: render input that contains the marker strings and inspect where they land.
In production
Pin the template with the model version and assert the rendered prefix in a test. When a provider changes the template, treat it as a model change and re-run the evaluation.
Common misconceptions
People think: Any clear role format works equally well.
Actually: Instruction-tuned models are sensitive to the exact markers they saw in training; a clear but unfamiliar format can underperform.
People think: The template is the provider's problem.
Actually: If you self-host or fine-tune, you own the rendering and every token it adds.
Interview angle
Say that the template is part of the model interface, show how it is rendered and tokenised, and name the failure it causes when wrong. A strong answer also mentions versioning the template with the model. A weak one treats it as string formatting and has no way to detect a mismatch.
Sources
- Hugging Face, Chat templating — templates stored with models and applied to message lists.
- Hugging Face, Tokenizers documentation — special tokens and the tokenisation pipeline.
- Ouyang et al., Training language models to follow instructions — instruction tuning, which is why role structure affects behaviour.