Why it matters
Chat models are trained on transcripts with turn structure, so the interface exposes turns rather than a flat prompt. That structure lets an application separate instructions from user text and tool output. It also creates the main operational cost of chat: the client owns the history, resends it on every call, and must decide what to keep when the transcript outgrows the context window.
Mental model
Messages are the rows of a conversation log, each with a speaker column, and the log is replayed in full on every request. The analogy breaks because a real log is append-only for the reader, while here the client can edit or drop rows before sending, changing the conversation the model believes it had.
How it works
- 1Represent the conversation as an ordered array of role and content objects, with ids you control rather than provider-assigned ones.
- 2Keep the system or developer turn first and stable, so it can be cached and diffed between versions.
- 3Append user and assistant turns in order, and store tool calls and their results as their own turns rather than as prose.
- 4Validate the sequence before sending: alternating roles where required, no orphan tool result, no empty content.
- 5Trim or summarise from the oldest turns when the total approaches the context limit, keeping the first turn and the most recent ones.
- 6Serialise the exact request payload for debugging, since the provider's rendering is not visible in the response.
Step 1 of 6
Represent the conversation as an ordered array of role and content objects, with ids you control rather than provider-assigned ones.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Represent the conversation as an ordered array of role and content objects, with ids you … 2. Keep the system or developer turn first and stable, so it can be cached and diffed betwee… 3. Append user and assistant turns in order, and store tool calls and their results as their… 4. Validate the sequence before sending: alternating roles where required, no orphan tool re… 5. Trim or summarise from the oldest turns when the total approaches the context limit, keep… 6. Serialise the exact request payload for debugging, since the provider's rendering is not …
- Represent the conversation as an ordered array of role and content objects, with ids you control rather than provider-assigned ones.
- Keep the system or developer turn first and stable, so it can be cached and diffed between versions.
- Append user and assistant turns in order, and store tool calls and their results as their own turns rather than as prose.
- Validate the sequence before sending: alternating roles where required, no orphan tool result, no empty content.
- Trim or summarise from the oldest turns when the total approaches the context limit, keeping the first turn and the most recent ones.
- Serialise the exact request payload for debugging, since the provider's rendering is not visible in the response.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Full history every call | Model sees the real conversation | Cost grows with every turn | Sessions run for hundreds of turns |
| Sliding window | Bounded cost, simple | Loses early commitments and identifiers | The user refers back to an early constraint |
| Summary of older turns | Keeps the gist at low cost | Summariser errors become permanent context | Precise details from early turns matter |
| Server-side conversation objects | Less client bookkeeping | Provider lock-in and opaque retention | You need to migrate or audit history |
Good for
Model sees the real conversation
Cost
Cost grows with every turn
Fails when
Sessions run for hundreds of turns
- Sessions run for hundreds of turns
- The user refers back to an early constraint
- Precise details from early turns matter
- You need to migrate or audit history
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full history every call is good for Model sees the real conversation, costs Cost grows with every turn, and fails when Sessions run for hundreds of turns. Sliding window is good for Bounded cost, simple, costs Loses early commitments and identifiers, and fails when The user refers back to an early constraint. Summary of older turns is good for Keeps the gist at low cost, costs Summariser errors become permanent context, and fails when Precise details from early turns matter. Server-side conversation objects is good for Less client bookkeeping, costs Provider lock-in and opaque retention, and fails when You need to migrate or audit history
Failure modes
- Orphaned tool result. A tool message appears without the assistant call that produced it and the request is rejected. Debug: validate the sequence against the provider's pairing rules before sending.
- History divergence. The stored transcript differs from what was sent, so replays do not reproduce the bug. Debug: log the exact message array and compare it with what the UI shows.
- Role alternation assumption. Two user turns in a row are merged or rejected depending on the provider. Debug: run the same sequence against each provider and inspect the rendered prompt.
In production
Track prompt tokens per turn and per session, and cap history length per route. Store the exact request payload for sampled sessions so a bad answer can be replayed.
Common misconceptions
People think: Messages are just formatting around one prompt string.
Actually: Turn boundaries become special tokens and change conditioning, so merging turns is a semantic change.
People think: The API remembers the conversation.
Actually: Most chat APIs are stateless; the client resends the history and pays for it every time.
Interview angle
Describe messages as typed turns that the server renders with a template, then focus on the consequences: history ownership, token growth and tool-call pairing. Strong answers mention validation and replay. Weak answers call messages a JSON wrapper and miss that turn structure is part of the model input.
Sources
- Anthropic, Messages API reference — the turn list, content blocks and system parameter.
- OpenAI, Chat Completions API reference — roles, message objects and tool messages.
- Hugging Face, Chat templating — how a message list becomes tokens.