Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 8 of 28

Messages

0 of 28 complete

Why it matters

Chat models are trained on transcripts with turn structure, so the interface exposes turns rather than a flat prompt. That structure lets an application separate instructions from user text and tool output. It also creates the main operational cost of chat: the client owns the history, resends it on every call, and must decide what to keep when the transcript outgrows the context window.

Mental model

Messages are the rows of a conversation log, each with a speaker column, and the log is replayed in full on every request. The analogy breaks because a real log is append-only for the reader, while here the client can edit or drop rows before sending, changing the conversation the model believes it had.

How it works

Mechanism, step by stepInteractive
  1. 1Represent the conversation as an ordered array of role and content objects, with ids you control rather than provider-assigned ones.
  2. 2Keep the system or developer turn first and stable, so it can be cached and diffed between versions.
  3. 3Append user and assistant turns in order, and store tool calls and their results as their own turns rather than as prose.
  4. 4Validate the sequence before sending: alternating roles where required, no orphan tool result, no empty content.
  5. 5Trim or summarise from the oldest turns when the total approaches the context limit, keeping the first turn and the most recent ones.
  6. 6Serialise the exact request payload for debugging, since the provider's rendering is not visible in the response.

Step 1 of 6

Represent the conversation as an ordered array of role and content objects, with ids you control rather than provider-assigned ones.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Represent the conversation as an ordered array of role and content objects, with ids you … 2. Keep the system or developer turn first and stable, so it can be cached and diffed betwee… 3. Append user and assistant turns in order, and store tool calls and their results as their… 4. Validate the sequence before sending: alternating roles where required, no orphan tool re… 5. Trim or summarise from the oldest turns when the total approaches the context limit, keep… 6. Serialise the exact request payload for debugging, since the provider's rendering is not …

  1. Represent the conversation as an ordered array of role and content objects, with ids you control rather than provider-assigned ones.
  2. Keep the system or developer turn first and stable, so it can be cached and diffed between versions.
  3. Append user and assistant turns in order, and store tool calls and their results as their own turns rather than as prose.
  4. Validate the sequence before sending: alternating roles where required, no orphan tool result, no empty content.
  5. Trim or summarise from the oldest turns when the total approaches the context limit, keeping the first turn and the most recent ones.
  6. Serialise the exact request payload for debugging, since the provider's rendering is not visible in the response.

Trade-offs

ChoiceGood forCostFails when
Full history every callModel sees the real conversationCost grows with every turnSessions run for hundreds of turns
Sliding windowBounded cost, simpleLoses early commitments and identifiersThe user refers back to an early constraint
Summary of older turnsKeeps the gist at low costSummariser errors become permanent contextPrecise details from early turns matter
Server-side conversation objectsLess client bookkeepingProvider lock-in and opaque retentionYou need to migrate or audit history
Trade-off mapInteractive

Good for

Model sees the real conversation

Cost

Cost grows with every turn

Fails when

Sessions run for hundreds of turns

  1. Sessions run for hundreds of turns
  2. The user refers back to an early constraint
  3. Precise details from early turns matter
  4. You need to migrate or audit history

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full history every call is good for Model sees the real conversation, costs Cost grows with every turn, and fails when Sessions run for hundreds of turns. Sliding window is good for Bounded cost, simple, costs Loses early commitments and identifiers, and fails when The user refers back to an early constraint. Summary of older turns is good for Keeps the gist at low cost, costs Summariser errors become permanent context, and fails when Precise details from early turns matter. Server-side conversation objects is good for Less client bookkeeping, costs Provider lock-in and opaque retention, and fails when You need to migrate or audit history

Failure modes

  • Orphaned tool result. A tool message appears without the assistant call that produced it and the request is rejected. Debug: validate the sequence against the provider's pairing rules before sending.
  • History divergence. The stored transcript differs from what was sent, so replays do not reproduce the bug. Debug: log the exact message array and compare it with what the UI shows.
  • Role alternation assumption. Two user turns in a row are merged or rejected depending on the provider. Debug: run the same sequence against each provider and inspect the rendered prompt.

In production

Track prompt tokens per turn and per session, and cap history length per route. Store the exact request payload for sampled sessions so a bad answer can be replayed.

Common misconceptions

People think: Messages are just formatting around one prompt string.

Actually: Turn boundaries become special tokens and change conditioning, so merging turns is a semantic change.

People think: The API remembers the conversation.

Actually: Most chat APIs are stateless; the client resends the history and pays for it every time.

Interview angle

Describe messages as typed turns that the server renders with a template, then focus on the consequences: history ownership, token growth and tool-call pairing. Strong answers mention validation and replay. Weak answers call messages a JSON wrapper and miss that turn structure is part of the model input.

Sources