Why it matters
A chat API accepts a list of messages and returns one, with no memory between calls. The application therefore decides what the conversation is: which turns to keep, how to represent tool calls, and what to do when the transcript outgrows the window. That record is also the audit trail for a bad answer. When it is implicit, spread across UI state and logs, neither trimming nor replay is reliable.
Mental model
Conversation state is a case file that is photocopied and handed over at every meeting. The comparison breaks because the file is edited between meetings, and the other party only sees the version you handed them. The original is not in the room to correct the copy.
How it works
- 1Define a state schema: turns with ids, roles, timestamps, tool calls and outcomes, plus pinned facts that must survive trimming.
- 2Persist it server-side keyed by conversation id, and treat the client as a view rather than the source of truth.
- 3Build each request's message list from state with a deterministic policy: pinned items, recent turns, then summary.
- 4Store the exact rendered message list with the response, so a failure can be replayed.
- 5Handle edits and deletions explicitly, since a corrected user turn changes later context.
- 6Expire state on a defined schedule and tell the user what is retained.
Step 1 of 6
Define a state schema: turns with ids, roles, timestamps, tool calls and outcomes, plus pinned facts that must survive trimming.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Define a state schema: turns with ids, roles, timestamps, tool calls and outcomes, plus p… 2. Persist it server-side keyed by conversation id, and treat the client as a view rather th… 3. Build each request's message list from state with a deterministic policy: pinned items, r… 4. Store the exact rendered message list with the response, so a failure can be replayed. 5. Handle edits and deletions explicitly, since a corrected user turn changes later context. 6. Expire state on a defined schedule and tell the user what is retained.
- Define a state schema: turns with ids, roles, timestamps, tool calls and outcomes, plus pinned facts that must survive trimming.
- Persist it server-side keyed by conversation id, and treat the client as a view rather than the source of truth.
- Build each request's message list from state with a deterministic policy: pinned items, recent turns, then summary.
- Store the exact rendered message list with the response, so a failure can be replayed.
- Handle edits and deletions explicitly, since a corrected user turn changes later context.
- Expire state on a defined schedule and tell the user what is retained.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Server-side state | Consistent trimming and audit | Storage and privacy obligations | The product is multi-device and offline |
| Client-held state | No server storage | Tampering and divergence | The conversation is long or sensitive |
| Full transcript | Faithful context | Cost grows with every turn | Sessions are long |
| Summary plus recent turns | Bounded cost | Lossy and hard to audit | Exact earlier wording matters |
Good for
Consistent trimming and audit
Cost
Storage and privacy obligations
Fails when
The product is multi-device and offline
- The product is multi-device and offline
- The conversation is long or sensitive
- Sessions are long
- Exact earlier wording matters
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Server-side state is good for Consistent trimming and audit, costs Storage and privacy obligations, and fails when The product is multi-device and offline. Client-held state is good for No server storage, costs Tampering and divergence, and fails when The conversation is long or sensitive. Full transcript is good for Faithful context, costs Cost grows with every turn, and fails when Sessions are long. Summary plus recent turns is good for Bounded cost, costs Lossy and hard to audit, and fails when Exact earlier wording matters
Failure modes
- State divergence. The UI shows a corrected message but the server still sends the original. Debug: compare the persisted turn with the rendered request for the same conversation id.
- Lost pinned fact. Trimming removes an early constraint and the assistant contradicts it. Debug: check whether pinned items are exempt from trimming and present in the rendered request.
- Tool state dropped. A pending tool call is trimmed and the next turn is rejected. Debug: assert that every tool result has its originating call in the same request.
In production
Log conversation id and a hash of the rendered messages per request, and cap stored turns per conversation. Review retention and deletion with the same rigour as any user data.
Common misconceptions
People think: The model remembers the conversation.
Actually: It receives whatever list you send; memory is an application feature built on that list.
People think: Trimming is just dropping old messages.
Actually: It must preserve pinned constraints and tool pairings, or the conversation becomes inconsistent.
Interview angle
Describe state as the source of truth and the request as a projection of it, then cover trimming and replay. Strong answers include pinned facts and tool-call pairing. Weak answers keep the transcript in the browser and cannot explain how a reported answer was produced.
Sources
- Packer et al., MemGPT — managing conversation history as an explicit memory hierarchy.
- Anthropic, Messages API reference — a stateless turn list supplied per request.
- Park et al., Generative Agents — a memory stream that outlives any single prompt.