Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 22 of 26

Conversation state

0 of 26 complete

Why it matters

A chat API accepts a list of messages and returns one, with no memory between calls. The application therefore decides what the conversation is: which turns to keep, how to represent tool calls, and what to do when the transcript outgrows the window. That record is also the audit trail for a bad answer. When it is implicit, spread across UI state and logs, neither trimming nor replay is reliable.

Mental model

Conversation state is a case file that is photocopied and handed over at every meeting. The comparison breaks because the file is edited between meetings, and the other party only sees the version you handed them. The original is not in the room to correct the copy.

How it works

Mechanism, step by stepInteractive
  1. 1Define a state schema: turns with ids, roles, timestamps, tool calls and outcomes, plus pinned facts that must survive trimming.
  2. 2Persist it server-side keyed by conversation id, and treat the client as a view rather than the source of truth.
  3. 3Build each request's message list from state with a deterministic policy: pinned items, recent turns, then summary.
  4. 4Store the exact rendered message list with the response, so a failure can be replayed.
  5. 5Handle edits and deletions explicitly, since a corrected user turn changes later context.
  6. 6Expire state on a defined schedule and tell the user what is retained.

Step 1 of 6

Define a state schema: turns with ids, roles, timestamps, tool calls and outcomes, plus pinned facts that must survive trimming.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Define a state schema: turns with ids, roles, timestamps, tool calls and outcomes, plus p… 2. Persist it server-side keyed by conversation id, and treat the client as a view rather th… 3. Build each request's message list from state with a deterministic policy: pinned items, r… 4. Store the exact rendered message list with the response, so a failure can be replayed. 5. Handle edits and deletions explicitly, since a corrected user turn changes later context. 6. Expire state on a defined schedule and tell the user what is retained.

  1. Define a state schema: turns with ids, roles, timestamps, tool calls and outcomes, plus pinned facts that must survive trimming.
  2. Persist it server-side keyed by conversation id, and treat the client as a view rather than the source of truth.
  3. Build each request's message list from state with a deterministic policy: pinned items, recent turns, then summary.
  4. Store the exact rendered message list with the response, so a failure can be replayed.
  5. Handle edits and deletions explicitly, since a corrected user turn changes later context.
  6. Expire state on a defined schedule and tell the user what is retained.

Trade-offs

ChoiceGood forCostFails when
Server-side stateConsistent trimming and auditStorage and privacy obligationsThe product is multi-device and offline
Client-held stateNo server storageTampering and divergenceThe conversation is long or sensitive
Full transcriptFaithful contextCost grows with every turnSessions are long
Summary plus recent turnsBounded costLossy and hard to auditExact earlier wording matters
Trade-off mapInteractive

Good for

Consistent trimming and audit

Cost

Storage and privacy obligations

Fails when

The product is multi-device and offline

  1. The product is multi-device and offline
  2. The conversation is long or sensitive
  3. Sessions are long
  4. Exact earlier wording matters

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Server-side state is good for Consistent trimming and audit, costs Storage and privacy obligations, and fails when The product is multi-device and offline. Client-held state is good for No server storage, costs Tampering and divergence, and fails when The conversation is long or sensitive. Full transcript is good for Faithful context, costs Cost grows with every turn, and fails when Sessions are long. Summary plus recent turns is good for Bounded cost, costs Lossy and hard to audit, and fails when Exact earlier wording matters

Failure modes

  • State divergence. The UI shows a corrected message but the server still sends the original. Debug: compare the persisted turn with the rendered request for the same conversation id.
  • Lost pinned fact. Trimming removes an early constraint and the assistant contradicts it. Debug: check whether pinned items are exempt from trimming and present in the rendered request.
  • Tool state dropped. A pending tool call is trimmed and the next turn is rejected. Debug: assert that every tool result has its originating call in the same request.

In production

Log conversation id and a hash of the rendered messages per request, and cap stored turns per conversation. Review retention and deletion with the same rigour as any user data.

Common misconceptions

People think: The model remembers the conversation.

Actually: It receives whatever list you send; memory is an application feature built on that list.

People think: Trimming is just dropping old messages.

Actually: It must preserve pinned constraints and tool pairings, or the conversation becomes inconsistent.

Interview angle

Describe state as the source of truth and the request as a projection of it, then cover trimming and replay. Strong answers include pinned facts and tool-call pairing. Weak answers keep the transcript in the browser and cannot explain how a reported answer was produced.

Sources