Why it matters
A conversation or an agent session outlives any single context window. Teams often conflate the two and try to keep everything in the prompt, which fails on cost and attention. Memory separates storage from presentation: facts, preferences and past outcomes are persisted, then selectively retrieved. The engineering problem is the retrieval and write policy, because a memory that returns stale or irrelevant items is worse than no memory.
Mental model
Memory is a filing cabinet and context is the folder on the desk. The comparison breaks because the cabinet can be searched only by similarity, so the right folder may not surface, and nothing tells you a better folder exists.
How it works
- 1Decide what belongs in memory: stable user facts, decisions, outcomes and corrections, not raw transcripts.
- 2Choose a store and an index: key-value for exact facts, vector for semantic recall, both when needed.
- 3Define a write policy with deduplication and a timestamp, so the same fact is not stored repeatedly.
- 4Retrieve a small number of memories per request, scored by relevance and recency, and mark them as memory in the prompt.
- 5Give the user a way to inspect and delete memories, and expire them on a schedule.
- 6Evaluate memory-assisted answers against a no-memory baseline, including cases where memory is wrong.
Step 1 of 6
Decide what belongs in memory: stable user facts, decisions, outcomes and corrections, not raw transcripts.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Decide what belongs in memory: stable user facts, decisions, outcomes and corrections, no… 2. Choose a store and an index: key-value for exact facts, vector for semantic recall, both … 3. Define a write policy with deduplication and a timestamp, so the same fact is not stored … 4. Retrieve a small number of memories per request, scored by relevance and recency, and mar… 5. Give the user a way to inspect and delete memories, and expire them on a schedule. 6. Evaluate memory-assisted answers against a no-memory baseline, including cases where memo…
- Decide what belongs in memory: stable user facts, decisions, outcomes and corrections, not raw transcripts.
- Choose a store and an index: key-value for exact facts, vector for semantic recall, both when needed.
- Define a write policy with deduplication and a timestamp, so the same fact is not stored repeatedly.
- Retrieve a small number of memories per request, scored by relevance and recency, and mark them as memory in the prompt.
- Give the user a way to inspect and delete memories, and expire them on a schedule.
- Evaluate memory-assisted answers against a no-memory baseline, including cases where memory is wrong.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Vector memory | Semantic recall of past context | Similarity errors and stale entries | Exact identifiers matter |
| Key-value memory | Precise facts | Requires a schema and extraction | The relevant memory is unstructured |
| Write everything | Nothing is lost | Noise, cost and privacy exposure | Sessions are long-lived |
| Write selectively | Clean store | Extraction errors lose facts | The value of a fact is unclear at write time |
Good for
Semantic recall of past context
Cost
Similarity errors and stale entries
Fails when
Exact identifiers matter
- Exact identifiers matter
- The relevant memory is unstructured
- Sessions are long-lived
- The value of a fact is unclear at write time
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Vector memory is good for Semantic recall of past context, costs Similarity errors and stale entries, and fails when Exact identifiers matter. Key-value memory is good for Precise facts, costs Requires a schema and extraction, and fails when The relevant memory is unstructured. Write everything is good for Nothing is lost, costs Noise, cost and privacy exposure, and fails when Sessions are long-lived. Write selectively is good for Clean store, costs Extraction errors lose facts, and fails when The value of a fact is unclear at write time
Failure modes
- Stale memory. An old preference overrides the current request. Debug: check the timestamp and whether a newer memory contradicts the one used.
- Memory poisoning. A wrong or injected fact is stored and resurfaces later. Debug: trace the memory id to the turn that wrote it and review the write policy.
- Recall without use. Retrieved memories are in the prompt but the answer ignores them. Debug: check placement and whether the memory is marked as authoritative.
In production
Monitor memory hit rate, the share of answers that use a retrieved memory, and deletion requests. Cap memories per user and expire them by type.
Common misconceptions
People think: Memory is a longer context window.
Actually: It is an external store with its own retrieval and write policies; the window only sees what is retrieved.
People think: More memory improves personalisation.
Actually: Irrelevant or stale memories add noise and can make answers confidently wrong.
Interview angle
Draw the line between storage and the per-call window, then spend the answer on write and retrieval policy. Strong answers include staleness, poisoning and deletion. Weak answers say memory means a bigger context and never say what gets written or forgotten.
Sources
- Packer et al., MemGPT — an explicit memory hierarchy with paging between store and context.
- Park et al., Generative Agents — a memory stream with retrieval, reflection and recency.
- Shinn et al., Reflexion — persisting lessons across attempts as episodic memory.