Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 23 of 26

Memory vs context

0 of 26 complete

Why it matters

A conversation or an agent session outlives any single context window. Teams often conflate the two and try to keep everything in the prompt, which fails on cost and attention. Memory separates storage from presentation: facts, preferences and past outcomes are persisted, then selectively retrieved. The engineering problem is the retrieval and write policy, because a memory that returns stale or irrelevant items is worse than no memory.

Mental model

Memory is a filing cabinet and context is the folder on the desk. The comparison breaks because the cabinet can be searched only by similarity, so the right folder may not surface, and nothing tells you a better folder exists.

How it works

Mechanism, step by stepInteractive
  1. 1Decide what belongs in memory: stable user facts, decisions, outcomes and corrections, not raw transcripts.
  2. 2Choose a store and an index: key-value for exact facts, vector for semantic recall, both when needed.
  3. 3Define a write policy with deduplication and a timestamp, so the same fact is not stored repeatedly.
  4. 4Retrieve a small number of memories per request, scored by relevance and recency, and mark them as memory in the prompt.
  5. 5Give the user a way to inspect and delete memories, and expire them on a schedule.
  6. 6Evaluate memory-assisted answers against a no-memory baseline, including cases where memory is wrong.

Step 1 of 6

Decide what belongs in memory: stable user facts, decisions, outcomes and corrections, not raw transcripts.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Decide what belongs in memory: stable user facts, decisions, outcomes and corrections, no… 2. Choose a store and an index: key-value for exact facts, vector for semantic recall, both … 3. Define a write policy with deduplication and a timestamp, so the same fact is not stored … 4. Retrieve a small number of memories per request, scored by relevance and recency, and mar… 5. Give the user a way to inspect and delete memories, and expire them on a schedule. 6. Evaluate memory-assisted answers against a no-memory baseline, including cases where memo…

  1. Decide what belongs in memory: stable user facts, decisions, outcomes and corrections, not raw transcripts.
  2. Choose a store and an index: key-value for exact facts, vector for semantic recall, both when needed.
  3. Define a write policy with deduplication and a timestamp, so the same fact is not stored repeatedly.
  4. Retrieve a small number of memories per request, scored by relevance and recency, and mark them as memory in the prompt.
  5. Give the user a way to inspect and delete memories, and expire them on a schedule.
  6. Evaluate memory-assisted answers against a no-memory baseline, including cases where memory is wrong.

Trade-offs

ChoiceGood forCostFails when
Vector memorySemantic recall of past contextSimilarity errors and stale entriesExact identifiers matter
Key-value memoryPrecise factsRequires a schema and extractionThe relevant memory is unstructured
Write everythingNothing is lostNoise, cost and privacy exposureSessions are long-lived
Write selectivelyClean storeExtraction errors lose factsThe value of a fact is unclear at write time
Trade-off mapInteractive

Good for

Semantic recall of past context

Cost

Similarity errors and stale entries

Fails when

Exact identifiers matter

  1. Exact identifiers matter
  2. The relevant memory is unstructured
  3. Sessions are long-lived
  4. The value of a fact is unclear at write time

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Vector memory is good for Semantic recall of past context, costs Similarity errors and stale entries, and fails when Exact identifiers matter. Key-value memory is good for Precise facts, costs Requires a schema and extraction, and fails when The relevant memory is unstructured. Write everything is good for Nothing is lost, costs Noise, cost and privacy exposure, and fails when Sessions are long-lived. Write selectively is good for Clean store, costs Extraction errors lose facts, and fails when The value of a fact is unclear at write time

Failure modes

  • Stale memory. An old preference overrides the current request. Debug: check the timestamp and whether a newer memory contradicts the one used.
  • Memory poisoning. A wrong or injected fact is stored and resurfaces later. Debug: trace the memory id to the turn that wrote it and review the write policy.
  • Recall without use. Retrieved memories are in the prompt but the answer ignores them. Debug: check placement and whether the memory is marked as authoritative.

In production

Monitor memory hit rate, the share of answers that use a retrieved memory, and deletion requests. Cap memories per user and expire them by type.

Common misconceptions

People think: Memory is a longer context window.

Actually: It is an external store with its own retrieval and write policies; the window only sees what is retrieved.

People think: More memory improves personalisation.

Actually: Irrelevant or stale memories add noise and can make answers confidently wrong.

Interview angle

Draw the line between storage and the per-call window, then spend the answer on write and retrieval policy. Strong answers include staleness, poisoning and deletion. Weak answers say memory means a bigger context and never say what gets written or forgotten.

Sources