Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 53 of 57

Document versioning

0 of 57 complete

Why it matters

Answers outlive the documents they came from. When a policy is revised, an answer given last month cites text that no longer exists at the same location, and without version identity the citation either breaks or silently points at different words. Versioning separates the stable document from its revisions, so the index can move forward while past answers remain explainable.

Mental model

Versioning is a contract archive: the contract has one file number, and each amendment is a dated revision inside it. The metaphor breaks because an amendment explicitly says what it changes, while a re-uploaded document is usually just a new blob with no diff.

How it works

Mechanism, step by stepInteractive
  1. 1Assign a stable document id from the source system and a version id from the content hash plus revision metadata.
  2. 2Derive chunk ids from document id, version id and position, so a new version never overwrites an old chunk.
  3. 3Index the new version and mark the previous one superseded in the same operation, rather than deleting first.
  4. 4Store the version id in every citation so an answer can be resolved to the exact revision.
  5. 5Retain superseded versions for the audit window, then expire them with a documented policy.
  6. 6Diff versions to find changed sections and re-embed only those, instead of rebuilding the document.

Step 1 of 6

Assign a stable document id from the source system and a version id from the content hash plus revision metadata.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Assign a stable document id from the source system and a version id from the content hash… 2. Derive chunk ids from document id, version id and position, so a new version never overwr… 3. Index the new version and mark the previous one superseded in the same operation, rather … 4. Store the version id in every citation so an answer can be resolved to the exact revision. 5. Retain superseded versions for the audit window, then expire them with a documented polic… 6. Diff versions to find changed sections and re-embed only those, instead of rebuilding the…

  1. Assign a stable document id from the source system and a version id from the content hash plus revision metadata.
  2. Derive chunk ids from document id, version id and position, so a new version never overwrites an old chunk.
  3. Index the new version and mark the previous one superseded in the same operation, rather than deleting first.
  4. Store the version id in every citation so an answer can be resolved to the exact revision.
  5. Retain superseded versions for the audit window, then expire them with a documented policy.
  6. Diff versions to find changed sections and re-embed only those, instead of rebuilding the document.

Trade-offs

ChoiceGood forCostFails when
Keep all versionsFull audit and historical answersStorage and near-duplicate resultsRanking cannot tell current from superseded
Current version onlySmall index and simple rankingPast answers become unverifiableThe domain has regulatory or legal exposure
Version per content hashDetects any changeNew version for whitespace-only editsDocuments are re-saved frequently
Version per source revisionMatches the source's own historyMisses changes the source does not versionThe source overwrites files in place
Trade-off mapInteractive

Good for

Full audit and historical answers

Cost

Storage and near-duplicate results

Fails when

Ranking cannot tell current from superseded

  1. Ranking cannot tell current from superseded
  2. The domain has regulatory or legal exposure
  3. Documents are re-saved frequently
  4. The source overwrites files in place

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Keep all versions is good for Full audit and historical answers, costs Storage and near-duplicate results, and fails when Ranking cannot tell current from superseded. Current version only is good for Small index and simple ranking, costs Past answers become unverifiable, and fails when The domain has regulatory or legal exposure. Version per content hash is good for Detects any change, costs New version for whitespace-only edits, and fails when Documents are re-saved frequently. Version per source revision is good for Matches the source's own history, costs Misses changes the source does not version, and fails when The source overwrites files in place

Failure modes

  • Chunk id collision across versions. The new version overwrites the old chunk and a citation now points at different text. Debug: assert chunk ids include the version id and that an old citation still resolves to its original text.
  • Superseded documents still retrieved. Both versions rank together and the older one sometimes wins. Debug: query for a known revised fact and check which version appears in the top results.
  • Unversioned in-place overwrite. The source replaces a file and the index has no record of the previous content. Debug: compare content hashes over time for a sample of documents and look for changes with no new version row.

In production

Monitor the count of open versions per document and the age of the oldest retained revision. Alert when a document changes without a new version record.

Common misconceptions

People think: Versioning is the same as re-indexing.

Actually: Re-indexing replaces content; versioning preserves the identity of what was replaced.

People think: Users only care about the current version.

Actually: Anyone checking a past answer cares about the version that produced it, and auditors care even more.

Interview angle

Separate document identity from version identity, and show how chunk ids and citations carry the version. Cover supersession and retention. Re-indexing when a document changes, with no plan for existing citations, is the incomplete answer.

Sources