Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 26 of 27

Data versioning

0 of 27 complete

Why it matters

Models are retrained, compared and audited long after the data has changed underneath them. Without a data version, a metric cannot be reproduced, a regression cannot be attributed, and a deleted source cannot be recovered. The engineering problem is scale: data is too large for ordinary source control, changes by appending rather than editing, and often lives in systems with their own retention rules.

Mental model

Data versioning is a lab notebook for ingredients: each batch gets a label, and every result references the batch it used. The metaphor breaks because ingredients are not static — a dataset can be mutated in place by an upstream job, so the label must be tied to content rather than to a location.

How it works

When the path stays the same and the bytes do notInteractive
workversionriskevent
6 d

Event · step 1 of 6

6 d · run A is reproducible

At 6 d

Run A records manifest m-104, split-3 and transform t-7. The manifest points at shard s-01 by content hash, so a later rebuild reads the same bytes or fails loudly.

On screen at this event

  • s-01
  • m-104
  • split-3
  • t-7
  • A
  • matches

Next at 9 d: 9 d · the shard is rewritten in place

Manifest
m-104
Split
split-3
Rebuild test
A ok
Comparable
—
Old shard
readable
Step 1/6 · 6 d · run A is reproducible

Figure. A synthetic 30-day history: run A records manifest m-104, the shard at path s-01 is rewritten in place on day 9, runs A and B can no longer rebuild, and a reseed on day 20 makes run C incomparable. Hashes, dates and run ids are illustrative.

What to notice. Content hashing is what makes the rewrite visible: the path never changes, the hash does, and the rebuild fails at the first shard. The split is a separate artefact, so a reseed breaks comparison even when the data hash matches. Pin manifest, split, transform and code revision in every run record, and rehearse a restore instead of trusting a versioning scheme that has never been restored.

Text version

Six lanes over 30 days: raw shard, manifest, split, transform, rebuild check and training runs. Shard s-01 runs 0 to 9, then new bytes in red to 30. Manifest m-104 runs 0 to 9 and m-105 to 30. Split-3 runs 0 to 20 and split-4 20 to 30. Transform t-7 spans the whole window. The rebuild check matches 0 to 9 and reports mismatch in red from 9. Runs are A 4 to 6, B 6 to 8 and C 22 to 24; A and B both pin m-104, C pins m-105 and split-4. Markers read in-place rewrite at day 9 and split reseeded at day 20. Counters: manifest m-104, m-105, m-105, m-105, m-105, m-105; split split-3 three times then split-4; rebuild A ok at day 6, A fail at day 9, then A and B fail; comparable dashes until day 24 then no; old shard readable, readable, then gone.

  1. Store raw data immutably and give each shard a content hash rather than relying on its path.
  2. Keep large files in object storage and version the manifest that points to them, not the bytes in git.
  3. Version the split definition separately, so the same rows always land in the same partition.
  4. Record the transform code and its parameters alongside the manifest, since the same rows processed differently are a different dataset.
  5. Tag each trained model with the dataset hash, split hash and code commit.
  6. Test restoration by rebuilding a past training set from its manifest and comparing hashes.

Trade-offs

ChoiceGood forCostFails when
Content-addressed storageExact reproducibility, deduplicationStorage overhead and toolingData changes continuously and must be mutable
Pointer-based versioningCheap, works with existing storageBreaks when the underlying object is overwrittenUpstream jobs mutate in place
Full snapshotsSimple restore, no dependency on historyStorage cost grows with every versionThe dataset is large and versioned often
Lineage metadata onlyLightweight, queryableCannot reconstruct the data itselfAudits require the original rows
Trade-off mapInteractive

Good for

Exact reproducibility, deduplication

Cost

Storage overhead and tooling

Fails when

Data changes continuously and must be mutable

  1. Data changes continuously and must be mutable
  2. Upstream jobs mutate in place
  3. The dataset is large and versioned often
  4. Audits require the original rows

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Content-addressed storage is good for Exact reproducibility, deduplication, costs Storage overhead and tooling, and fails when Data changes continuously and must be mutable. Pointer-based versioning is good for Cheap, works with existing storage, costs Breaks when the underlying object is overwritten, and fails when Upstream jobs mutate in place. Full snapshots is good for Simple restore, no dependency on history, costs Storage cost grows with every version, and fails when The dataset is large and versioned often. Lineage metadata only is good for Lightweight, queryable, costs Cannot reconstruct the data itself, and fails when Audits require the original rows

Failure modes

  • Mutable source path. A dataset is regenerated in place and an old model can no longer be reproduced. Debug: attempt to rebuild a past run from its recorded manifest and compare hashes.
  • Split drift. Rows are re-split with a new seed and metrics are compared across incompatible partitions. Debug: hash the split assignment and check it matches the run's metadata.
  • Transform not recorded. The data hash matches but preprocessing code changed, so the model saw different inputs. Debug: include the transform version in the model's lineage record.

In production

Make the dataset hash a required field in every training job and model registry entry, and fail the job if it is missing. Schedule a periodic restore test, because a versioning scheme that has never been restored is a hypothesis.

Common misconceptions

People think: Git LFS is a complete data versioning solution.

Actually: It versions files but not the pipeline that produced them, the split assignment or the lineage to a trained model.

People think: Hashing a dataset once at creation is enough.

Actually: Datasets are joined, filtered and relabelled; each derived artefact needs its own identity or the chain breaks.

Interview angle

Break the problem into four artefacts to version — storage, manifest, split and transform — and describe a restore test that proves the chain works. Connect lineage to debugging a regression. Weak answers say 'use DVC' without saying what is hashed, what a split artefact is, or how a model is tied to its inputs.

Sources