Skip to content
AI Engineer Guide.

concept · data · intermediate · concept 16 of 17

Data versioning & lineage

0 of 17 complete

Why it matters

A model or a report is only trustworthy if you can say what it was computed from. In practice data changes underneath a query: a source corrects a value, a job is rerun with new code, and last month's figure can no longer be produced. Versioning and lineage exist to answer two operational questions, what exactly produced this number and what depends on this input, before either question becomes an audit.

Mental model

A versioned dataset is a commit history for data; lineage is the dependency graph a build system uses. The metaphor breaks because data commits are not reviewed the way code is, so an unreviewed snapshot can still be a faithful record of a wrong load.

How it works

A fact snapshot, an older dimension, seven days of nullsInteractive
workversion skewversionriskgap · no spanevent
4 d

Event · step 1 of 5

4 d · the first pinned pair

At 4 d

The report pins fact snapshot f-309 to dimension snapshot d-042 and records both ids with the run. Neither table is read at its latest version; the run names the pair it used.

On screen at this event

  • f-309
  • d-042
  • r-77
  • 0%

Next at 8 d: 8 d · a new fact snapshot lands

Fact
f-309
Dimension
d-042
Null rows
0%
Read event
yes
Step 1/5 · 4 d · the first pinned pair

Figure. Synthetic snapshot lifetimes over 21 days. The fact table advances four times, at days 0, 4, 8 and 15; the dimension advances once, at day 15. The report pins f-310 to d-042 on day 8 and reads nulls until d-043 arrives. Snapshot ids, null rates and dates are illustrative.

What to notice. No deploy happened on day 11. The skew started when the fact snapshot landed and is visible only in the pinned pair: f-310 and d-042 each look current alone. A read event is missing from day 9 to day 14, so lineage can name the tables and cannot name the column change that broke the report. Emit lineage from the runs that actually read, and validate the pair at report build time rather than at the table level.

Text version

Five lanes over 21 days: fact table, dimension, report run, lineage and output. Fact snapshots are f-308 0 to 4, f-309 4 to 8, f-310 8 to 15 in ochre and f-311 15 to 21. The dimension is d-042 0 to 15 in ochre and d-043 15 to 21 in teal. Report runs are r-77 4 to 8, r-78 8 to 15 in red, and r-79 15 to 21. A version skew band spans 8 to 15. Lineage events run 0 to 9, then a red gap 9 to 14 labelled no read, then 14 to 21. Output nulls read 0% 4 to 8, 11% 8 to 15 in red, 0% 15 to 21. Markers read f-310 lands at day 8 and d-043 lands at day 15. Counters: fact f-309, f-310, f-310, f-311, f-311; dimension d-042 three times then d-043; null rows 0, 0, 11, 0, 0 percent; read event yes, yes, missing, emitted, emitted.

  1. Decide the version boundary: a snapshot per load, per day, or per table commit, and keep it consistent across related tables.
  2. Store data immutably and reference versions by identifier rather than overwriting files in place.
  3. Record the job, code revision, parameters and input versions with every output, in a metadata table rather than a comment.
  4. Emit lineage events for reads and writes so the graph is built from what ran, not from documentation.
  5. Expose time travel or snapshot reads to analysts so an old report can be recomputed without a restore.
  6. Set retention per dataset, since immutable history grows without bound and older versions are rarely read.

Trade-offs

ChoiceGood forCostFails when
Full snapshotsSimple restore and auditStorage grows with every runDatasets are large and change little
Table format time travelCheap versions and metadata pruningDependence on one format and catalogConsumers read files directly outside the format
Column-level lineagePrecise impact analysisParser coverage gaps for dynamic SQLJobs build SQL at runtime
Job-level lineage onlyEasy to emit and robustCoarse impact answersA single column change must be traced by hand
Trade-off mapInteractive

Good for

Simple restore and audit

Cost

Storage grows with every run

Fails when

Datasets are large and change little

  1. Datasets are large and change little
  2. Consumers read files directly outside the format
  3. Jobs build SQL at runtime
  4. A single column change must be traced by hand

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full snapshots is good for Simple restore and audit, costs Storage grows with every run, and fails when Datasets are large and change little. Table format time travel is good for Cheap versions and metadata pruning, costs Dependence on one format and catalog, and fails when Consumers read files directly outside the format. Column-level lineage is good for Precise impact analysis, costs Parser coverage gaps for dynamic SQL, and fails when Jobs build SQL at runtime. Job-level lineage only is good for Easy to emit and robust, costs Coarse impact answers, and fails when A single column change must be traced by hand

Failure modes

  • Orphaned output. A table exists with no recorded inputs, so nobody can rebuild it. Debug: query the metadata store for outputs whose input version is null.
  • Version skew across tables. A report joins a new fact snapshot to an old dimension snapshot and numbers move. Debug: compare snapshot ids used by each table in one run.
  • Lineage from documentation. The graph misses a job added last month and impact analysis is wrong. Debug: diff jobs that ran against nodes present in the lineage graph.

In production

Emit lineage from the orchestrator and the query engine, then reconcile the two weekly. Cap snapshot retention by age and size, and rehearse restoring one table to a previous version.

Common misconceptions

People think: Version control for code is enough.

Actually: Code plus inputs plus parameters determine an output; without the data version the same code produces a different number.

People think: Lineage is a diagram for architecture reviews.

Actually: Its operational use is impact analysis and debugging, which requires it to be generated from actual runs.

Interview angle

Separate reproducibility from impact analysis, then name the metadata you would record and where. Mention snapshot boundaries and retention cost. A weak answer says they use Git and object storage versioning without explaining how a past query is recomputed.

Sources