Why it matters
A model or a report is only trustworthy if you can say what it was computed from. In practice data changes underneath a query: a source corrects a value, a job is rerun with new code, and last month's figure can no longer be produced. Versioning and lineage exist to answer two operational questions, what exactly produced this number and what depends on this input, before either question becomes an audit.
Mental model
A versioned dataset is a commit history for data; lineage is the dependency graph a build system uses. The metaphor breaks because data commits are not reviewed the way code is, so an unreviewed snapshot can still be a faithful record of a wrong load.
How it works
Event · step 1 of 5
4 d · the first pinned pair
At 4 d
The report pins fact snapshot f-309 to dimension snapshot d-042 and records both ids with the run. Neither table is read at its latest version; the run names the pair it used.
On screen at this event
- f-309
- d-042
- r-77
- 0%
Next at 8 d: 8 d · a new fact snapshot lands
- Fact
- f-309
- Dimension
- d-042
- Null rows
- 0%
- Read event
- yes
Figure. Synthetic snapshot lifetimes over 21 days. The fact table advances four times, at days 0, 4, 8 and 15; the dimension advances once, at day 15. The report pins f-310 to d-042 on day 8 and reads nulls until d-043 arrives. Snapshot ids, null rates and dates are illustrative.
What to notice. No deploy happened on day 11. The skew started when the fact snapshot landed and is visible only in the pinned pair: f-310 and d-042 each look current alone. A read event is missing from day 9 to day 14, so lineage can name the tables and cannot name the column change that broke the report. Emit lineage from the runs that actually read, and validate the pair at report build time rather than at the table level.
Text version
Five lanes over 21 days: fact table, dimension, report run, lineage and output. Fact snapshots are f-308 0 to 4, f-309 4 to 8, f-310 8 to 15 in ochre and f-311 15 to 21. The dimension is d-042 0 to 15 in ochre and d-043 15 to 21 in teal. Report runs are r-77 4 to 8, r-78 8 to 15 in red, and r-79 15 to 21. A version skew band spans 8 to 15. Lineage events run 0 to 9, then a red gap 9 to 14 labelled no read, then 14 to 21. Output nulls read 0% 4 to 8, 11% 8 to 15 in red, 0% 15 to 21. Markers read f-310 lands at day 8 and d-043 lands at day 15. Counters: fact f-309, f-310, f-310, f-311, f-311; dimension d-042 three times then d-043; null rows 0, 0, 11, 0, 0 percent; read event yes, yes, missing, emitted, emitted.
- Decide the version boundary: a snapshot per load, per day, or per table commit, and keep it consistent across related tables.
- Store data immutably and reference versions by identifier rather than overwriting files in place.
- Record the job, code revision, parameters and input versions with every output, in a metadata table rather than a comment.
- Emit lineage events for reads and writes so the graph is built from what ran, not from documentation.
- Expose time travel or snapshot reads to analysts so an old report can be recomputed without a restore.
- Set retention per dataset, since immutable history grows without bound and older versions are rarely read.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Full snapshots | Simple restore and audit | Storage grows with every run | Datasets are large and change little |
| Table format time travel | Cheap versions and metadata pruning | Dependence on one format and catalog | Consumers read files directly outside the format |
| Column-level lineage | Precise impact analysis | Parser coverage gaps for dynamic SQL | Jobs build SQL at runtime |
| Job-level lineage only | Easy to emit and robust | Coarse impact answers | A single column change must be traced by hand |
Good for
Simple restore and audit
Cost
Storage grows with every run
Fails when
Datasets are large and change little
- Datasets are large and change little
- Consumers read files directly outside the format
- Jobs build SQL at runtime
- A single column change must be traced by hand
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full snapshots is good for Simple restore and audit, costs Storage grows with every run, and fails when Datasets are large and change little. Table format time travel is good for Cheap versions and metadata pruning, costs Dependence on one format and catalog, and fails when Consumers read files directly outside the format. Column-level lineage is good for Precise impact analysis, costs Parser coverage gaps for dynamic SQL, and fails when Jobs build SQL at runtime. Job-level lineage only is good for Easy to emit and robust, costs Coarse impact answers, and fails when A single column change must be traced by hand
Failure modes
- Orphaned output. A table exists with no recorded inputs, so nobody can rebuild it. Debug: query the metadata store for outputs whose input version is null.
- Version skew across tables. A report joins a new fact snapshot to an old dimension snapshot and numbers move. Debug: compare snapshot ids used by each table in one run.
- Lineage from documentation. The graph misses a job added last month and impact analysis is wrong. Debug: diff jobs that ran against nodes present in the lineage graph.
In production
Emit lineage from the orchestrator and the query engine, then reconcile the two weekly. Cap snapshot retention by age and size, and rehearse restoring one table to a previous version.
Common misconceptions
People think: Version control for code is enough.
Actually: Code plus inputs plus parameters determine an output; without the data version the same code produces a different number.
People think: Lineage is a diagram for architecture reviews.
Actually: Its operational use is impact analysis and debugging, which requires it to be generated from actual runs.
Interview angle
Separate reproducibility from impact analysis, then name the metadata you would record and where. Mention snapshot boundaries and retention cost. A weak answer says they use Git and object storage versioning without explaining how a past query is recomputed.
Sources
- Apache Iceberg specification — snapshots, manifests and time travel over table files.
- OpenLineage — an open event schema for job and dataset lineage.
- dbt documentation, snapshots — capturing row history with validity windows.