Why it matters
Pipelines fail quietly. A source renames a status value, a join drops unmatched rows, a job runs twice, and the dashboard still renders a number. Because the output looks plausible, nobody investigates until a decision has been made on it. Data quality work makes those assumptions explicit and turns them into assertions that run with the data, so a broken assumption stops the pipeline instead of reaching a reader.
Mental model
Tests are the seatbelt of a pipeline: they do not prevent the crash, they limit the damage and make it visible. The metaphor breaks because a seatbelt is passive, while data tests need thresholds that someone chose, and a badly chosen threshold either never fires or fires constantly.
How it works
Synthetic quality report · orders_daily
Two tables · five checks · one batch
bounding boxes
Step 1Encode the decision
The table looks plausible, but each check exists because a decision depends on it: customer roll-ups need a customer id, revenue needs unique orders, and today's dashboard needs today's events.
DATA PLATFORM · QUALITY REPORT · 2026-11-17 21:04Z
orders_daily — run 8f21
| order_id | customer_id | total | event_ts |
|---|---|---|---|
| A-1042 | C-88 | 42.00 | 11-16 07:04 |
| A-1043 | ✕— | 18.50 | 11-16 02:58 |
| ✕A-1043 | C-91 | 18.50 | 11-16 02:58 |
| A-1044 | C-12 | 7.25 | !11-14 22:40 |
| Check | Observed | State |
|---|---|---|
| customer_id not null | 3 missing | ✕fail |
| order_id unique | 1 id on 2 rows | ✕fail |
| max(event_ts) ≤ 24 h | 38 h old | ✕fail |
| rows vs 7-day median | 48,201 / 47,900 | ✓pass |
| status in 4 values | 0 invalid | ✓pass |
CHECKS· region 8· table
Five checks, three failures. The verdict is written as text in the state column, so colour is not the only channel.
| Check | Observed | State |
| customer_id not null | 3 missing | fail |
| order_id unique | 1 id on 2 rows | fail |
| max(event_ts) ≤ 24 h | 38 h old | fail |
| rows vs 7-day median | 48,201 / 47,900 | pass |
| status in 4 values | 0 invalid | pass |
The text version summarises this figure; select a region for its full text.
Figure. Five checks on one daily table. The sample rows show three distinct defects — a missing customer id, a duplicated order id and a stale maximum timestamp — and the checks table counts them. Colour is backed by a state label and a count.
What to notice. Synthetic dataset. Row counts, thresholds and timestamps are authored to make the failures visible. The figure shows what each check protects and why a stale batch stops the pipeline; it is not a measured defect rate.
Text version
A synthetic data-quality report for the daily orders table. Four metric fields show 48,201 rows, the batch date, a maximum event timestamp 38 hours old and two duplicate rows; the timestamp and duplicate-row fields carry fail states. A sample table of four rows highlights a missing customer id, a duplicated order id and a late event timestamp with text marks and tinted cells. A checks table lists five checks with observed values and pass or fail states. A quarantine block records three rows with the reason null_customer_id, a pipeline block says stopped because freshness is a stop-severity check, and a previous-run field gives the comparison row count. The steps move from the sample rows and the checks to the quarantine record and the stop, with the verdict written in the state column as text.
- List the decisions the dataset supports and the property each one depends on, such as every order having a customer or every daily row being present.
- Encode each property as a test with an owner and a severity, distinguishing a warning from a stop.
- Run column-level checks on ingestion, uniqueness and referential checks after joins, and freshness checks on the final table.
- Route failing rows to a quarantine table with the failure reason and the source batch id.
- Track test results over time so a slow drift in null rate is visible before it becomes a hard failure.
- Review tests when the schema or source changes, because a test that encodes an old assumption blocks a correct change.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Hard fail on any test | Bad data never reaches consumers | Fragile pipelines and frequent manual unblocks | Sources are noisy and mostly correct |
| Warn and continue | Pipelines keep running | Defects reach dashboards quietly | Nobody reads the warning channel |
| Row-level quarantine | Good rows still flow and defects are counted | More storage and a repair process | The quarantine has no owner and grows forever |
| Distribution checks | Catches drift without a fixed schema | False positives on seasonal data | Traffic genuinely changes shape and nobody tunes thresholds |
Good for
Bad data never reaches consumers
Cost
Fragile pipelines and frequent manual unblocks
Fails when
Sources are noisy and mostly correct
- Sources are noisy and mostly correct
- Nobody reads the warning channel
- The quarantine has no owner and grows forever
- Traffic genuinely changes shape and nobody tunes thresholds
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Hard fail on any test is good for Bad data never reaches consumers, costs Fragile pipelines and frequent manual unblocks, and fails when Sources are noisy and mostly correct. Warn and continue is good for Pipelines keep running, costs Defects reach dashboards quietly, and fails when Nobody reads the warning channel. Row-level quarantine is good for Good rows still flow and defects are counted, costs More storage and a repair process, and fails when The quarantine has no owner and grows forever. Distribution checks is good for Catches drift without a fixed schema, costs False positives on seasonal data, and fails when Traffic genuinely changes shape and nobody tunes thresholds
Failure modes
- Null rate creep. A new source field arrives empty and downstream averages shift. Debug: compare the null fraction per column against the previous seven runs.
- Duplicate delivery. A retried batch loads twice and counts rise. Debug: count distinct source ids per batch and compare with expected row counts.
- Stale table. The job succeeds but writes nothing and the dashboard shows yesterday. Debug: assert maximum event timestamp is within the expected window.
In production
Alert on freshness and volume before value-level checks, since a missing load is the most common incident. Keep a quarantine table with counts per reason and a weekly review, or the defects accumulate unread.
Common misconceptions
People think: Data quality is a dashboard.
Actually: A dashboard shows defects after they land; tests stop them at the boundary and record why.
People think: More tests always mean better quality.
Actually: Untuned tests create alert fatigue, and a muted channel is worse than no channel because it looks like coverage.
Interview angle
Frame quality as assertions tied to decisions, with severity, ownership and a quarantine path. Name freshness and volume as the checks that catch most incidents. Weak answers list a tool or say they would add more tests without saying what failure each test prevents.
Sources
- dbt documentation, data tests — uniqueness, not-null and relationship tests on models.
- Great Expectations documentation — expectations, validation and data docs.
- JSON Schema — contract-level validation for incoming records.