Skip to content
AI Engineer Guide.

concept · data · intermediate · concept 8 of 17

Data quality

0 of 17 complete

Why it matters

Pipelines fail quietly. A source renames a status value, a join drops unmatched rows, a job runs twice, and the dashboard still renders a number. Because the output looks plausible, nobody investigates until a decision has been made on it. Data quality work makes those assumptions explicit and turns them into assertions that run with the data, so a broken assumption stops the pipeline instead of reaching a reader.

Mental model

Tests are the seatbelt of a pipeline: they do not prevent the crash, they limit the damage and make it visible. The metaphor breaks because a seatbelt is passive, while data tests need thresholds that someone chose, and a badly chosen threshold either never fires or fires constantly.

How it works

Checks on one daily tableInteractive

Synthetic quality report · orders_daily

Two tables · five checks · one batch

bounding boxes

Step 1/4 · Encode the decision

Step 1Encode the decision

The table looks plausible, but each check exists because a decision depends on it: customer roll-ups need a customer id, revenue needs unique orders, and today's dashboard needs today's events.

DATA PLATFORM · QUALITY REPORT · 2026-11-17 21:04Z

, running header, region 1. Select to inspect.

orders_daily — run 8f21

, title, region 2. Select to inspect.
Rows48,201
, form field Rows, region 3. Select to inspect.
Batch2026-11-17
, form field Batch, region 4. Select to inspect.
Max event_ts11-16 07:04
✕, form field Max event_ts, region 5, check failed. Select to inspect.
Duplicate rows2
✕, form field Duplicate rows, region 6, check failed. Select to inspect.
SAMPLE · 4 OF 48,201 ROWS
order_idcustomer_idtotalevent_ts
A-1042C-8842.0011-16 07:04
A-1043✕—18.5011-16 02:58
✕A-1043C-9118.5011-16 02:58
A-1044C-127.25!11-14 22:40
, table SAMPLE · 4 OF 48,201 ROWS, region 7. Select to inspect.
CHECKS
CheckObservedState
customer_id not null3 missing✕fail
order_id unique1 id on 2 rows✕fail
max(event_ts) ≤ 24 h38 h old✕fail
rows vs 7-day median48,201 / 47,900✓pass
status in 4 values0 invalid✓pass
, table CHECKS, region 8. Select to inspect.
Previous run2026-11-16 · 47,900 rows · pass
, form field Previous run, region 11. Select to inspect.
9 regions

CHECKS· region 8· table

Five checks, three failures. The verdict is written as text in the state column, so colour is not the only channel.

CheckObservedState
customer_id not null3 missingfail
order_id unique1 id on 2 rowsfail
max(event_ts) ≤ 24 h38 h oldfail
rows vs 7-day median48,201 / 47,900pass
status in 4 values0 invalidpass

The text version summarises this figure; select a region for its full text.

Markscheck failedwarningpass

Figure. Five checks on one daily table. The sample rows show three distinct defects — a missing customer id, a duplicated order id and a stale maximum timestamp — and the checks table counts them. Colour is backed by a state label and a count.

What to notice. Synthetic dataset. Row counts, thresholds and timestamps are authored to make the failures visible. The figure shows what each check protects and why a stale batch stops the pipeline; it is not a measured defect rate.

Text version

A synthetic data-quality report for the daily orders table. Four metric fields show 48,201 rows, the batch date, a maximum event timestamp 38 hours old and two duplicate rows; the timestamp and duplicate-row fields carry fail states. A sample table of four rows highlights a missing customer id, a duplicated order id and a late event timestamp with text marks and tinted cells. A checks table lists five checks with observed values and pass or fail states. A quarantine block records three rows with the reason null_customer_id, a pipeline block says stopped because freshness is a stop-severity check, and a previous-run field gives the comparison row count. The steps move from the sample rows and the checks to the quarantine record and the stop, with the verdict written in the state column as text.

  1. List the decisions the dataset supports and the property each one depends on, such as every order having a customer or every daily row being present.
  2. Encode each property as a test with an owner and a severity, distinguishing a warning from a stop.
  3. Run column-level checks on ingestion, uniqueness and referential checks after joins, and freshness checks on the final table.
  4. Route failing rows to a quarantine table with the failure reason and the source batch id.
  5. Track test results over time so a slow drift in null rate is visible before it becomes a hard failure.
  6. Review tests when the schema or source changes, because a test that encodes an old assumption blocks a correct change.

Trade-offs

ChoiceGood forCostFails when
Hard fail on any testBad data never reaches consumersFragile pipelines and frequent manual unblocksSources are noisy and mostly correct
Warn and continuePipelines keep runningDefects reach dashboards quietlyNobody reads the warning channel
Row-level quarantineGood rows still flow and defects are countedMore storage and a repair processThe quarantine has no owner and grows forever
Distribution checksCatches drift without a fixed schemaFalse positives on seasonal dataTraffic genuinely changes shape and nobody tunes thresholds
Trade-off mapInteractive

Good for

Bad data never reaches consumers

Cost

Fragile pipelines and frequent manual unblocks

Fails when

Sources are noisy and mostly correct

  1. Sources are noisy and mostly correct
  2. Nobody reads the warning channel
  3. The quarantine has no owner and grows forever
  4. Traffic genuinely changes shape and nobody tunes thresholds

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Hard fail on any test is good for Bad data never reaches consumers, costs Fragile pipelines and frequent manual unblocks, and fails when Sources are noisy and mostly correct. Warn and continue is good for Pipelines keep running, costs Defects reach dashboards quietly, and fails when Nobody reads the warning channel. Row-level quarantine is good for Good rows still flow and defects are counted, costs More storage and a repair process, and fails when The quarantine has no owner and grows forever. Distribution checks is good for Catches drift without a fixed schema, costs False positives on seasonal data, and fails when Traffic genuinely changes shape and nobody tunes thresholds

Failure modes

  • Null rate creep. A new source field arrives empty and downstream averages shift. Debug: compare the null fraction per column against the previous seven runs.
  • Duplicate delivery. A retried batch loads twice and counts rise. Debug: count distinct source ids per batch and compare with expected row counts.
  • Stale table. The job succeeds but writes nothing and the dashboard shows yesterday. Debug: assert maximum event timestamp is within the expected window.

In production

Alert on freshness and volume before value-level checks, since a missing load is the most common incident. Keep a quarantine table with counts per reason and a weekly review, or the defects accumulate unread.

Common misconceptions

People think: Data quality is a dashboard.

Actually: A dashboard shows defects after they land; tests stop them at the boundary and record why.

People think: More tests always mean better quality.

Actually: Untuned tests create alert fatigue, and a muted channel is worse than no channel because it looks like coverage.

Interview angle

Frame quality as assertions tied to decisions, with severity, ownership and a quarantine path. Name freshness and volume as the checks that catch most incidents. Weak answers list a tool or say they would add more tests without saying what failure each test prevents.

Sources