Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 5 of 22

Data quality

0 of 22 complete

Why it matters

A fine-tuning run fits whatever is consistent in the data, including the mistakes. If half the answers are terse and half are essays, the model learns to be unpredictable. If a tenth of the labels are wrong, the loss still falls. Quality engineering is about making the target behaviour the path of least resistance, which is mostly removing contradictions rather than adding polish.

Mental model

Data quality is the recipe, not the ingredient list: a dish made from good ingredients in the wrong proportions still fails. The metaphor breaks because a model, unlike a cook, will faithfully reproduce a bad recipe and call the result learned behaviour.

How it works

Mechanism, step by stepInteractive
  1. 1Define the target behaviour in one paragraph with two positive and two negative examples, and use it as the rubric for every reviewer.
  2. 2Deduplicate at the shingle and embedding level, because repeated examples act as an accidental learning-rate multiplier on whatever they contain.
  3. 3Audit labels by sampling stratified by source and by label, since error rates differ by annotator and by class.
  4. 4Normalise style where the variation is accidental, such as date formats and casing, and keep variation where it is the behaviour you want.
  5. 5Score difficulty and keep a deliberate mix, because a set that is uniformly easy teaches nothing about edge cases.
  6. 6Re-audit after every batch addition, since quality decays through accretion more often than through one bad import.

Step 1 of 6

Define the target behaviour in one paragraph with two positive and two negative examples, and use it as the rubric for every reviewer.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Define the target behaviour in one paragraph with two positive and two negative examples,… 2. Deduplicate at the shingle and embedding level, because repeated examples act as an accid… 3. Audit labels by sampling stratified by source and by label, since error rates differ by a… 4. Normalise style where the variation is accidental, such as date formats and casing, and k… 5. Score difficulty and keep a deliberate mix, because a set that is uniformly easy teaches … 6. Re-audit after every batch addition, since quality decays through accretion more often th…

  1. Define the target behaviour in one paragraph with two positive and two negative examples, and use it as the rubric for every reviewer.
  2. Deduplicate at the shingle and embedding level, because repeated examples act as an accidental learning-rate multiplier on whatever they contain.
  3. Audit labels by sampling stratified by source and by label, since error rates differ by annotator and by class.
  4. Normalise style where the variation is accidental, such as date formats and casing, and keep variation where it is the behaviour you want.
  5. Score difficulty and keep a deliberate mix, because a set that is uniformly easy teaches nothing about edge cases.
  6. Re-audit after every batch addition, since quality decays through accretion more often than through one bad import.

Trade-offs

ChoiceGood forCostFails when
Aggressive filteringConsistent, clean signalRemoves rare but valid casesThe long tail is the product
Keep everythingCoverage and real distributionContradictions and noiseThe behaviour must be predictable
Human reviewJudgement on tone and correctnessCost per example and slow cyclesVolume is high and the task is mechanical
Automated scoringCheap and repeatableProxy metrics miss semantic errorsQuality is about nuance
Trade-off mapInteractive

Good for

Consistent, clean signal

Cost

Removes rare but valid cases

Fails when

The long tail is the product

  1. The long tail is the product
  2. The behaviour must be predictable
  3. Volume is high and the task is mechanical
  4. Quality is about nuance

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Aggressive filtering is good for Consistent, clean signal, costs Removes rare but valid cases, and fails when The long tail is the product. Keep everything is good for Coverage and real distribution, costs Contradictions and noise, and fails when The behaviour must be predictable. Human review is good for Judgement on tone and correctness, costs Cost per example and slow cycles, and fails when Volume is high and the task is mechanical. Automated scoring is good for Cheap and repeatable, costs Proxy metrics miss semantic errors, and fails when Quality is about nuance

Failure modes

  • Contradiction pairs. Two examples give opposite answers to the same request and the model oscillates. Debug: cluster by request similarity and compare responses within each cluster.
  • Filter collapse. A quality filter removes every hard example and the tuned model fails on edge cases. Debug: compare the difficulty distribution before and after filtering.
  • Annotator drift. Later batches follow a different rubric and the model learns a moving target. Debug: score a fixed gold set with each annotator batch and plot agreement over time.

In production

Keep a fixed gold set of a few hundred examples and score every new data batch against it before training. Track per-source acceptance rates so a degraded supplier becomes visible.

Common misconceptions

People think: Quality means removing anything unusual.

Actually: Unusual examples often carry the behaviour you need; quality means removing contradictions and errors, not rarity.

People think: A bigger model compensates for noisy labels.

Actually: Capacity fits noise as readily as signal, and the failure appears as confident wrong answers.

Interview angle

Frame quality as agreement with a written target behaviour and describe the audit that proves it, stratified by source and label. Mention deduplication and the risk of over-filtering. An answer that lists cleaning tools without saying what behaviour the data should teach is missing the point.

Sources