Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 25 of 27

Dataset curation

0 of 27 complete

Why it matters

A model can only learn what the data contains, and it learns the data's flaws as readily as its signal. Curation is where you decide the effective training distribution: which sources, which languages, which duplicates to collapse, which labels to trust. Because these decisions are upstream of every experiment, a curation mistake is more expensive than a modelling mistake and much harder to detect from loss alone.

Mental model

Curation is editing an anthology: you choose what goes in, remove repeated pages, and note where each piece came from. The metaphor breaks because the reader is also changed by the book — the model's behaviour is a direct function of the selection, including the parts you left out.

How it works

A duplicate is a distance, not a labelInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Eight synthetic documents placed by two coordinates around one seed document. Scores are cosine similarity computed from those coordinates; the dot-product button removes the length normalisation. The two copies are exact scalar multiples of the seed, so cosine puts both at 1.00 whatever their length. The axes are dimensionless, the layout is hand-placed, and no dedup run produced it. Filtering, label quality and provenance are not in the figure.

What to notice. Both copies of the seed score 1.00 under cosine although one is three times the seed's length; a cut at 0.95 removes both and keeps the paraphrase at 0.89. Under the dot product the long copy still scores 0.41, but the short copy falls to 0.07 — below both paraphrases at 0.20 and 0.17 — so a threshold applied to raw dot products would keep a duplicate and drop a paraphrase. Cosine is the default here for that reason: the cut is a claim about content, not length. A cut above 0.89 removes that paraphrase too, and a cut above 0.82 removes the second. The figure cannot choose it for you.

Text version

A scatter of eight synthetic documents around a seed document in a hand-placed 2-D layout. The default frame uses cosine similarity and k = 3: the long copy and the short copy both score 1.00 and are returned first and second, and a paraphrase scores 0.89 and is third; a second paraphrase at 0.82 is outside k. A callout on the long copy notes that it is three times the seed's length. The metric control switches to the dot product, where the long copy leads at 0.41 but the short copy drops to 0.07, below both paraphrases at 0.20 and 0.17. A slider sets k from one to six.

  1. Define the target distribution from the deployment task, including languages, formats and edge cases.
  2. Collect from sources with compatible licences and record provenance for every shard.
  3. Deduplicate at document and near-duplicate level using hashes and similarity, and report the removal rate.
  4. Filter for quality with rules or a classifier, then inspect a sample of both kept and removed items.
  5. Label with written guidelines, measure inter-annotator agreement, and adjudicate disagreements.
  6. Publish a datasheet covering motivation, composition, collection process, gaps and intended use.

Trade-offs

ChoiceGood forCostFails when
Aggressive filteringHigher average quality, less noiseRemoves rare patterns and can bias the distributionThe task needs long-tail or informal text
Broad collectionCoverage of rare casesNoise, duplication and licence riskQuality is the binding constraint
Automatic labellingScale at low costSystematic label bias, silent errorsThe label needs judgement
Human labellingAccurate and auditableCost and slow iterationVolume is large and time is short
Trade-off mapInteractive

Good for

Higher average quality, less noise

Cost

Removes rare patterns and can bias the distribution

Fails when

The task needs long-tail or informal text

  1. The task needs long-tail or informal text
  2. Quality is the binding constraint
  3. The label needs judgement
  4. Volume is large and time is short

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Aggressive filtering is good for Higher average quality, less noise, costs Removes rare patterns and can bias the distribution, and fails when The task needs long-tail or informal text. Broad collection is good for Coverage of rare cases, costs Noise, duplication and licence risk, and fails when Quality is the binding constraint. Automatic labelling is good for Scale at low cost, costs Systematic label bias, silent errors, and fails when The label needs judgement. Human labelling is good for Accurate and auditable, costs Cost and slow iteration, and fails when Volume is large and time is short

Failure modes

  • Duplicate-driven memorisation. Repeated documents make the model recite training text and distort evaluation. Debug: measure near-duplicate rate within the corpus and between train and evaluation sets.
  • Filter bias. A quality classifier removes a dialect or format and the model underperforms there. Debug: compare kept and removed samples by source and language.
  • Label guideline drift. Annotators apply different rules over time and the label means different things in different shards. Debug: re-label a fixed audit set at intervals and track agreement against the original.

In production

Track the composition of every training corpus by source, language and filter decision, and re-run the audit set on each new shard. Keep a small human-reviewed sample as a canary for label drift when annotation is outsourced or automated.

Common misconceptions

People think: More data always improves a model.

Actually: Duplicated or low-quality data can dominate the loss and teach the wrong distribution; curation often beats volume.

People think: Dataset work is a one-time preprocessing step.

Actually: Distributions drift, sources change and labels age, so curation is a recurring pipeline with its own monitoring.

Interview angle

A strong answer treats the dataset as a designed distribution: sources, deduplication, filters, labels and documentation. It should describe how to audit each step. Weak answers say 'collect more data and clean it' without naming what cleaning removes or how the removal changes the target distribution.

Sources