Why it matters
A model can only learn what the data contains, and it learns the data's flaws as readily as its signal. Curation is where you decide the effective training distribution: which sources, which languages, which duplicates to collapse, which labels to trust. Because these decisions are upstream of every experiment, a curation mistake is more expensive than a modelling mistake and much harder to detect from loss alone.
Mental model
Curation is editing an anthology: you choose what goes in, remove repeated pages, and note where each piece came from. The metaphor breaks because the reader is also changed by the book — the model's behaviour is a direct function of the selection, including the parts you left out.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Eight synthetic documents placed by two coordinates around one seed document. Scores are cosine similarity computed from those coordinates; the dot-product button removes the length normalisation. The two copies are exact scalar multiples of the seed, so cosine puts both at 1.00 whatever their length. The axes are dimensionless, the layout is hand-placed, and no dedup run produced it. Filtering, label quality and provenance are not in the figure.
What to notice. Both copies of the seed score 1.00 under cosine although one is three times the seed's length; a cut at 0.95 removes both and keeps the paraphrase at 0.89. Under the dot product the long copy still scores 0.41, but the short copy falls to 0.07 — below both paraphrases at 0.20 and 0.17 — so a threshold applied to raw dot products would keep a duplicate and drop a paraphrase. Cosine is the default here for that reason: the cut is a claim about content, not length. A cut above 0.89 removes that paraphrase too, and a cut above 0.82 removes the second. The figure cannot choose it for you.
Text version
A scatter of eight synthetic documents around a seed document in a hand-placed 2-D layout. The default frame uses cosine similarity and k = 3: the long copy and the short copy both score 1.00 and are returned first and second, and a paraphrase scores 0.89 and is third; a second paraphrase at 0.82 is outside k. A callout on the long copy notes that it is three times the seed's length. The metric control switches to the dot product, where the long copy leads at 0.41 but the short copy drops to 0.07, below both paraphrases at 0.20 and 0.17. A slider sets k from one to six.
- Define the target distribution from the deployment task, including languages, formats and edge cases.
- Collect from sources with compatible licences and record provenance for every shard.
- Deduplicate at document and near-duplicate level using hashes and similarity, and report the removal rate.
- Filter for quality with rules or a classifier, then inspect a sample of both kept and removed items.
- Label with written guidelines, measure inter-annotator agreement, and adjudicate disagreements.
- Publish a datasheet covering motivation, composition, collection process, gaps and intended use.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Aggressive filtering | Higher average quality, less noise | Removes rare patterns and can bias the distribution | The task needs long-tail or informal text |
| Broad collection | Coverage of rare cases | Noise, duplication and licence risk | Quality is the binding constraint |
| Automatic labelling | Scale at low cost | Systematic label bias, silent errors | The label needs judgement |
| Human labelling | Accurate and auditable | Cost and slow iteration | Volume is large and time is short |
Good for
Higher average quality, less noise
Cost
Removes rare patterns and can bias the distribution
Fails when
The task needs long-tail or informal text
- The task needs long-tail or informal text
- Quality is the binding constraint
- The label needs judgement
- Volume is large and time is short
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Aggressive filtering is good for Higher average quality, less noise, costs Removes rare patterns and can bias the distribution, and fails when The task needs long-tail or informal text. Broad collection is good for Coverage of rare cases, costs Noise, duplication and licence risk, and fails when Quality is the binding constraint. Automatic labelling is good for Scale at low cost, costs Systematic label bias, silent errors, and fails when The label needs judgement. Human labelling is good for Accurate and auditable, costs Cost and slow iteration, and fails when Volume is large and time is short
Failure modes
- Duplicate-driven memorisation. Repeated documents make the model recite training text and distort evaluation. Debug: measure near-duplicate rate within the corpus and between train and evaluation sets.
- Filter bias. A quality classifier removes a dialect or format and the model underperforms there. Debug: compare kept and removed samples by source and language.
- Label guideline drift. Annotators apply different rules over time and the label means different things in different shards. Debug: re-label a fixed audit set at intervals and track agreement against the original.
In production
Track the composition of every training corpus by source, language and filter decision, and re-run the audit set on each new shard. Keep a small human-reviewed sample as a canary for label drift when annotation is outsourced or automated.
Common misconceptions
People think: More data always improves a model.
Actually: Duplicated or low-quality data can dominate the loss and teach the wrong distribution; curation often beats volume.
People think: Dataset work is a one-time preprocessing step.
Actually: Distributions drift, sources change and labels age, so curation is a recurring pipeline with its own monitoring.
Interview angle
A strong answer treats the dataset as a designed distribution: sources, deduplication, filters, labels and documentation. It should describe how to audit each step. Weak answers say 'collect more data and clean it' without naming what cleaning removes or how the removal changes the target distribution.
Sources
- Gebru et al., Datasheets for Datasets — the documentation dimensions a dataset should record.
- Mitchell et al., Model Cards for Model Reporting — intended use, evaluation conditions and limitations.
- Lee et al., Deduplicating Training Data Makes Language Models Better — the effect of duplication on memorisation and evaluation.