Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 4 of 22

Dataset construction

0 of 22 complete

Why it matters

Fine-tuning data is usually assembled from support tickets, logs and model output, and each source carries its own conventions. Without a schema the same intent appears with three prompt styles and two label formats, and the model learns the noise. Without provenance you cannot remove a bad batch, and without a leakage-aware split your evaluation reports a number that will not survive contact with real traffic.

Mental model

Building the dataset is laying a foundation, not buying furniture: the shape you pour decides what can sit on it later. The analogy breaks because a foundation is inspected before the building goes up, while dataset defects usually surface as a mediocre metric weeks into training.

How it works

Mechanism, step by stepInteractive
  1. 1Write the record schema as fields: system prompt, user turn, optional context, target response, and the mask marking which tokens carry loss.
  2. 2Map each source into that schema explicitly, recording the transformation so a reviewer can reproduce an example from the raw record.
  3. 3Deduplicate near-duplicates with shingling or embedding similarity before splitting, since duplicates inflate both training and evaluation.
  4. 4Split by group such as customer, author or document, so paraphrases of the same case cannot land on both sides.
  5. 5Reserve a slice from a source you did not train on, to measure whether the behaviour transfers beyond the collection channel.
  6. 6Store provenance per record: source, date, annotator and the reason it was included.

Step 1 of 6

Write the record schema as fields: system prompt, user turn, optional context, target response, and the mask marking which tokens carry loss.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write the record schema as fields: system prompt, user turn, optional context, target res… 2. Map each source into that schema explicitly, recording the transformation so a reviewer c… 3. Deduplicate near-duplicates with shingling or embedding similarity before splitting, sinc… 4. Split by group such as customer, author or document, so paraphrases of the same case cann… 5. Reserve a slice from a source you did not train on, to measure whether the behaviour tran… 6. Store provenance per record: source, date, annotator and the reason it was included.

  1. Write the record schema as fields: system prompt, user turn, optional context, target response, and the mask marking which tokens carry loss.
  2. Map each source into that schema explicitly, recording the transformation so a reviewer can reproduce an example from the raw record.
  3. Deduplicate near-duplicates with shingling or embedding similarity before splitting, since duplicates inflate both training and evaluation.
  4. Split by group such as customer, author or document, so paraphrases of the same case cannot land on both sides.
  5. Reserve a slice from a source you did not train on, to measure whether the behaviour transfers beyond the collection channel.
  6. Store provenance per record: source, date, annotator and the reason it was included.

Trade-offs

ChoiceGood forCostFails when
Curated small setFast iteration, high signalCeiling on coverageThe task has long-tail inputs
Mined production logsReal distributionContains failures, personal data and duplicatesYou cannot afford cleaning
Model-generated setCheap scale and coverageAmplifies the generator's mistakesCorrectness is the point
Multi-source mixBroad coverageSchema drift between sourcesSources disagree on the target style
Trade-off mapInteractive

Good for

Fast iteration, high signal

Cost

Ceiling on coverage

Fails when

The task has long-tail inputs

  1. The task has long-tail inputs
  2. You cannot afford cleaning
  3. Correctness is the point
  4. Sources disagree on the target style

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Curated small set is good for Fast iteration, high signal, costs Ceiling on coverage, and fails when The task has long-tail inputs. Mined production logs is good for Real distribution, costs Contains failures, personal data and duplicates, and fails when You cannot afford cleaning. Model-generated set is good for Cheap scale and coverage, costs Amplifies the generator's mistakes, and fails when Correctness is the point. Multi-source mix is good for Broad coverage, costs Schema drift between sources, and fails when Sources disagree on the target style

Failure modes

  • Leakage through paraphrases. Evaluation looks strong and production does not match. Debug: nearest-neighbour each test example against the training set and inspect the top matches.
  • Schema drift between batches. A later batch swaps roles or drops the system prompt and the model learns two contracts. Debug: assert field presence and role order on ingest, and diff the schema hash per batch.
  • Silent label noise. A subset of responses is truncated or off-policy and the model learns to stop early. Debug: sample the shortest and longest responses and read them against the request.

In production

Freeze a dataset version hash and log it with every training run, so a regression can be attributed to data rather than to hyperparameters. Re-run deduplication whenever a new source is added.

Common misconceptions

People think: A random split is fine for text data.

Actually: Random splits leak paraphrases and near-duplicates; group by source or author to measure anything real.

People think: More rows always help.

Actually: Past the point where the behaviour is covered, extra rows add noise and cost unless they add new templates.

Interview angle

A good answer treats the dataset as a product with a schema, provenance and a split policy, and names the leakage failure that a random split hides. It should say how many examples the behaviour needs and why. Weak answers talk about scraping volume and never mention deduplication or how they would debug a suspiciously high score.

Sources