Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 2 of 28

Datasets & golden sets

0 of 28 complete

Why it matters

Every eval number is conditional on the cases it was computed over. A 0.92 pass rate on easy, clean, single-turn questions says nothing about the messy, multi-turn, ambiguous requests that reach production. The dataset is where you decide what "quality" means for this product, and it is the part of an eval that is hardest to fix later. A clever scorer on an unrepresentative set produces a precise answer to the wrong question.

The second reason is comparability. If the case set changes between runs, a score difference confounds the system change with the data change. A golden set is frozen precisely so that yesterday's number and today's number refer to the same thing.

Mental model

A golden set is a sampling frame from statistics, with a test suite's discipline. You define the population (the requests the product receives), draw a sample that covers its strata, and record the measurement procedure so someone else can repeat it.

The analogy breaks because the population moves. Software test suites test deterministic code, so a fixed suite stays valid until the code changes. Your users change what they ask, so a fixed set slowly stops representing the population. A golden set needs both stability and periodic refresh, which pull against each other.

How it works

Mechanism, step by stepInteractive
  1. 1Define the unit of evaluation. A single request-response pair, a multi-turn conversation, or a full agent trajectory. The unit determines what a case contains: for an agent, the initial state, the available tools and the expected terminal condition, not just a final string.
  2. 2Sample from reality. Pull from logs, support tickets, sales calls and failed production traces. Redact personal data and get permission before storing it. If the product has no traffic yet, borrow adjacent traffic and label the assumption explicitly.
  3. 3Stratify before you sample. Group by intent, difficulty, tenant or segment, language, input length, and the failure modes you already know. Then sample within each stratum so rare-but-important cases are not drowned out. A set that mirrors traffic proportionally will contain almost no edge cases.
  4. 4Write the grading rule with the case. The rule can be an exact answer, a set of required facts, a schema, a rubric with observable criteria, or a reference answer for a judge. If two people cannot apply it and agree, it is not a rule yet.
  5. 5Label and adjudicate. Have a second person label a subset. Discuss every disagreement and revise the guideline, not just the label. Record the guideline version; annotation drift is a real source of eval drift.
  6. 6Split deliberately. Keep a development set for iteration, a holdout set that is run but never inspected case by case, and a fresh sample drawn on a schedule. Tuning against the holdout quietly converts it into a development set.

Step 1 of 6

Define the unit of evaluation. A single request-response pair, a multi-turn conversation, or a full agent trajectory. The unit determines what a case contains: for an agent, the initial state, the available tools and the expected terminal condition, not just a final string.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Define the unit of evaluation. A single request-response pair, a multi-turn conversation,… 2. Sample from reality. Pull from logs, support tickets, sales calls and failed production t… 3. Stratify before you sample. Group by intent, difficulty, tenant or segment, language, inp… 4. Write the grading rule with the case. The rule can be an exact answer, a set of required … 5. Label and adjudicate. Have a second person label a subset. Discuss every disagreement and… 6. Split deliberately. Keep a development set for iteration, a holdout set that is run but n…

The set ages out of coverageInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. The cumulative age of a 240-case golden set, and the share of cases of each age that still appear in this month's traffic. More than half the set is at least a year old, and most twelve-month-old cases no longer occur; the case count stays fixed while its coverage decays. The counts are illustrative.

What to notice. 52% of the set is at least twelve months old, and only 44% of twelve-month-old cases still appear in this month's traffic. A suite with 240 cases therefore carries a large block that tests a population the product has left, while the frozen count makes the set look unchanged. The freshness line belongs on the dashboard beside the score, because a stable score on ageing cases is not a stable product.

Text version

A stepped survival curve of case age from 0 to 36 months: 100% of cases are at least 0 months old, 70% at least 6 months, 52% at least 12, 38% at least 18 and 22% at least 24. A vertical line marks one year and a marker there reports 52% older. Below, a second stepped curve plots the share of cases of each age still seen in traffic, falling from 78% at 6 months to 44% at 12 and 22% at 18. A slider reads both shares at any age; a panel reports that 52% are older than a year, that 44% of twelve-month-old cases still occur, and that the set holds 240 cases.

  1. Define the unit of evaluation. A single request-response pair, a multi-turn conversation, or a full agent trajectory. The unit determines what a case contains: for an agent, the initial state, the available tools and the expected terminal condition, not just a final string.
  2. Sample from reality. Pull from logs, support tickets, sales calls and failed production traces. Redact personal data and get permission before storing it. If the product has no traffic yet, borrow adjacent traffic and label the assumption explicitly.
  3. Stratify before you sample. Group by intent, difficulty, tenant or segment, language, input length, and the failure modes you already know. Then sample within each stratum so rare-but-important cases are not drowned out. A set that mirrors traffic proportionally will contain almost no edge cases.
  4. Write the grading rule with the case. The rule can be an exact answer, a set of required facts, a schema, a rubric with observable criteria, or a reference answer for a judge. If two people cannot apply it and agree, it is not a rule yet.
  5. Label and adjudicate. Have a second person label a subset. Discuss every disagreement and revise the guideline, not just the label. Record the guideline version; annotation drift is a real source of eval drift.
  6. Split deliberately. Keep a development set for iteration, a holdout set that is run but never inspected case by case, and a fresh sample drawn on a schedule. Tuning against the holdout quietly converts it into a development set.
  7. Version and freeze. Store cases as immutable records with stable IDs. When a case is wrong, deprecate it and add a replacement rather than editing history, so old runs remain interpretable.
  8. Refresh on a cadence. Add cases from recent failures and recent traffic. Retire cases that no longer occur, but keep a small stable core so longitudinal comparisons remain possible.

A case format that survives review

The schema matters less than the fact that every field is mandatory and validated at load time.

import json
from dataclasses import dataclass
 
REQUIRED = {"id", "input", "grading", "source", "created_at"}
 
@dataclass(frozen=True)
class Case:
    id: str
    input: str
    grading: dict          # {"type": "rubric", "criteria": [...]}
    source: str            # "support-ticket", "synthetic", "red-team"
    created_at: str
    tags: tuple[str, ...] = ()
 
def load(path: str) -> list[Case]:
    cases, seen = [], set()
    for line in open(path):
        row = json.loads(line)
        missing = REQUIRED - row.keys()
        if missing:
            raise ValueError(f"{row.get('id')}: missing {sorted(missing)}")
        if extra := row.keys() - REQUIRED - {"tags"}:
            raise ValueError(f"{row['id']}: unknown fields {sorted(extra)}")
        if row["id"] in seen:
            raise ValueError(f"duplicate case id {row['id']}")
        seen.add(row["id"])
        cases.append(Case(**{**row, "tags": tuple(row.get("tags", []))}))
    return cases

Two properties are doing the work: grading is structured rather than prose in a spreadsheet cell, and source records provenance so you can later ask whether synthetic cases behave like real ones.

Trade-offs

SourceGood forCostFails when
Sampled production trafficRepresentativeness, real failure modesPrivacy review, labelling effortTraffic is new, skewed, or dominated by one intent
Expert-authored casesCoverage of required behaviourSlow, reflects known risks onlyThe team has not seen real usage
Synthetic generationVolume, rare combinations, red-team seedsModel bias, unrealistic phrasingUsed without a real-data calibration subset
Public benchmarkModel comparison, sanity checksContamination, different distributionUsed as the release gate for your product
Failure-derived casesRegression protection where it mattersBiased toward past incidentsThe product's usage shifts to new intents
Trade-off mapInteractive

Good for

Representativeness, real failure modes

Cost

Privacy review, labelling effort

Fails when

Traffic is new, skewed, or dominated by one intent

  1. Traffic is new, skewed, or dominated by one intent
  2. The team has not seen real usage
  3. Used without a real-data calibration subset
  4. Used as the release gate for your product
  5. The product's usage shifts to new intents

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Sampled production traffic is good for Representativeness, real failure modes, costs Privacy review, labelling effort, and fails when Traffic is new, skewed, or dominated by one intent. Expert-authored cases is good for Coverage of required behaviour, costs Slow, reflects known risks only, and fails when The team has not seen real usage. Synthetic generation is good for Volume, rare combinations, red-team seeds, costs Model bias, unrealistic phrasing, and fails when Used without a real-data calibration subset. Public benchmark is good for Model comparison, sanity checks, costs Contamination, different distribution, and fails when Used as the release gate for your product. Failure-derived cases is good for Regression protection where it matters, costs Biased toward past incidents, and fails when The product's usage shifts to new intents

Failure modes

Contamination and leakage. Cases end up in a prompt's few-shot examples, in fine-tuning data, or in a public benchmark's training set. Debug by hashing cases against every prompt and training corpus, and by keeping a holdout that no engineer can inspect.

Guideline drift. The rubric says "concise" and two annotators disagree about length. Debug by re-labelling a fixed calibration subset at each labelling round and reporting agreement between annotators, not just between annotator and model.

Label noise. A case's expected answer is wrong, so a correct system is scored as a failure. Debug by auditing every failure in a sampled run by hand; recurring "model errors" are often case errors.

Stale distribution. The set still contains questions about a deprecated plan or a renamed feature. Debug by tracking case age and the share of cases that no longer appear in traffic, and refresh before the score becomes fiction.

In production

Treat the dataset as a versioned artifact next to the prompt. Store a content hash with each eval run so two scores can be compared honestly, and pin the annotation guideline version alongside it. Keep personally identifying data out of the repository: redact at collection, store restricted fields separately, and respect retention limits. For a large set, keep a small fast subset for every commit and a full run nightly.

Ownership matters more than tooling. Someone must be responsible for case quality, or the set accumulates whatever was convenient to add during a debugging session.

Common misconceptions

People think: A bigger eval set is always a better eval set.

Actually: More cases reduce sampling error but do not fix coverage or label quality. A thousand near-duplicate easy cases still cannot tell you how the system handles a refund dispute.

People think: A public benchmark is a ready-made eval set for your product.

Actually: Public benchmarks measure general capability on someone else's tasks. They are useful for choosing a base model and unsuitable for gating your application's behaviour.

People think: Cases should contain the exact expected output string.

Actually: Exact strings punish valid paraphrase. Use them only where the output is genuinely fixed, such as JSON fields or identifiers; otherwise grade properties and required facts.

Interview angle

"The eval set is too small and the score went up two points — do you ship?" is a question about datasets as much as statistics. A strong answer asks what the set represents, whether it changed, how cases were labelled, whether the improvement holds on a holdout and on recent traffic, and what the interval is. See the evaluation bank and coverage.

Sources