Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 26 of 28

Benchmarks

0 of 28 complete

Why it matters

Comparability is valuable, and a shared task set provides it cheaply. The construct validity problem remains: a benchmark measures what its authors decided to sample, under a harness they wrote, on data that may have entered training. Treating a leaderboard position as evidence of fitness for a specific product is the common error, because the thing being measured and the thing being bought are rarely the same.

Mental model

It is a standardised exam: comparable across schools, silent about the job. The metaphor breaks because the exam leaks and later candidates have seen it, so a rising score can mean a better candidate or a better memory rather than a better system.

How it works

Mechanism, step by stepInteractive
  1. 1Identify the construct the benchmark claims to measure and whether your task shares it.
  2. 2Check the harness: prompting, attempts, tools and scoring rules.
  3. 3Look for contamination evidence and compare the benchmark's date with the model's training window.
  4. 4Read the per-category breakdown and variance rather than the headline rank.
  5. 5Reproduce one or two tasks locally to confirm the reported setup.
  6. 6Combine the result with an internal set drawn from your own traffic.

Step 1 of 6

Identify the construct the benchmark claims to measure and whether your task shares it.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Identify the construct the benchmark claims to measure and whether your task shares it. 2. Check the harness: prompting, attempts, tools and scoring rules. 3. Look for contamination evidence and compare the benchmark's date with the model's trainin… 4. Read the per-category breakdown and variance rather than the headline rank. 5. Reproduce one or two tasks locally to confirm the reported setup. 6. Combine the result with an internal set drawn from your own traffic.

A score gap is a distributionInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. The difference between two systems' benchmark scores when the same task set is resampled, with the 95% interval on that difference and the share of reruns whose sign flips. Normal sampling from a fixed per-item pass rate at three task counts; the spread, not either score, is the claim.

What to notice. At 800 tasks the B − A difference for a 2.3-point lead has a 95% interval of −7.1 to 2.5, and the lower-scoring system still comes out ahead in 17.3% of reruns. At 200 tasks the interval widens to −11.9 to 7.3 and the sign flips in 31.8% of reruns; at 3,600 it narrows to −4.6 to −0.1 and the flip rate is 2.3%. A leaderboard order is only as stable as the task count behind it.

Text version

A distribution of the difference between two benchmark scores on a rerun, from minus 12 to plus 12 points, with a vertical line at no difference. At 800 tasks the curve is centred on a minus 2.3-point gap, peaks at 8.2% of reruns per half-point, and the 95% interval runs from minus 7.1 to plus 2.5: it crosses zero, and the lower-scoring system leads in 17.3% of reruns. A second panel plots that flip share against tasks: 37% at 100 tasks, 17% at 800 and 2.3% at 3,600, with the selected count marked. Three buttons switch the task count to 200 or 3,600, widening or narrowing the curve and moving both the interval and the flip marker; a panel reports the gap, the interval and the flip share.

  1. Identify the construct the benchmark claims to measure and whether your task shares it.
  2. Check the harness: prompting, attempts, tools and scoring rules.
  3. Look for contamination evidence and compare the benchmark's date with the model's training window.
  4. Read the per-category breakdown and variance rather than the headline rank.
  5. Reproduce one or two tasks locally to confirm the reported setup.
  6. Combine the result with an internal set drawn from your own traffic.

Trade-offs

ChoiceGood forCostFails when
Public leaderboardFree comparisonContamination, gamingYou need a decision about your product
Held-out private setLess contaminatedCost and upkeepYou need external comparability
Human preference arenaRealistic promptsStyle and length biasFactual accuracy is the target
Static examStable over timeSaturates and leaksModel is trained on it
Trade-off mapInteractive

Good for

Free comparison

Cost

Contamination, gaming

Fails when

You need a decision about your product

  1. You need a decision about your product
  2. You need external comparability
  3. Factual accuracy is the target
  4. Model is trained on it

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Public leaderboard is good for Free comparison, costs Contamination, gaming, and fails when You need a decision about your product. Held-out private set is good for Less contaminated, costs Cost and upkeep, and fails when You need external comparability. Human preference arena is good for Realistic prompts, costs Style and length bias, and fails when Factual accuracy is the target. Static exam is good for Stable over time, costs Saturates and leaks, and fails when Model is trained on it

Failure modes

  • Contamination. Scores rise because the test data was in training. Debug: check whether the items predate the model and whether paraphrased variants score the same.
  • Harness difference. Two reported scores use different attempt budgets or prompts. Debug: read both harness configs and reproduce one task.
  • Rank instability. Confidence intervals overlap and the ordering is noise. Debug: compute intervals or run repeated samples before claiming a winner.

In production

Track benchmark results with their harness version and date. Treat a new state-of-the-art claim as a hypothesis to test on your own set.

Common misconceptions

People think: Benchmark rankings are objective.

Actually: They are reproducible measurements of a chosen construct, which may not be yours.

People think: A saturated benchmark stops being useful.

Actually: It stops discriminating at the top, while its per-category failures and contamination still inform.

Interview angle

Ask what construct is measured, under which harness and against which contamination risk, then describe an internal set. A strong answer mentions variance. Quoting a leaderboard rank as a purchase decision, with no task distribution, is the weak response.

Sources