Why it matters
Buyers need a comparison, and a shared task set with hidden tests is the cheapest one available. The trouble is that a benchmark is a curated sample: often self-contained, single-language and drawn from repositories with good test coverage. Your repository has its own build system, flaky tests and conventions. A score tells you how an agent did on someone else's sample under someone else's harness, which is a starting point rather than a prediction.
Mental model
It is a driving test on a closed course. The metaphor breaks because your commute has cyclists, roadworks and a car that behaves differently from the test vehicle, and the test never reports how many attempts the driver needed or who else was in the car.
How it works
Step 1 of 8
One number, four omissions
The score says how this harness did on this sample. It does not say whether the tests were visible, what was sampled, what a solve cost, or where it failed.
root · current
Reported pass rate
The row states the share of sampled tasks the harness resolved. Everything the number depends on sits outside it, so the figure splits the claim rather than the score.
Root of the hierarchy
Figure. A reported pass rate decomposed into the four questions it leaves open — the harness, the sample, the price per solve and the failure distribution — with the held-out set drawn as the replacement rather than a component. The crossed branch is the leaderboard rank used as a prediction about your repository, which is the one claim the number cannot support.
What to notice. No score is drawn, because the page publishes none; the harness fields and the held-out set are an authored example. The tree is a decomposition of the claim, not a ranking of benchmarks: four children are questions the number leaves open, the fifth is the set that replaces it, and the crossed branch is a claim the number cannot carry. Omitted: the local re-run of one sampled task and individual benchmark names and dates; the re-run confirms the reported setup, and the names change faster than the page.
Text version
A reported pass rate drawn as a hierarchy. The root is one number scored by hidden tests. Four branches are the questions it leaves open. The harness asks how it was run, with an attempt budget of one try or several and whether the tests were visible to the model. The task source asks what was sampled: familiar, well-tested, self-contained repositories, and whether the tasks were published before the model's cutoff, which is the contamination check. Cost and time asks what a solve cost, in tokens and wall clock per solved task. The failure distribution asks where it fails, by repository, language and task type, including a category that always fails and disappears into the mean. A fifth branch is labelled as replacing the score rather than feeding it: your own held-out set, built from closed issues with the fixing commit as the oracle. The leaderboard rank as a prediction is drawn crossed and ruled out at the last step.
- Read the harness: which tests run, how many attempts are allowed and whether the model can see the tests.
- Check the task source and publication date against the model's training window for contamination risk.
- Compare cost and wall-clock time per solved task alongside the pass rate.
- Re-run a sample of tasks in your own sandbox to confirm the reported setup.
- Build a held-out set from your own closed issues, using the fixing commit as the oracle.
- Report the distribution and failure categories rather than one aggregate number.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Public benchmark | Comparable and cheap | Contamination, distribution mismatch | Your repo uses another language or framework |
| Internal issue set | Representative | Small, needs curation | Team has no closed issues with tests |
| Hidden tests only | Objective pass or fail | Misses maintainability | Task has several valid solutions |
| Human-graded benchmark | Judges intent | Expensive and subjective | Scale matters |
Good for
Comparable and cheap
Cost
Contamination, distribution mismatch
Fails when
Your repo uses another language or framework
- Your repo uses another language or framework
- Team has no closed issues with tests
- Task has several valid solutions
- Scale matters
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Public benchmark is good for Comparable and cheap, costs Contamination, distribution mismatch, and fails when Your repo uses another language or framework. Internal issue set is good for Representative, costs Small, needs curation, and fails when Team has no closed issues with tests. Hidden tests only is good for Objective pass or fail, costs Misses maintainability, and fails when Task has several valid solutions. Human-graded benchmark is good for Judges intent, costs Expensive and subjective, and fails when Scale matters
Failure modes
- Test leakage. The model saw the tests or the fix during training and scores high without generalising. Debug: check whether the task's tests were public before the model's cutoff.
- Harness mismatch. The reported score used a different attempt budget than the one you assume. Debug: read the run configuration and reproduce one task end to end.
- Aggregate illusion. A high average hides a category where the agent always fails. Debug: break the score down by repository, language and task type.
In production
Re-run your internal set on every model or harness change and store per-task results. Track cost per solved task, not only the pass rate.
Common misconceptions
People think: A benchmark score predicts performance on your codebase.
Actually: It predicts performance on that benchmark's task distribution and harness.
People think: A higher pass rate means a better tool.
Actually: Attempt budget, cost and review burden decide usefulness more often than the headline number.
Interview angle
Read the harness before the score and check contamination before comparing. A strong answer separates cost from accuracy and describes an internal set built from real issues. Quoting a leaderboard rank as a prediction, with no attempt budget or language mix, is the weak response.
Sources
- Jimenez et al., SWE-bench — real GitHub issues scored by hidden tests.
- SWE-bench — the leaderboard and its evaluation harness.
- Liu et al., RepoBench — repository-level tasks and cross-file context.