Why it matters
Human labels are the reference that judges and metrics are calibrated against, and noisy labels cap the measurable quality of everything downstream. Two annotators who agree ninety percent of the time may still be guessing if one label covers ninety percent of the items. Chance-corrected agreement is the check that separates a real signal from a shared habit, and it is the fastest way to discover that a category is undefined.
Mental model
It is two clocks checked against each other: if they disagree you cannot tell which is right. The metaphor breaks because clocks can be synchronised to the same wrong time, which is shared bias rather than agreement, and no statistic detects it.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- observed agreement
- 79/100 = 79.0%
- chance agreement
- 38.0%
- Cohen's kappa
- 0.661
- weakest label
- partial: 16/25 = 64%
Step 1 of 3
The diagonal
44 relevant, 16 partial and 19 off-topic items were labelled the same way: 79 of 100, or 79%.
Next: Where they disagree
Figure. Two annotators, 100 overlapping items, illustrative counts. Rows are annotator A's label and columns annotator B's; the diagonal is agreement. The marginals in the totals give chance agreement 0.380, so kappa is (0.790 − 0.380) / (1 − 0.380) = 0.661.
What to notice. Raw agreement of 79% looks strong, but 52 of 100 items are relevant, so two annotators guessing from the label distribution would already agree 38% of the time. The weakest label is partial: 16 of 25 agree, with 5 called relevant and 4 called off-topic.
Text version
A three by three table with rows and columns relevant, partial and off-topic. The diagonal counts 44, 16 and 19 agreements, 79 of 100 items. Off the diagonal, A's relevant items become partial 6 times and off-topic twice; A's partial items become relevant 5 times and off-topic 4 times; A's off-topic items become relevant once and partial 3 times. Row totals are 52, 25 and 23; column totals are 50, 25 and 25. Steps highlight the diagonal, the disagreements and the chance correction; the readout gives both labels and the count.
- Choose the statistic from the label type and number of annotators before collecting data.
- Have at least two annotators label an overlapping sample independently.
- Compute agreement overall and per label, with intervals when the sample is small.
- Inspect confusion between specific label pairs rather than only the summary number.
- Revise the rubric where disagreement concentrates, then re-measure on fresh items.
- Store raw labels so agreement can be recomputed after adjudication.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Percent agreement | Trivial to explain | Inflated by class imbalance | One label dominates |
| Cohen's kappa | Chance-corrected and standard | Two annotators, nominal only | Ordinal scale or many annotators |
| Krippendorff's alpha | Flexible across designs | Harder to compute and explain | Team needs a quick number |
| Majority vote without agreement check | Simple labels | Hides systematic disagreement | Labels are subjective |
Good for
Trivial to explain
Cost
Inflated by class imbalance
Fails when
One label dominates
- One label dominates
- Ordinal scale or many annotators
- Team needs a quick number
- Labels are subjective
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Percent agreement is good for Trivial to explain, costs Inflated by class imbalance, and fails when One label dominates. Cohen's kappa is good for Chance-corrected and standard, costs Two annotators, nominal only, and fails when Ordinal scale or many annotators. Krippendorff's alpha is good for Flexible across designs, costs Harder to compute and explain, and fails when Team needs a quick number. Majority vote without agreement check is good for Simple labels, costs Hides systematic disagreement, and fails when Labels are subjective
Failure modes
- Kappa paradox. High percent agreement with low kappa because one class dominates. Debug: print the label distribution and the per-class agreement.
- Non-independent labels. Annotators discussed items before labelling. Debug: check the collection protocol and timestamps for shared sessions.
- Scale mismatch. Ordinal labels treated as nominal distort the statistic. Debug: confirm the statistic matches the measurement scale.
In production
Compute agreement on every new batch and on a fixed calibration set. Alert when a label's agreement drops after a rubric change.
Common misconceptions
People think: High agreement proves the labels are correct.
Actually: It proves consistency, and annotators can share the same misunderstanding.
People think: Low agreement means the annotators are bad.
Actually: It usually means the category is undefined or the rubric leaves room for interpretation.
Interview angle
Pick the statistic from the design, report per-label agreement and treat low agreement as a rubric problem first. A strong answer mentions the class-imbalance paradox. Quoting percent agreement alone, with no view of which labels disagreed, is the weak response.
Sources
- Artstein and Poesio, Inter-Coder Agreement for Computational Linguistics — chance-corrected agreement for annotation studies.
- scikit-learn, cohen_kappa_score — the standard two-annotator implementation and its caveats.
- Krippendorff's alpha — agreement across multiple annotators and scales.