Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 8 of 28

Inter-annotator agreement

0 of 28 complete

Why it matters

Human labels are the reference that judges and metrics are calibrated against, and noisy labels cap the measurable quality of everything downstream. Two annotators who agree ninety percent of the time may still be guessing if one label covers ninety percent of the items. Chance-corrected agreement is the check that separates a real signal from a shared habit, and it is the fastest way to discover that a category is undefined.

Mental model

It is two clocks checked against each other: if they disagree you cannot tell which is right. The metaphor breaks because clocks can be synchronised to the same wrong time, which is shared bias rather than agreement, and no statistic detects it.

How it works

Two annotators · 100 overlapping itemsInteractive

hover or focus a cell

044
annotator A ↓annotator B →
relevantpartialoff-topicitems
relevant52
partial25
off-topic23
total502525

Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.

observed agreement
79/100 = 79.0%
chance agreement
38.0%
Cohen's kappa
0.661
weakest label
partial: 16/25 = 64%

Step 1 of 3

The diagonal

44 relevant, 16 partial and 19 off-topic items were labelled the same way: 79 of 100, or 79%.

Next: Where they disagree

Step 1/3 · The diagonal

Figure. Two annotators, 100 overlapping items, illustrative counts. Rows are annotator A's label and columns annotator B's; the diagonal is agreement. The marginals in the totals give chance agreement 0.380, so kappa is (0.790 − 0.380) / (1 − 0.380) = 0.661.

What to notice. Raw agreement of 79% looks strong, but 52 of 100 items are relevant, so two annotators guessing from the label distribution would already agree 38% of the time. The weakest label is partial: 16 of 25 agree, with 5 called relevant and 4 called off-topic.

Text version

A three by three table with rows and columns relevant, partial and off-topic. The diagonal counts 44, 16 and 19 agreements, 79 of 100 items. Off the diagonal, A's relevant items become partial 6 times and off-topic twice; A's partial items become relevant 5 times and off-topic 4 times; A's off-topic items become relevant once and partial 3 times. Row totals are 52, 25 and 23; column totals are 50, 25 and 25. Steps highlight the diagonal, the disagreements and the chance correction; the readout gives both labels and the count.

  1. Choose the statistic from the label type and number of annotators before collecting data.
  2. Have at least two annotators label an overlapping sample independently.
  3. Compute agreement overall and per label, with intervals when the sample is small.
  4. Inspect confusion between specific label pairs rather than only the summary number.
  5. Revise the rubric where disagreement concentrates, then re-measure on fresh items.
  6. Store raw labels so agreement can be recomputed after adjudication.

Trade-offs

ChoiceGood forCostFails when
Percent agreementTrivial to explainInflated by class imbalanceOne label dominates
Cohen's kappaChance-corrected and standardTwo annotators, nominal onlyOrdinal scale or many annotators
Krippendorff's alphaFlexible across designsHarder to compute and explainTeam needs a quick number
Majority vote without agreement checkSimple labelsHides systematic disagreementLabels are subjective
Trade-off mapInteractive

Good for

Trivial to explain

Cost

Inflated by class imbalance

Fails when

One label dominates

  1. One label dominates
  2. Ordinal scale or many annotators
  3. Team needs a quick number
  4. Labels are subjective

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Percent agreement is good for Trivial to explain, costs Inflated by class imbalance, and fails when One label dominates. Cohen's kappa is good for Chance-corrected and standard, costs Two annotators, nominal only, and fails when Ordinal scale or many annotators. Krippendorff's alpha is good for Flexible across designs, costs Harder to compute and explain, and fails when Team needs a quick number. Majority vote without agreement check is good for Simple labels, costs Hides systematic disagreement, and fails when Labels are subjective

Failure modes

  • Kappa paradox. High percent agreement with low kappa because one class dominates. Debug: print the label distribution and the per-class agreement.
  • Non-independent labels. Annotators discussed items before labelling. Debug: check the collection protocol and timestamps for shared sessions.
  • Scale mismatch. Ordinal labels treated as nominal distort the statistic. Debug: confirm the statistic matches the measurement scale.

In production

Compute agreement on every new batch and on a fixed calibration set. Alert when a label's agreement drops after a rubric change.

Common misconceptions

People think: High agreement proves the labels are correct.

Actually: It proves consistency, and annotators can share the same misunderstanding.

People think: Low agreement means the annotators are bad.

Actually: It usually means the category is undefined or the rubric leaves room for interpretation.

Interview angle

Pick the statistic from the design, report per-label agreement and treat low agreement as a rubric problem first. A strong answer mentions the class-imbalance paradox. Quoting percent agreement alone, with no view of which labels disagreed, is the weak response.

Sources