Judge agreement

  • Glossary
  • Published
  • Reviewed
  • By

Judge agreement is how often an LLM judge's decisions match people's labels on the same records. Attify reports it two ways, percent agreement with a 95% interval and Cohen's kappa (Cohen, 1960): on people's first answers, and, in the standards' own sample audits, after adjudication too.

The thresholds#

From the human audit rules in each standard:

  • Validation: the judge agrees with people at least 85% on each judged decision (for support: resolved, and handoff), on at least 50 human-labelled records drawn for that standard version.
  • Replacing the reference judge with a cheaper one: at least 90% agreement with the reference on resolved and handoff, on a fresh random sample of 500 claims that no person labelled and no judge read before.

Percent agreement and kappa#

Percent agreement = records where the judge and the person gave the same label ÷ records labelled.

Kappa = (observed agreement − agreement expected by chance) ÷ (1 − agreement expected by chance), where chance agreement comes from how often each rater uses each label.

Synthetic example: 50 records. The person labels 40 resolved and 10 not; the judge labels 42 resolved and 8 not; they agree on 38 resolved and 6 not, 44 in all. Percent agreement 44 ÷ 50 = 88.0%. Chance agreement = (40/50 × 42/50) + (10/50 × 8/50) = 0.672 + 0.032 = 0.704. Kappa = (0.88 − 0.704) ÷ (1 − 0.704) = 0.59.

That judge passes the 85% bar with a kappa near 0.6, which is why both are published: the reader sees the agreement and how much of it chance would explain.

Sample size#

With 50 records the interval around percent agreement is wide (76.2% to 94.4% at 88.0%). The audit sample size calculator shows how the interval narrows as labels are added.

Questions

Why report kappa as well as percent agreement?

When one answer is far more common than the other, two raters agree often by chance. Cohen's kappa corrects for that, so a high percent agreement with a low kappa is a warning.

What agreement does Attify require?

At least 85% agreement with people on the outcome the judge decides (for support, resolved and handoff), on at least 50 human-labelled records per judged line of a standard. A cheaper production judge replaces the reference only at 90% agreement with it on a fresh 500.

Sources

  1. Educational and Psychological Measurement 20(1). A coefficient of agreement for nominal scales (Cohen, 1960). Read 8 Nov 2026.
  2. Attify. Attify Audited Resolution Standard v0.11: Human audit. Published 9 Nov 2026. Read 9 Nov 2026.