Judge agreement
Judge agreement is how often an LLM judge's decisions match people's labels on the same records. Attify reports it two ways, percent agreement with a 95% interval and Cohen's kappa (Cohen, 1960): on people's first answers, and, in the standards' own sample audits, after adjudication too.
The thresholds#
From the human audit rules in each standard:
- Validation: the judge agrees with people at least 85% on each judged decision (for support: resolved, and handoff), on at least 50 human-labelled records drawn for that standard version.
- Replacing the reference judge with a cheaper one: at least 90% agreement with the reference on resolved and handoff, on a fresh random sample of 500 claims that no person labelled and no judge read before.
Percent agreement and kappa#
Percent agreement = records where the judge and the person gave the same label ÷ records labelled.
Kappa = (observed agreement − agreement expected by chance) ÷ (1 − agreement expected by chance), where chance agreement comes from how often each rater uses each label.
Synthetic example: 50 records. The person labels 40 resolved and 10 not; the judge labels 42 resolved and 8 not; they agree on 38 resolved and 6 not, 44 in all. Percent agreement 44 ÷ 50 = 88.0%. Chance agreement = (40/50 × 42/50) + (10/50 × 8/50) = 0.672 + 0.032 = 0.704. Kappa = (0.88 − 0.704) ÷ (1 − 0.704) = 0.59.
That judge passes the 85% bar with a kappa near 0.6, which is why both are published: the reader sees the agreement and how much of it chance would explain.
Sample size#
With 50 records the interval around percent agreement is wide (76.2% to 94.4% at 88.0%). The audit sample size calculator shows how the interval narrows as labels are added.
Questions
Why report kappa as well as percent agreement?
- When one answer is far more common than the other, two raters agree often by chance. Cohen's kappa corrects for that, so a high percent agreement with a low kappa is a warning.
What agreement does Attify require?
- At least 85% agreement with people on the outcome the judge decides (for support, resolved and handoff), on at least 50 human-labelled records per judged line of a standard. A cheaper production judge replaces the reference only at 90% agreement with it on a fresh 500.