# Judge agreement

> Judge agreement is how often an LLM judge's decisions match people's labels on the same records, reported as a percentage with an interval and as Cohen's kappa.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/glossary/judge-agreement

Judge agreement is how often an LLM judge's decisions match people's labels on the same records. Attify reports it two ways, percent agreement with a 95% interval and Cohen's kappa ([Cohen, 1960](https://doi.org/10.1177/001316446002000104)): on people's first answers, and, in the standards' own sample audits, after adjudication too.

## The thresholds

From the [human audit](https://attify.app/glossary/human-audit) rules in each standard:

- **Validation**: the judge agrees with people at least 85% on each judged decision (for support: resolved, and handoff), on at least 50 human-labelled records drawn for that standard version.
- **Replacing the reference judge with a cheaper one**: at least 90% agreement with the reference on resolved and handoff, on a fresh random sample of 500 claims that no person labelled and no judge read before.

## Percent agreement and kappa

Percent agreement = records where the judge and the person gave the same label ÷ records labelled.

Kappa = (observed agreement − agreement expected by chance) ÷ (1 − agreement expected by chance), where chance agreement comes from how often each rater uses each label.

Synthetic example: 50 records. The person labels 40 resolved and 10 not; the judge labels 42 resolved and 8 not; they agree on 38 resolved and 6 not, 44 in all. Percent agreement 44 ÷ 50 = 88.0%. Chance agreement = (40/50 × 42/50) + (10/50 × 8/50) = 0.672 + 0.032 = 0.704. Kappa = (0.88 − 0.704) ÷ (1 − 0.704) = 0.59.

That judge passes the 85% bar with a kappa near 0.6, which is why both are published: the reader sees the agreement and how much of it chance would explain.

## Sample size

With 50 records the interval around percent agreement is wide (76.2% to 94.4% at 88.0%). The [audit sample size calculator](https://attify.app/tools/audit-sample-size) shows how the interval narrows as labels are added.

## Questions

### Why report kappa as well as percent agreement?

When one answer is far more common than the other, two raters agree often by chance. Cohen's kappa corrects for that, so a high percent agreement with a low kappa is a warning.

### What agreement does Attify require?

At least 85% agreement with people on the outcome the judge decides (for support, resolved and handoff), on at least 50 human-labelled records per judged line of a standard. A cheaper production judge replaces the reference only at 90% agreement with it on a fresh 500.

## Sources

- Educational and Psychological Measurement 20(1): [A coefficient of agreement for nominal scales (Cohen, 1960)](https://doi.org/10.1177/001316446002000104), read 2026-11-08
- Attify: [Attify Audited Resolution Standard v0.11: Human audit](https://attify.app/standards/support-resolution#human-audit), read 2026-11-09
