# Human audit

> A human audit is people labelling a random sample of what an LLM judge decided, blind to the judge, so the judge's agreement with people is measured rather than assumed.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/glossary/human-audit

A human audit is a person labelling a random sample of the records an LLM judge decided, without seeing the judge's answers, so that the judge's agreement with people is measured and published instead of assumed. In Attify's standards no judged score publishes until both the judge's validation and the score's own human audit are complete.

## Two audits

| | Validation of the judge | A score's own audit |
|---|---|---|
| When | Once per standard version's judged line | For every published score |
| Sample | At least 50 records drawn from that version's verification samples, never used to change a prompt | A random 50 of the score's judged records (all, when fewer) |
| Passes when | The judge agrees with people at least 85% on the outcome it decides (support: resolved and handoff; sales meetings: sales meeting; bookings: matches) | Every record is labelled |

## What people label

Exactly what the judge decided, per standard: for support, whether the issue was resolved, whether the final reply was a handoff, and the contact reason; for sales meetings, whether it was a sales meeting; for bookings, whether the appointment matches what was agreed.

## Adjudication

In a vendor verification, the bar above is measured on people's first answers in Attify's audit tool, with a 95% interval and Cohen's kappa ([judge agreement](https://attify.app/glossary/judge-agreement)). The standards' own sample audits add an adjudication step: a reference judge (the published prompt on the most capable model) labels the same rows, and where the reference and the production judge agree with each other but not with the person, the person re-checks blind. Those reports give first-pass and adjudicated agreement side by side.

## Versions

A patch version of a standard changes nothing the judge decides and keeps the validation of the line it patches. So does a minor version that changes only status or publishing rules: Standard #1 v0.10 and v0.11 keep v0.9's judged line. A minor version that changes what the judge decides (its prompt, an outcome definition or the judged population) needs new labels before a judged score publishes under it.

## Questions

### How many conversations do people label?

At least 50 per judged line of a standard to validate the judge, and a random 50 of each published score's own judged records (all of them when fewer).

### Do the labellers see what the judge decided?

No. They label blind: the audit tool shows the conversation, never the judge's label.

### Are human-labelled conversations used to train the judge?

No. No human-labelled record is ever an example in a judge prompt, and a prompt changed after reading labelled rows cannot be validated on those rows.

## Sources

- Attify: [Attify Audited Resolution Standard v0.11: Human audit](https://attify.app/standards/support-resolution#human-audit), read 2026-11-09
