Human audit
A human audit is a person labelling a random sample of the records an LLM judge decided, without seeing the judge's answers, so that the judge's agreement with people is measured and published instead of assumed. In Attify's standards no judged score publishes until both the judge's validation and the score's own human audit are complete.
Two audits#
What people label#
Exactly what the judge decided, per standard: for support, whether the issue was resolved, whether the final reply was a handoff, and the contact reason; for sales meetings, whether it was a sales meeting; for bookings, whether the appointment matches what was agreed.
Adjudication#
In a vendor verification, the bar above is measured on people's first answers in Attify's audit tool, with a 95% interval and Cohen's kappa (judge agreement). The standards' own sample audits add an adjudication step: a reference judge (the published prompt on the most capable model) labels the same rows, and where the reference and the production judge agree with each other but not with the person, the person re-checks blind. Those reports give first-pass and adjudicated agreement side by side.
Versions#
A patch version of a standard changes nothing the judge decides and keeps the validation of the line it patches. So does a minor version that changes only status or publishing rules: Standard #1 v0.10 and v0.11 keep v0.9's judged line. A minor version that changes what the judge decides (its prompt, an outcome definition or the judged population) needs new labels before a judged score publishes under it.
Questions
How many conversations do people label?
- At least 50 per judged line of a standard to validate the judge, and a random 50 of each published score's own judged records (all of them when fewer).
Do the labellers see what the judge decided?
- No. They label blind: the audit tool shows the conversation, never the judge's label.
Are human-labelled conversations used to train the judge?
- No. No human-labelled record is ever an example in a judge prompt, and a prompt changed after reading labelled rows cannot be validated on those rows.