# How to audit conversations your AI agent marked resolved

> A step-by-step audit of AI-resolved support conversations: draw a seeded random sample, label it blind with written rules, measure agreement, and report the share with an interval.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/guides/audit-ai-resolved-conversations

Draw a random sample of the conversations your AI agent counted as resolved (with a recorded seed, so anyone can draw it again), have people label each one against written rules without seeing the vendor's verdict, and report the share they found resolved with its 95% interval. Before reading anything, remove the claims your help desk already shows failed: a repeat contact within 7 days, or a person stepping in.

## 1. Define the labels before you read

Write them down and keep them fixed for the whole audit. The labels in Attify's standard, which you can use as they are:

| Label | Values | Rule |
|---|---|---|
| Resolved | yes, no, unclear | Yes: the reply answered the customer's request specifically enough that nothing was left to do. Unclear: there was no request (a thanks, a greeting), or the transcript is too thin to tell |
| Handoff | yes, no | Yes: the final reply sent the customer elsewhere instead of answering (another channel, a phone number, a link with no answer) |
| Reason | order, billing, account access, technical, complaint, other | The customer's main reason for contacting support |

The judge prompt that applies these labels is published word for word in the [standard's appendix](https://attify.app/standards/support-resolution).

## 2. Remove what the records already settle

From all claimed conversations of the period, set aside those where the same customer came back within 7 days, and those where a person replied after the AI. Those failed without anyone reading them. Audit the rest.

## 3. Draw the sample

- Sort the remaining conversations by id. Draw a simple random sample with a fixed seed (for example `random.Random("2026-10:audit")` in Python) and record the seed.
- Sample size: 200 is a practical default. Use the [audit sample size calculator](https://attify.app/tools/audit-sample-size) to choose.
- Do not let anyone hand-pick or swap conversations.

## 4. Label blind

- Labellers see the conversation only, never the vendor's verdict or other labellers' answers.
- Have two people label the same 50 conversations, then compare. Report percent agreement and Cohen's kappa ([Cohen, 1960](https://doi.org/10.1177/001316446002000104); worked example on [judge agreement](https://attify.app/glossary/judge-agreement)). If they agree less than 85% on resolved, tighten the written rules and start a fresh sample.
- Resolve disagreements by a third reading, blind.

## 5. Report

- **Share resolved** among sampled conversations with a request (leave out "unclear"), with its 95% Wilson interval.
- **Share with no request**, with its interval: those leave both the claimed and the audited count.
- **Handoffs found by reading**, with examples by id.
- **Reasons**: the share resolved per contact reason, with the number read behind each.
- **The seed, the period, the labels' wording and the agreement** between labellers.

To turn the sample into an audited rate for the whole period, see [step 5 of the measurement guide](https://attify.app/guides/measure-ai-support-agent-resolution-rate#5-compute-the-rates).

## Common mistakes

- Reading conversations the vendor flagged as "best examples", or the first 200 in the export.
- Letting the labeller see the AI's resolution flag.
- Changing the rules after reading some labelled conversations, then counting those conversations.
- Comparing the audit share with a rate that has a different denominator.

## Questions

### How many conversations do I need to read?

200 gives a 95% interval of about ±7 points when the true share is near 50%, and narrower when it is near 0% or 100%. 385 gets it to about ±5. The audit sample size calculator shows the numbers for your case.

### Can I use an LLM to grade the sample instead of people?

You can, if you also measure it against people. Label 50 of the same conversations yourself, blind, and check agreement. Attify's judge prompt is published word for word in its standard, so you can read exactly what it asks.

### Should the vendor do the audit?

The vendor can help with access, but the sample, the labels and the counting should be yours. Whoever picks the sample can move the result.

## Sources

- Attify: [Attify Audited Resolution Standard v0.11: Appendix B (judge prompt), Human audit](https://attify.app/standards/support-resolution#human-audit), read 2026-11-09
- Educational and Psychological Measurement 20(1): [A coefficient of agreement for nominal scales (Cohen, 1960)](https://doi.org/10.1177/001316446002000104), read 2026-11-08
