Three standards for what AI agents actually achieve
Today we're publishing three standards: one for customer-support resolution, one for meetings booked by AI sales agents, and one for appointments booked by AI receptionists. Each says what counts as a real outcome for that job, and each asks the same three questions of every claim: did it happen, did it stick, did a human have to redo it.
Why standards#
AI agent vendors report outcome numbers, each counted under its own rule.
In support, the vendors we read that define a resolution count a conversation the customer didn't escalate. Intercom, for example, counts an "assumed resolution" when the customer leaves after Fin's answer without asking for more (Intercom). Where a vendor states a waiting window, it is 72 hours or less, and none waits a week (our research).
In sales, the AI SDR vendors that document a counting rule count a meeting at booking or earlier (our research). Human SDR teams get measured on meetings held, then on the opportunities those meetings turn into (The Bridge Group, 2025). Those are different numbers.
For AI receptionists, one of the 19 vendors whose pages we read defines what it counts as a booking (our research).
So the claim and the check need the same words. That's what a standard is for.
What they count#
- Support resolution (v0.11): a claimed resolution passes the audit if the customer didn't come back within 7 days, nobody had to step in or redirect them, and a judge reading the conversation confirms the issue was solved.
- Sales meetings (v0.6): a booked meeting passes the audit if the AI booked it, the person was a new prospect, no rep stepped in, and it was a real sales meeting; and, for a vendor that claims or bills held meetings, it wasn't cancelled and someone from the prospect's side showed up. A vendor that counts bookings sees its cancellations and no-shows beside the claim.
- Bookings (v0.6): a booked appointment passes the audit if the AI booked it, it was new, staff didn't have to fix it, and it matched what the customer asked for; and, for a vendor that claims or bills kept appointments, it wasn't cancelled and the customer came. A vendor that counts bookings sees its cancellations and no-shows beside the claim.
Three rules we hold ourselves to#
The fairest version of the vendor's own rule. We test the vendor's claim as the vendor set it. Where there's no vendor flag, we reconstruct the claim with the most generous rule a careful vendor would use. If someone can shrink the gap by pointing at something no careful vendor would count, the standard is wrong and we change it.
The customer's own records. Outcomes are checked in the customer's help desk, CRM, calendar or booking system wherever it holds them, never in the vendor's dashboard. Data the vendor streams counts only after the customer reconciles it against its own records.
People check the judge. Where a check needs reading, an LLM judge reads a sample with a prompt published word for word in the standard. Nothing it decides gets published until it agrees with people at least 85% of the time on at least 50 labelled records for that version.
What's next#
They're drafts. Tell us where they're wrong: how to comment. Coding tasks, collections and back-office processing come next, each as its own standard.
If you cite them, the citation blocks have a permanent link for each version.
Dor
Questions
Are the standards final?
- No. All three are drafts, open for comment. Every change is versioned and logged with its reason, and a change to what the judge decides needs new human labels before anything publishes under it.
Can I use the standards without Attify?
- Yes. They are published in full, judge prompts included, and the guides show how to apply them to an export by hand.