Methodology
Attify checks each outcome an AI agent claims against what the customer's own records show happened, under a published standard for that job type, and reports the audited rate next to the claimed rate with 95% intervals. The comparison is always against the fairest version of the vendor's own counting rule, so the gap cannot be shrunk by pointing at something no careful vendor would count.
This page explains the method shared by every standard. Each standard holds the exact rules for its job type, its judge prompt word for word, and a changelog:
Three questions#
Every standard asks the same three questions of a claimed outcome:
- Did it happen? The record exists in the customer's own system: the conversation, the meeting on the calendar, the appointment in the booking system.
- Did it stick? It held up afterwards: no repeat contact within 7 days for support; the meeting or appointment took place and was not cancelled or a no-show.
- Did a human have to redo it? No person on the customer's side stepped in: no handoff to a support agent, no salesperson writing to the prospect before the booking, no staff member fixing the appointment.
A claim that passes every check is an audited outcome. A claim that fails is counted at the first check it fails, so every report can say why claims failed, in shares that add up.
The claim tested#
Attify tests the claim the vendor actually makes.
- When the data holds the vendor's own flag (Intercom's resolution state on a Fin conversation, HubSpot's Customer Agent status, a witnessed
outcome.claimedevent), that flag is the claim, exactly as the vendor set it. Nothing is removed from it before testing. - When it does not, Attify reconstructs the claim with the most generous rule a careful vendor would use. For support: the support side replied, did not hand the customer off, and the customer sent nothing more for 24 hours. For meetings and appointments: the AI created the booking in the customer's own system.
Each standard publishes its claimed rule. If a skeptic could shrink the gap by pointing at something the rule counts that no careful vendor would count, the standard is wrong, and it is changed, versioned and logged.
Where the data comes from#
Every result states its provenance next to its numbers.
Outcomes are always checked in the customer's own systems where they exist: a repeat contact in the customer's help desk, a meeting's attendance in its calendar, an appointment's status in its booking system. A vendor's dashboard is never the evidence. More: data provenance, reconciliation.
Rules first, then a judge on a sample#
Most checks are rules applied to every record: who wrote each reply, whether the customer came back, whether the meeting was cancelled. One check needs reading: whether the issue was resolved, whether the meeting was a sales meeting, whether the appointment matches what the customer agreed to. An LLM judge reads a random sample of 200 claims per company for that check, with a prompt published word for word in the standard. No human-labelled record is ever an example in a prompt.
Every audited number states its evidence tier: the rules tier (no judge read it), the judge-confirmed tier (the headline), and a stronger tier where the data holds it (the customer's explicit answer that the issue was resolved, an opportunity created after a meeting, a recorded visit). Customer satisfaction ratings are reported as their own signal and never count as confirmation (why).
People check the judge#
A judge is only as good as its agreement with people.
- Validation. For each standard version, people label at least 50 records in Attify's audit tool, blind to the judge. The judge is validated when it agrees with them at least 85% of the time on each judged decision. Until then nothing it decided publishes.
- Every score's own audit. Each published score also needs a random 50 of its own judged records labelled by a person.
- Reported, not assumed. Reports give agreement with 95% intervals and Cohen's kappa. More: human audit, judge agreement.
Intervals and samples#
- Every rate carries a 95% interval: Wilson intervals for single rates (Wilson, 1927) and MOVER intervals for differences such as the gap (Newcombe, 1998). The derivations are in the standards' methods notes.
- Pooled numbers (a vendor's score across its customers, a buyer's benchmark, the Attify Index) use an interval that treats each company as a cluster, because outcomes at one company move together. It is never narrower than the interval on the pooled counts.
- Samples are seeded and logged, so anyone with the seed can draw the same sample again. Judge calls are cached by prompt, model and transcript, so a re-run gives the same answer.
- Rates print with one decimal everywhere, and a number reads the same on every surface that shows it.
The verdict sentence#
Every result opens with one sentence, written by a fixed template over the numbers, never by an LLM, so it cannot contradict them. The public demo's sentence, on synthetic data: "On synthetic data, audited resolution was 53.1% against 85.9% claimed, a 32.8-point gap; 15.8 of those points are conversations counted as resolved where support redirected the customer instead of answering."
For meetings and appointments, where every claim is a booking, the sentence gives the audited share of the claims: "58.3% of the meetings counted as booked passed the audit, a 41.7-point gap; ..." (synthetic example). Data that is self-reported, or witnessed and not yet reconciled, opens the sentence with that label, because the sentence travels without the page around it.
Vendor verification#
A vendor's agent is audited on its customers' data, under the program rules (rules v5):
- Independent customers only. A customer counts after four checks: a separate company on its own domain, real history before its invitation, Attify's direct confirmation when drawn at random, and the vendor's attestation that its customers are genuine. A customer proven fake revokes the program, publicly. More: customer independence.
- Coverage. The vendor sets the rule when it creates the program: every customer it invites counts, or Attify draws the customers that count at random among its connected independent customers. The vendor still chooses whom to invite, so a score describes those customers.
- Minimums. At least 3 independent customers and 500 counted records in the period (150 meetings for sales meetings). Below them there is no score, and no threshold is lowered for early vendors.
- Publishing. A vendor's first audit is private, and the vendor chooses whether to publish it. Once published, results are continuous: every later score publishes after a 7-day review window, where the vendor can add one public comment and no number changes. A published score cannot be hidden or edited. A vendor that stops verification shows Lapsed, with its full history kept.
- Status. Claim confirmed · within 5.0 points when Attify is 95% sure the vendor's count overstates audited outcomes by at most the claim tolerance (5.0 points to start, a published parameter reviewed after the first real programs: the gap's upper 95% limit, each customer a cluster, is within it), Claim not confirmed when it does not; the rule is the same for every job type. Lapsed after 14 days without fresh data, or Revoked with a public reason. Every badge, profile and attestation shows the audited rate, the gap to the claim, the job type and the period, and every change is logged on the vendor's public changelog.
- Attestation. Every published score and status is signed (Ed25519) and checkable at /verify.
Versions#
A standard version is major.minor.patch. A minor version changes what the judge decides (its prompt, the outcome definitions, or the population it judges), or a status or publishing rule. One that changes what the judge decides needs new human labels before anything publishes under it; one that changes only status or publishing rules keeps the judge's validation. A patch changes only names and wording. Every change, with its reason, is in the standard's changelog.
What this method cannot tell you#
- Satisfaction. Whether a customer was happy is a different question from whether the issue was resolved.
- Outcomes outside the record. A repeat contact by phone to a number the help desk does not link, a meeting held on a tool the customer did not connect.
- Quality beyond the standard. Tone, brand voice, upsell, compliance with a script. The standards measure outcomes, not style.
- Other job types. Results compare only within one job type, and support results only next to the contact-reason mix.
Where a check cannot be run on a customer's data, the result says "not testable" and shows the audited share without that check as an upper bound; such a customer cannot count toward a vendor verification.
Neutrality#
Attify describes where and why outcomes fail. It never advises a vendor on how to raise its own score. Prices never depend on a result. See Is Attify independent? and the corrections policy.
Questions
Does Attify use an LLM to grade AI agents?
- Partly. Rules decide most checks on every record. An LLM judge reads a random sample of 200 claims per company with a prompt published word for word in the standard, and a score publishes only after the judge agrees with people at least 85% of the time on at least 50 human-labelled records.
Whose numbers does Attify use?
- The customer's own systems: its help desk, CRM, calendar or booking system. A vendor's own stream counts only after the customer consents and reconciles it each month. Uploaded files never support a verification.
Why is the audited rate lower than the vendor's rate?
- Vendors count outcomes at the earliest moment they can: a customer going quiet, a meeting being booked. Attify checks what happened next: whether the customer came back within 7 days, whether a person had to step in, whether the meeting or appointment took place.
Can a vendor see or change its score before it publishes?
- A vendor's first audit is private, and the vendor chooses whether to publish it. After that, each new score is private for 7 days and the vendor can add one public comment. Nobody can change a number: the score is frozen when it is made.
How often are results updated?
- Rules-based numbers update daily from each connected system. A judged score is made every 30 days or when the vendor asks, and badges always show the current status.