# Methodology

> How Attify checks an AI agent's claimed outcomes against the customer's own records, under a published standard per job type, with 95% intervals.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/methodology

Attify checks each outcome an AI agent claims against what the customer's own records show happened, under a published standard for that job type, and reports the audited rate next to the claimed rate with 95% intervals. The comparison is always against the fairest version of the vendor's own counting rule, so the gap cannot be shrunk by pointing at something no careful vendor would count.

This page explains the method shared by every standard. Each standard holds the exact rules for its job type, its judge prompt word for word, and a changelog:

| Job type | Standard | Version |
|---|---|---|
| Customer-support resolution | [Attify Audited Resolution Standard](https://attify.app/standards/support-resolution) | v0.11 |
| Meetings booked by AI sales agents | [Attify Audited Meeting Standard](https://attify.app/standards/sales-meetings) | v0.6 |
| Appointments booked by AI receptionists | [Attify Audited Booking Standard](https://attify.app/standards/bookings) | v0.6 |

## Three questions

Every standard asks the same three questions of a claimed outcome:

1. **Did it happen?** The record exists in the customer's own system: the conversation, the meeting on the calendar, the appointment in the booking system.
2. **Did it stick?** It held up afterwards: no repeat contact within 7 days for support; the meeting or appointment took place and was not cancelled or a no-show.
3. **Did a human have to redo it?** No person on the customer's side stepped in: no handoff to a support agent, no salesperson writing to the prospect before the booking, no staff member fixing the appointment.

A claim that passes every check is an audited outcome. A claim that fails is counted at the first check it fails, so every report can say why claims failed, in shares that add up.

## The claim tested

Attify tests the claim the vendor actually makes.

- **When the data holds the vendor's own flag** (Intercom's resolution state on a Fin conversation, HubSpot's Customer Agent status, a witnessed `outcome.claimed` event), that flag is the claim, exactly as the vendor set it. Nothing is removed from it before testing.
- **When it does not**, Attify reconstructs the claim with the most generous rule a careful vendor would use. For support: the support side replied, did not hand the customer off, and the customer sent nothing more for 24 hours. For meetings and appointments: the AI created the booking in the customer's own system.

Each standard publishes its claimed rule. If a skeptic could shrink the gap by pointing at something the rule counts that no careful vendor would count, the standard is wrong, and it is changed, versioned and logged.

## Where the data comes from

Every result states its provenance next to its numbers.

| Provenance | What it is | What it can support |
|---|---|---|
| Connected | Attify read every record of the period from the customer's own system (help desk, CRM, calendar, booking system, or an agent platform account the customer owns), through the customer's own authorization. Nobody chose what to include. | A vendor verification and badge; a buyer's own check |
| Witnessed | The vendor's agent sent each interaction to Attify as it happened, signed and timestamped on arrival, through the [Ingest API](https://attify.app/docs/ingest), an SDK or OpenTelemetry, or Attify read the vendor's own platform account as it happened. | A verification only after the customer consents in Attify and reconciles each month; before that, the vendor's private report |
| Uploaded | A file someone chose and mapped. | A buyer's own check only, labelled self-reported |

Outcomes are always checked in the customer's own systems where they exist: a repeat contact in the customer's help desk, a meeting's attendance in its calendar, an appointment's status in its booking system. A vendor's dashboard is never the evidence. More: [data provenance](https://attify.app/glossary/data-provenance), [reconciliation](https://attify.app/glossary/reconciliation).

## Rules first, then a judge on a sample

Most checks are rules applied to every record: who wrote each reply, whether the customer came back, whether the meeting was cancelled. One check needs reading: whether the issue was resolved, whether the meeting was a sales meeting, whether the appointment matches what the customer agreed to. An LLM judge reads a random sample of 200 claims per company for that check, with a prompt published word for word in the standard. No human-labelled record is ever an example in a prompt.

Every audited number states its [evidence tier](https://attify.app/glossary/evidence-tier): the rules tier (no judge read it), the judge-confirmed tier (the headline), and a stronger tier where the data holds it (the customer's explicit answer that the issue was resolved, an opportunity created after a meeting, a recorded visit). Customer satisfaction ratings are reported as their own signal and never count as confirmation ([why](https://attify.app/glossary/customer-confirmed-vs-satisfaction)).

## People check the judge

A judge is only as good as its agreement with people.

- **Validation.** For each standard version, people label at least 50 records in Attify's audit tool, blind to the judge. The judge is validated when it agrees with them at least 85% of the time on each judged decision. Until then nothing it decided publishes.
- **Every score's own audit.** Each published score also needs a random 50 of its own judged records labelled by a person.
- **Reported, not assumed.** Reports give agreement with 95% intervals and Cohen's kappa. More: [human audit](https://attify.app/glossary/human-audit), [judge agreement](https://attify.app/glossary/judge-agreement).

## Intervals and samples

- Every rate carries a 95% interval: Wilson intervals for single rates ([Wilson, 1927](https://doi.org/10.1080/01621459.1927.10502953)) and MOVER intervals for differences such as the gap ([Newcombe, 1998](https://doi.org/10.1002/%28SICI%291097-0258%2819980430%2917%3A8%3C873%3A%3AAID-SIM779%3E3.0.CO%3B2-I)). The derivations are in the standards' methods notes.
- Pooled numbers (a vendor's score across its customers, a buyer's benchmark, the [Attify Index](https://attify.app/index)) use an interval that treats each company as a cluster, because outcomes at one company move together. It is never narrower than the interval on the pooled counts.
- Samples are seeded and logged, so anyone with the seed can draw the same sample again. Judge calls are cached by prompt, model and transcript, so a re-run gives the same answer.
- Rates print with one decimal everywhere, and a number reads the same on every surface that shows it.

## The verdict sentence

Every result opens with one sentence, written by a fixed template over the numbers, never by an LLM, so it cannot contradict them. The public demo's sentence, on synthetic data: "On synthetic data, audited resolution was 53.1% against 85.9% claimed, a 32.8-point gap; 15.8 of those points are conversations counted as resolved where support redirected the customer instead of answering."

For meetings and appointments, where every claim is a booking, the sentence gives the audited share of the claims: "58.3% of the meetings counted as booked passed the audit, a 41.7-point gap; ..." (synthetic example). Data that is self-reported, or witnessed and not yet reconciled, opens the sentence with that label, because the sentence travels without the page around it.

## Vendor verification

A vendor's agent is audited on its customers' data, under the [program rules](https://attify.app/standards/program-rules) (rules v5):

- **Independent customers only.** A customer counts after four checks: a separate company on its own domain, real history before its invitation, Attify's direct confirmation when drawn at random, and the vendor's attestation that its customers are genuine. A customer proven fake revokes the program, publicly. More: [customer independence](https://attify.app/glossary/customer-independence).
- **Coverage.** The vendor sets the rule when it creates the program: every customer it invites counts, or Attify draws the customers that count at random among its connected independent customers. The vendor still chooses whom to invite, so a score describes those customers.
- **Minimums.** At least 3 independent customers and 500 counted records in the period (150 meetings for sales meetings). Below them there is no score, and no threshold is lowered for early vendors.
- **Publishing.** A vendor's first audit is private, and the vendor chooses whether to publish it. Once published, results are continuous: every later score publishes after a 7-day review window, where the vendor can add one public comment and no number changes. A published score cannot be hidden or edited. A vendor that stops verification shows Lapsed, with its full history kept.
- **Status.** Claim confirmed · within 5.0 points when Attify is 95% sure the vendor's count overstates audited outcomes by at most the claim tolerance (5.0 points to start, a published parameter reviewed after the first real programs: the gap's upper 95% limit, each customer a cluster, is within it), Claim not confirmed when it does not; the rule is the same for every job type. Lapsed after 14 days without fresh data, or Revoked with a public reason. Every badge, profile and attestation shows the audited rate, the gap to the claim, the job type and the period, and every change is logged on the vendor's public changelog.
- **Attestation.** Every published score and status is signed (Ed25519) and checkable at [/verify](https://attify.app/verify).

## Versions

A standard version is `major.minor.patch`. A minor version changes what the judge decides (its prompt, the outcome definitions, or the population it judges), or a status or publishing rule. One that changes what the judge decides needs new human labels before anything publishes under it; one that changes only status or publishing rules keeps the judge's validation. A patch changes only names and wording. Every change, with its reason, is in the standard's changelog.

## What this method cannot tell you

- **Satisfaction.** Whether a customer was happy is a different question from whether the issue was resolved.
- **Outcomes outside the record.** A repeat contact by phone to a number the help desk does not link, a meeting held on a tool the customer did not connect.
- **Quality beyond the standard.** Tone, brand voice, upsell, compliance with a script. The standards measure outcomes, not style.
- **Other job types.** Results compare only within one job type, and support results only next to the contact-reason mix.

Where a check cannot be run on a customer's data, the result says "not testable" and shows the audited share without that check as an upper bound; such a customer cannot count toward a vendor verification.

## Neutrality

Attify describes where and why outcomes fail. It never advises a vendor on how to raise its own score. Prices never depend on a result. See [Is Attify independent?](https://attify.app/independence) and the [corrections policy](https://attify.app/corrections).

## Questions

### Does Attify use an LLM to grade AI agents?

Partly. Rules decide most checks on every record. An LLM judge reads a random sample of 200 claims per company with a prompt published word for word in the standard, and a score publishes only after the judge agrees with people at least 85% of the time on at least 50 human-labelled records.

### Whose numbers does Attify use?

The customer's own systems: its help desk, CRM, calendar or booking system. A vendor's own stream counts only after the customer consents and reconciles it each month. Uploaded files never support a verification.

### Why is the audited rate lower than the vendor's rate?

Vendors count outcomes at the earliest moment they can: a customer going quiet, a meeting being booked. Attify checks what happened next: whether the customer came back within 7 days, whether a person had to step in, whether the meeting or appointment took place.

### Can a vendor see or change its score before it publishes?

A vendor's first audit is private, and the vendor chooses whether to publish it. After that, each new score is private for 7 days and the vendor can add one public comment. Nobody can change a number: the score is frozen when it is made.

### How often are results updated?

Rules-based numbers update daily from each connected system. A judged score is made every 30 days or when the vendor asks, and badges always show the current status.

## Sources

- Attify: [Attify Audited Resolution Standard v0.11](https://attify.app/standards/support-resolution), read 2026-11-09
- Attify: [Attify Audited Meeting Standard v0.6](https://attify.app/standards/sales-meetings), read 2026-11-09
- Attify: [Attify Audited Booking Standard v0.6](https://attify.app/standards/bookings), read 2026-11-09
- Journal of the American Statistical Association: [Probable inference, the law of succession, and statistical inference (Wilson, 1927)](https://doi.org/10.1080/01621459.1927.10502953), read 2026-11-08
- Statistics in Medicine: [Interval estimation for a difference between independent proportions (Newcombe, 1998)](https://doi.org/10.1002/%28SICI%291097-0258%2819980430%2917%3A8%3C873%3A%3AAID-SIM779%3E3.0.CO%3B2-I), read 2026-11-08
