How to check an AI vendor's claims before buying
Get the vendor's definition of its outcome in writing, ask for results measured in customers' own systems with the sample size, period and denominator, and test that definition on your own data before you commit. Then write the definition and your right to audit billed outcomes into the contract.
1. Get the definition in writing#
For the number the vendor quotes (a resolution rate, meetings booked, a booking rate):
- When is it counted? At booking, after a quiet period (how long), after the customer confirms?
- What voids it? A repeat contact (within what window, on which channels)? Which handoffs? A cancellation or no-show?
- What is in the denominator? All conversations, AI-handled ones, those with a request? Calls of any length?
- Who decides? A rule, the vendor's own model, or the customer?
Compare it with how other vendors define the same outcome: support, sales meetings, bookings.
2. Ask where the number comes from#
In the US, the FTC has said there is no AI exemption from the laws against deceptive claims, and has brought cases over AI performance and earnings claims (FTC, 25 September 2024). Asking for substantiation is normal.
3. Read case studies for what they leave out#
Named customer, period, baseline, defined metric, and whether you can talk to that customer. Check whether revenue figures for bookings and meetings are priced at scheduled value, before no-shows and cancellations.
4. Test on your own data#
Run a pilot measured in your own systems under a rule you set before it starts. See how to evaluate an AI agent pilot.
5. Separate safety from outcomes#
Security and safety attestations (SOC 2, AIUC-1, ISO/IEC 42001) answer whether the agent and vendor are safe to connect. They do not measure outcomes. Benchmarks and pre-purchase evaluations test simulated tasks. See ways to verify an AI agent, compared.
6. Put it in the contract#
- The outcome definition, word for word, including the window and what voids it.
- Your right to match every billed outcome to a record in your own systems, and to dispute those that do not match.
- What happens to outcomes that fail later (a customer who comes back, a meeting that never happens): credited, or not billed.
- Access to the per-outcome data in a format you can audit (ids that match your systems).
7. Check for a verification#
If the vendor has published an Attify verification, it is listed in the registry with its status (Claim confirmed · within 5.0 points when Attify is 95% sure the vendor's count overstates audited outcomes by at most the claim tolerance, otherwise Claim not confirmed; or Lapsed, or Revoked), the audited rate, the gap to the claim, its standard version, evidence tier, provenance and period. Paste its attestation into /verify to check the signature and whether it is still current. A badge copied onto another site is not a badge: it should load from attify.app and link to the vendor's profile.
Questions
What should I ask an AI vendor to prove its resolution numbers?
- The written definition of a resolution, the period and sample size behind the number, whether it comes from customers' own systems or the vendor's dashboard, and whether you can test the same definition on your own data during a pilot.
Are case studies evidence?
- They are a start. Check whether the customer is named, the period and baseline are stated, the metric is defined, and whether you can speak to that customer. A number without a denominator or a period is a claim, not evidence.
Is SOC 2 or AIUC-1 enough?
- They answer different questions: whether the vendor's controls and the agent's behaviour are safe and secure. Neither measures whether the agent's claimed outcomes held up for customers.
How do I check a vendor's Attify verification?
- Find the vendor in the registry, then paste its attestation into /verify, which checks the signature and says whether it is current, superseded, expired or revoked.