# How to check an AI vendor's claims before buying

> Get the outcome definition in writing, ask for results from customers' own systems with sample size and period, test on your own data, and put audit rights in the contract.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/guides/check-ai-vendor-claims-before-buying

Get the vendor's definition of its outcome in writing, ask for results measured in customers' own systems with the sample size, period and denominator, and test that definition on your own data before you commit. Then write the definition and your right to audit billed outcomes into the contract.

## 1. Get the definition in writing

For the number the vendor quotes (a resolution rate, meetings booked, a booking rate):

- **When is it counted?** At booking, after a quiet period (how long), after the customer confirms?
- **What voids it?** A repeat contact (within what window, on which channels)? Which handoffs? A cancellation or no-show?
- **What is in the denominator?** All conversations, AI-handled ones, those with a request? Calls of any length?
- **Who decides?** A rule, the vendor's own model, or the customer?

Compare it with how other vendors define the same outcome: [support](https://attify.app/compare/support-ai-agent-outcome-definitions), [sales meetings](https://attify.app/compare/ai-sdr-meeting-definitions), [bookings](https://attify.app/compare/ai-receptionist-booking-definitions).

## 2. Ask where the number comes from

| Ask | Why |
|---|---|
| Period and sample size | A rate from one good week, or from 40 conversations, is not a rate |
| Which customers, and how chosen | All customers, or the best ones |
| Source of the data | The customers' own help desks, CRMs and booking systems, or the vendor's dashboard ([provenance](https://attify.app/glossary/data-provenance)) |
| Interval | How much the number could move by chance |
| Who checked it | The vendor, the customer, or an independent party under a published rule |

In the US, the FTC has said there is no AI exemption from the laws against deceptive claims, and has brought cases over AI performance and earnings claims ([FTC, 25 September 2024](https://www.ftc.gov/news-events/news/press-releases/2024/09/ftc-announces-crackdown-deceptive-ai-claims-schemes)). Asking for substantiation is normal.

## 3. Read case studies for what they leave out

Named customer, period, baseline, defined metric, and whether you can talk to that customer. Check whether revenue figures for bookings and meetings are priced at scheduled value, before no-shows and cancellations.

## 4. Test on your own data

Run a pilot measured in your own systems under a rule you set before it starts. See [how to evaluate an AI agent pilot](https://attify.app/guides/evaluate-ai-agent-pilot).

## 5. Separate safety from outcomes

Security and safety attestations (SOC 2, AIUC-1, ISO/IEC 42001) answer whether the agent and vendor are safe to connect. They do not measure outcomes. Benchmarks and pre-purchase evaluations test simulated tasks. See [ways to verify an AI agent, compared](https://attify.app/compare).

## 6. Put it in the contract

- The outcome definition, word for word, including the window and what voids it.
- Your right to match every billed outcome to a record in your own systems, and to dispute those that do not match.
- What happens to outcomes that fail later (a customer who comes back, a meeting that never happens): credited, or not billed.
- Access to the per-outcome data in a format you can audit (ids that match your systems).

## 7. Check for a verification

If the vendor has published an Attify verification, it is listed in the [registry](https://attify.app/registry) with its status (Claim confirmed · within 5.0 points when Attify is 95% sure the vendor's count overstates audited outcomes by at most the claim tolerance, otherwise Claim not confirmed; or Lapsed, or Revoked), the audited rate, the gap to the claim, its standard version, evidence tier, provenance and period. Paste its attestation into [/verify](https://attify.app/verify) to check the signature and whether it is still current. A badge copied onto another site is not a badge: it should load from attify.app and link to the vendor's profile.

## Questions

### What should I ask an AI vendor to prove its resolution numbers?

The written definition of a resolution, the period and sample size behind the number, whether it comes from customers' own systems or the vendor's dashboard, and whether you can test the same definition on your own data during a pilot.

### Are case studies evidence?

They are a start. Check whether the customer is named, the period and baseline are stated, the metric is defined, and whether you can speak to that customer. A number without a denominator or a period is a claim, not evidence.

### Is SOC 2 or AIUC-1 enough?

They answer different questions: whether the vendor's controls and the agent's behaviour are safe and secure. Neither measures whether the agent's claimed outcomes held up for customers.

### How do I check a vendor's Attify verification?

Find the vendor in the registry, then paste its attestation into /verify, which checks the signature and says whether it is current, superseded, expired or revoked.

## Sources

- US Federal Trade Commission: [FTC Announces Crackdown on Deceptive AI Claims and Schemes](https://www.ftc.gov/news-events/news/press-releases/2024/09/ftc-announces-crackdown-deceptive-ai-claims-schemes), read 2026-11-08
- Attify: [Attify verification program, rules v5](https://attify.app/standards/program-rules), read 2026-11-09
