# Research: other ways to verify an AI agent

> What benchmarks, evaluations, certifications, ROI studies, reviews, QA tools, eval platforms, dashboards and pilots each measure, from their own documentation.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/research/ways-to-verify-ai-agents

Every other way of checking an AI agent answers a question of its own: benchmarks and pre-purchase evaluations test a simulated or controlled copy, certifications test controls, ROI studies model a composite company, analyst reports and reviews collect vendor-supplied evidence and opinion, and QA tools, evaluation platforms, vendor dashboards and pilots score real traffic under a rule the vendor or the buyer writes. Production verification checks the outcomes an agent claimed, continuously, in the buyer's own records, against a published standard that neither the vendor nor the buyer sets.

Facts below come from each method's own documentation, read on 8 November 2026; a date in a source link is the date the page shows. Conclusions of Attify's own are labelled as such.

## At a glance

Read on 8 November 2026.

| Method | What it measures | Stage | Whose data | Who sets what counts | Public | Refresh | Source |
|---|---|---|---|---|---|---|---|
| Public benchmarks (tau-bench, tau2-bench, CRMArena-Pro) | Task success of a model or agent setup with simulated users in sandboxes; tau-bench judges success by the final database state and adds pass^k, a reliability measure over repeated trials | Before purchase | Public, simulated | The researchers | Yes | When authors publish a new version; the tau-bench repository says its tasks are no longer updated | [tau-bench, 17 Jun 2024](https://arxiv.org/abs/2406.12045), [repository](https://github.com/sierra-research/tau-bench), [tau2-bench, 9 Jun 2025](https://arxiv.org/abs/2506.07982), [CRMArena-Pro, 24 May 2025](https://arxiv.org/abs/2505.18878) |
| G2 Agent Evaluations | Task success in a simulated support company (38 business tools, a private set of 50 tasks), plus accuracy, policy compliance and relevance (LLM judges) and completeness (a deterministic check) | Before purchase | G2's simulated company and tasks | G2 | Leaderboards, separate from G2 reviews | Several times a year, or after a material product change | [G2, updated 8 Sep 2026](https://documentation.g2.com/docs/agent-evaluations), [methodology](https://ai.g2.com/evaluations/methodology) |
| Vals AI | Model and agent accuracy on tasks in finance, legal, coding, web search and public benefits; no support or sales benchmark was listed when read | Before purchase | Privately held test sets | Vals AI | Scores yes, test sets no | A date per benchmark | [About](https://www.vals.ai/about), [Benchmarks](https://www.vals.ai/benchmarks) |
| AIUC-1 | Security, safety and reliability controls in six domains, and adversarial tests: jailbreaks, prompt injection, data leakage, hallucinations, unsafe tool calls | Before and during operation | AIUC's scenarios run against the agent; an accredited auditor's review of controls | The standard and the auditor | Whether the audit report is published is not stated | Standard quarterly; retests quarterly; certificate valid one year | [AIUC-1, updated 15 Oct 2026](https://aiuc-1.com) |
| SOC 2 | Design and operating effectiveness of a service's controls for security, availability, processing integrity, confidentiality or privacy | Service in operation | The organization's controls | A licensed CPA, under AICPA attestation standards | Not stated on the pages read | Renewed yearly | [AICPA](https://www.aicpa-cima.com/resources/landing/system-and-organization-controls-soc-suite-of-services), [AICPA and AIUC, 19 Oct 2026](https://www.aicpa-cima.com/news/article/auditors-meet-urgent-demand-for-ai-assurance-with-new-offering-from-aicpa-and-aiuc) |
| ISO/IEC 42001 | Requirements for an AI management system within an organization | Organization in operation | The organization's records | The standard | Not confirmed | Not confirmed | [ISO](https://www.iso.org/standard/42001) |
| COPC CX Standard, Release 8.0 | Management requirements for customer-experience operations, with the same performance standards whether a human or technology serves the customer; technology that makes frequent errors must be taken out of transactions until fixed | In production, organization-wide | The organization's operations | COPC | Not stated | Not stated | [COPC](https://www.copc.com/copc-standards/r8-whats-new/) |
| Forrester Total Economic Impact | Modeled financial impact on a composite organization built from customer interviews; studies for technology companies are commissioned and marked so | Before or after purchase | Interviews, then a model | Forrester, which keeps editorial control | The study document | One study at a time | [Forrester](https://www.forrester.com/policies/tei/), [sample study, 2022](https://tei.forrester.com/go/forrester/tei) |
| Forrester Wave | Vendor scores from a questionnaire, briefings, demos and reference customers, against the analyst's criteria and weights | Before purchase | Vendor-supplied, and references the vendor names | The analyst | The report | Not stated | [Forrester](https://www.forrester.com/policies/forrester-wave-methodology/) |
| G2 reviews | User opinion, filtered automatically and checked by hand; incentivized reviews labelled; the experience described must be from the past two years | After purchase | Reviewers' own experience | Reviewers | Yes | As reviews arrive | [G2](https://legal.g2.com/community-guidelines) |
| Conversation QA (Zendesk QA, Observe.AI) | Every conversation scored against built-in categories and categories the customer writes; Zendesk QA also reviews AI agent conversations and flags unresolved ones | In production | The customer's own conversations | The customer, through the rubric | No | Continuous | [Zendesk QA, updated 21 Sep 2026](https://zendesk.com/service/quality-assurance/), [Observe.AI](https://observe.ai/platform/interaction-intelligence) |
| Evaluation and observability platforms (Braintrust, LangSmith, Langfuse, Arize AX) | Evaluator scores (LLM judges, code checks, human review) on datasets before release and on production traces; online scores have no ground truth and rest on the judge | Before release and in production | The builder's datasets and traces | The team that builds or runs the agent | No | Continuous | [Braintrust](https://docs.braintrust.dev/evaluate), [LangSmith](https://docs.langchain.com/langsmith/evaluation-concepts), [Langfuse](https://langfuse.com/docs/evaluation/overview), [Arize](https://arize.com/docs/ax/evaluate/evaluators) |
| Vendor dashboards (examples: Intercom, Zendesk) | Outcomes under the vendor's own definition, which at Intercom and Zendesk also sets the bill | In production | The vendor's platform | The vendor | No | Continuous | [Intercom, updated 23 Sep 2026](https://www.intercom.com/help/en/articles/8205718-fin-ai-agent-outcomes), [Zendesk](https://support.zendesk.com/hc/en-us/articles/9570369117338-About-automated-resolution-tiers) |
| Pilots and holdout tests | The agent against a control group handled the old way, as a randomized controlled experiment | In production, for weeks | The buyer's own traffic | The buyer | No | Once per pilot | [Kohavi, Tang and Xu, 2020](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B518F59) |
| Production verification (Attify) | The outcomes an agent claimed, checked against a published standard per job type, with 95% intervals | In production, continuous | The customer's own systems (connected), or the vendor's stream that the customer consents to and reconciles (witnessed) | A published standard | A vendor's scores once it publishes its first (the first audit is private), and the Attify Index | Rules tier daily; a judged score every 30 days | [Methodology](https://attify.app/methodology) |

## What each cannot tell a buyer

This section is Attify's own reading of the documentation above.

- **Benchmarks** compare models or agent setups on fixed tasks with a simulated user. Public tasks can be tuned to, which is one reason G2 and Vals AI keep theirs private.
- **Pre-purchase evaluations** are a dated snapshot of a configured test copy. G2 states that its results reflect a controlled environment and may not match every real deployment.
- **Certifications and attestations** test controls. None of the pages read says it measures whether a customer's issue was solved or a meeting was held, and a safe agent can still fail the job.
- **ROI studies** project results for a composite company. Forrester's sample study says it makes no assumption about the return other organizations will get and tells readers to use their own estimates.
- **Analyst reports and reviews** rest on vendor-supplied answers, demos, references the vendor names, and opinions of people who chose to write. Reviews have no denominator, so they give no rate.
- **QA tools and evaluation platforms** score against a rubric or evaluator written by whoever runs the tool, often the vendor. A conversation scored as good can still come back as a repeat contact, and outcomes outside the trace (a later contact in the help desk, an attended meeting) appear only if someone wires them in.
- **Vendor dashboards** are the vendor reporting on itself under its own rule.
- **Pilots** show whether the agent beat the old process during the pilot, on a configuration the vendor may tune for it. Nobody outside the buyer checks the result, and results do not compare across buyers.
- **Production verification (Attify)** describes only the customers the vendor chose to invite, and a vendor's first audit stays private unless it publishes. It sees only outcomes recorded in the systems a customer connected, does not measure satisfaction or quality beyond the standard (tone, compliance with a script), and covers only job types with a published standard ([methodology](https://attify.app/methodology)).

Which method answers which buyer question is in [ways to verify an AI agent, compared](https://attify.app/compare), with one page per method.

## Not read or not confirmed

- Gartner's Magic Quadrant and Peer Insights method pages refused automated reading; nothing here relies on search summaries of them.
- ISO/IEC 42001's audit stages and cadence. ISO's page refused automated reading, and secondary summaries are not used.
- SOC 2 Type 1 and Type 2: the AICPA pages read do not define them.
- COPC's certification process, cost and listing, and when certification to Release 8.0 begins.
- Who pays for AIUC-1, COPC and Vals AI: not stated on the pages read. G2 says its evaluations are free to vendors during the beta and that it plans paid evaluations.
- Prices of the QA tools and evaluation platforms.
- Whether each of G2's evaluation categories has a published methodology.
- Other independent, vendor-by-vendor evaluation programs for support or sales agents. None turned up in this research, which does not mean none exists.

## Questions

### What is the difference between a SOC 2 report, a model benchmark and an outcome verification?

A SOC 2 report is a licensed CPA's opinion on whether a service's controls for security and related criteria are designed and operate well. A benchmark scores a model or agent on fixed, usually simulated tasks. An outcome verification checks, in the buyer's own records and under a published rule, what share of the outcomes an agent claimed held up.

### Does tau-bench predict how an AI agent performs on my real tickets?

Not directly. tau-bench scores agents against LLM-simulated users in airline and retail sandboxes, and its repository says the tasks are no longer updated. Your customers, policies and integrations are not in it.

### Which analyst firms publish measured results for AI customer service agents?

None of the analyst methods read for this research measures outcome rates in a buyer's deployment. A Forrester Wave scores vendors from a questionnaire, briefings, demos and reference customers; Gartner's method pages refused automated reading and are not described here.

### Can I trust G2 ratings of AI agents for performance?

G2 moderates reviews, labels incentivized ones and bars the product's employees and direct competitors from reviewing it. Reviews are the opinions of people who chose to write one, with no denominator, so they cannot give an outcome rate or an interval.

## Sources

- arXiv: [tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao, Shinn, Razavi, Narasimhan)](https://arxiv.org/abs/2406.12045), read 2026-11-08
- Sierra Research: [tau-bench repository](https://github.com/sierra-research/tau-bench), read 2026-11-08
- arXiv: [tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres, Dong, Ray, Si, Narasimhan)](https://arxiv.org/abs/2506.07982), read 2026-11-08
- arXiv: [CRMArena-Pro (Huang et al.)](https://arxiv.org/abs/2505.18878), read 2026-11-08
- G2: [Agent Evaluations](https://documentation.g2.com/docs/agent-evaluations), read 2026-11-08
- G2: [Customer Experience AI Agent Evaluation methodology](https://ai.g2.com/evaluations/methodology), read 2026-11-08
- Vals AI: [About](https://www.vals.ai/about), read 2026-11-08
- Vals AI: [Benchmarks](https://www.vals.ai/benchmarks), read 2026-11-08
- AIUC: [AIUC-1](https://aiuc-1.com), read 2026-11-08
- AICPA and CIMA: [Auditors meet urgent demand for AI assurance with new offering from AICPA and AIUC](https://www.aicpa-cima.com/news/article/auditors-meet-urgent-demand-for-ai-assurance-with-new-offering-from-aicpa-and-aiuc), read 2026-11-08
- AICPA and CIMA: [SOC suite of services](https://www.aicpa-cima.com/resources/landing/system-and-organization-controls-soc-suite-of-services), read 2026-11-08
- ISO: [ISO/IEC 42001:2023 Artificial intelligence: Management system](https://www.iso.org/standard/42001), read 2026-11-08
- COPC Inc.: [Key updates at a glance: CX Standard Release 8.0](https://www.copc.com/copc-standards/r8-whats-new/), read 2026-11-08
- Forrester: [Total Economic Impact methodology policy](https://www.forrester.com/policies/tei/), read 2026-11-08
- Forrester: [A sample commissioned Total Economic Impact study (July 2022)](https://tei.forrester.com/go/forrester/tei), read 2026-11-08
- Forrester: [Forrester Wave methodology](https://www.forrester.com/policies/forrester-wave-methodology/), read 2026-11-08
- G2: [Community Guidelines](https://legal.g2.com/community-guidelines), read 2026-11-08
- Zendesk: [Zendesk QA](https://zendesk.com/service/quality-assurance/), read 2026-11-08
- Observe.AI: [Interaction Intelligence and Auto QA](https://observe.ai/platform/interaction-intelligence), read 2026-11-08
- Braintrust: [Evaluate](https://docs.braintrust.dev/evaluate), read 2026-11-08
- LangChain (LangSmith): [Evaluation concepts](https://docs.langchain.com/langsmith/evaluation-concepts), read 2026-11-08
- Langfuse: [Evaluation overview](https://langfuse.com/docs/evaluation/overview), read 2026-11-08
- Arize: [Evaluators](https://arize.com/docs/ax/evaluate/evaluators), read 2026-11-08
- Intercom: [Fin AI Agent outcomes](https://www.intercom.com/help/en/articles/8205718-fin-ai-agent-outcomes), read 2026-11-08
- Zendesk: [About automated resolution tiers](https://support.zendesk.com/hc/en-us/articles/9570369117338-About-automated-resolution-tiers), read 2026-11-08
- Cambridge University Press: [Trustworthy Online Controlled Experiments (Kohavi, Tang and Xu, 2020)](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B518F59), read 2026-11-08
