Research: other ways to verify an AI agent

  • Research
  • Published
  • Reviewed
  • By

Every other way of checking an AI agent answers a question of its own: benchmarks and pre-purchase evaluations test a simulated or controlled copy, certifications test controls, ROI studies model a composite company, analyst reports and reviews collect vendor-supplied evidence and opinion, and QA tools, evaluation platforms, vendor dashboards and pilots score real traffic under a rule the vendor or the buyer writes. Production verification checks the outcomes an agent claimed, continuously, in the buyer's own records, against a published standard that neither the vendor nor the buyer sets.

Facts below come from each method's own documentation, read on 8 November 2026; a date in a source link is the date the page shows. Conclusions of Attify's own are labelled as such.

At a glance#

Read on 8 November 2026.

MethodWhat it measuresStageWhose dataWho sets what countsPublicRefreshSource
Public benchmarks (tau-bench, tau2-bench, CRMArena-Pro)Task success of a model or agent setup with simulated users in sandboxes; tau-bench judges success by the final database state and adds pass^k, a reliability measure over repeated trialsBefore purchasePublic, simulatedThe researchersYesWhen authors publish a new version; the tau-bench repository says its tasks are no longer updatedtau-bench, 17 Jun 2024, repository, tau2-bench, 9 Jun 2025, CRMArena-Pro, 24 May 2025
G2 Agent EvaluationsTask success in a simulated support company (38 business tools, a private set of 50 tasks), plus accuracy, policy compliance and relevance (LLM judges) and completeness (a deterministic check)Before purchaseG2's simulated company and tasksG2Leaderboards, separate from G2 reviewsSeveral times a year, or after a material product changeG2, updated 8 Sep 2026, methodology
Vals AIModel and agent accuracy on tasks in finance, legal, coding, web search and public benefits; no support or sales benchmark was listed when readBefore purchasePrivately held test setsVals AIScores yes, test sets noA date per benchmarkAbout, Benchmarks
AIUC-1Security, safety and reliability controls in six domains, and adversarial tests: jailbreaks, prompt injection, data leakage, hallucinations, unsafe tool callsBefore and during operationAIUC's scenarios run against the agent; an accredited auditor's review of controlsThe standard and the auditorWhether the audit report is published is not statedStandard quarterly; retests quarterly; certificate valid one yearAIUC-1, updated 15 Oct 2026
SOC 2Design and operating effectiveness of a service's controls for security, availability, processing integrity, confidentiality or privacyService in operationThe organization's controlsA licensed CPA, under AICPA attestation standardsNot stated on the pages readRenewed yearlyAICPA, AICPA and AIUC, 19 Oct 2026
ISO/IEC 42001Requirements for an AI management system within an organizationOrganization in operationThe organization's recordsThe standardNot confirmedNot confirmedISO
COPC CX Standard, Release 8.0Management requirements for customer-experience operations, with the same performance standards whether a human or technology serves the customer; technology that makes frequent errors must be taken out of transactions until fixedIn production, organization-wideThe organization's operationsCOPCNot statedNot statedCOPC
Forrester Total Economic ImpactModeled financial impact on a composite organization built from customer interviews; studies for technology companies are commissioned and marked soBefore or after purchaseInterviews, then a modelForrester, which keeps editorial controlThe study documentOne study at a timeForrester, sample study, 2022
Forrester WaveVendor scores from a questionnaire, briefings, demos and reference customers, against the analyst's criteria and weightsBefore purchaseVendor-supplied, and references the vendor namesThe analystThe reportNot statedForrester
G2 reviewsUser opinion, filtered automatically and checked by hand; incentivized reviews labelled; the experience described must be from the past two yearsAfter purchaseReviewers' own experienceReviewersYesAs reviews arriveG2
Conversation QA (Zendesk QA, Observe.AI)Every conversation scored against built-in categories and categories the customer writes; Zendesk QA also reviews AI agent conversations and flags unresolved onesIn productionThe customer's own conversationsThe customer, through the rubricNoContinuousZendesk QA, updated 21 Sep 2026, Observe.AI
Evaluation and observability platforms (Braintrust, LangSmith, Langfuse, Arize AX)Evaluator scores (LLM judges, code checks, human review) on datasets before release and on production traces; online scores have no ground truth and rest on the judgeBefore release and in productionThe builder's datasets and tracesThe team that builds or runs the agentNoContinuousBraintrust, LangSmith, Langfuse, Arize
Vendor dashboards (examples: Intercom, Zendesk)Outcomes under the vendor's own definition, which at Intercom and Zendesk also sets the billIn productionThe vendor's platformThe vendorNoContinuousIntercom, updated 23 Sep 2026, Zendesk
Pilots and holdout testsThe agent against a control group handled the old way, as a randomized controlled experimentIn production, for weeksThe buyer's own trafficThe buyerNoOnce per pilotKohavi, Tang and Xu, 2020
Production verification (Attify)The outcomes an agent claimed, checked against a published standard per job type, with 95% intervalsIn production, continuousThe customer's own systems (connected), or the vendor's stream that the customer consents to and reconciles (witnessed)A published standardA vendor's scores once it publishes its first (the first audit is private), and the Attify IndexRules tier daily; a judged score every 30 daysMethodology

What each cannot tell a buyer#

This section is Attify's own reading of the documentation above.

  • Benchmarks compare models or agent setups on fixed tasks with a simulated user. Public tasks can be tuned to, which is one reason G2 and Vals AI keep theirs private.
  • Pre-purchase evaluations are a dated snapshot of a configured test copy. G2 states that its results reflect a controlled environment and may not match every real deployment.
  • Certifications and attestations test controls. None of the pages read says it measures whether a customer's issue was solved or a meeting was held, and a safe agent can still fail the job.
  • ROI studies project results for a composite company. Forrester's sample study says it makes no assumption about the return other organizations will get and tells readers to use their own estimates.
  • Analyst reports and reviews rest on vendor-supplied answers, demos, references the vendor names, and opinions of people who chose to write. Reviews have no denominator, so they give no rate.
  • QA tools and evaluation platforms score against a rubric or evaluator written by whoever runs the tool, often the vendor. A conversation scored as good can still come back as a repeat contact, and outcomes outside the trace (a later contact in the help desk, an attended meeting) appear only if someone wires them in.
  • Vendor dashboards are the vendor reporting on itself under its own rule.
  • Pilots show whether the agent beat the old process during the pilot, on a configuration the vendor may tune for it. Nobody outside the buyer checks the result, and results do not compare across buyers.
  • Production verification (Attify) describes only the customers the vendor chose to invite, and a vendor's first audit stays private unless it publishes. It sees only outcomes recorded in the systems a customer connected, does not measure satisfaction or quality beyond the standard (tone, compliance with a script), and covers only job types with a published standard (methodology).

Which method answers which buyer question is in ways to verify an AI agent, compared, with one page per method.

Not read or not confirmed#

  • Gartner's Magic Quadrant and Peer Insights method pages refused automated reading; nothing here relies on search summaries of them.
  • ISO/IEC 42001's audit stages and cadence. ISO's page refused automated reading, and secondary summaries are not used.
  • SOC 2 Type 1 and Type 2: the AICPA pages read do not define them.
  • COPC's certification process, cost and listing, and when certification to Release 8.0 begins.
  • Who pays for AIUC-1, COPC and Vals AI: not stated on the pages read. G2 says its evaluations are free to vendors during the beta and that it plans paid evaluations.
  • Prices of the QA tools and evaluation platforms.
  • Whether each of G2's evaluation categories has a published methodology.
  • Other independent, vendor-by-vendor evaluation programs for support or sales agents. None turned up in this research, which does not mean none exists.

Questions

What is the difference between a SOC 2 report, a model benchmark and an outcome verification?

A SOC 2 report is a licensed CPA's opinion on whether a service's controls for security and related criteria are designed and operate well. A benchmark scores a model or agent on fixed, usually simulated tasks. An outcome verification checks, in the buyer's own records and under a published rule, what share of the outcomes an agent claimed held up.

Does tau-bench predict how an AI agent performs on my real tickets?

Not directly. tau-bench scores agents against LLM-simulated users in airline and retail sandboxes, and its repository says the tasks are no longer updated. Your customers, policies and integrations are not in it.

Which analyst firms publish measured results for AI customer service agents?

None of the analyst methods read for this research measures outcome rates in a buyer's deployment. A Forrester Wave scores vendors from a questionnaire, briefings, demos and reference customers; Gartner's method pages refused automated reading and are not described here.

Can I trust G2 ratings of AI agents for performance?

G2 moderates reviews, labels incentivized ones and bars the product's employees and direct competitors from reviewing it. Reviews are the opinions of people who chose to write one, with no denominator, so they cannot give an outcome rate or an interval.

Sources

  1. arXiv. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao, Shinn, Razavi, Narasimhan). Published 17 Jun 2024. Read 8 Nov 2026.
  2. Sierra Research. tau-bench repository. Read 8 Nov 2026.
  3. arXiv. tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres, Dong, Ray, Si, Narasimhan). Published 9 Jun 2025. Read 8 Nov 2026.
  4. arXiv. CRMArena-Pro (Huang et al.). Published 24 May 2025. Read 8 Nov 2026.
  5. G2. Agent Evaluations. Updated 8 Sept 2026. Read 8 Nov 2026.
  6. G2. Customer Experience AI Agent Evaluation methodology. Read 8 Nov 2026.
  7. Vals AI. About. Read 8 Nov 2026.
  8. Vals AI. Benchmarks. Read 8 Nov 2026.
  9. AIUC. AIUC-1. Updated 15 Oct 2026. Read 8 Nov 2026.
  10. AICPA and CIMA. Auditors meet urgent demand for AI assurance with new offering from AICPA and AIUC. Published 19 Oct 2026. Read 8 Nov 2026.
  11. AICPA and CIMA. SOC suite of services. Read 8 Nov 2026.
  12. ISO. ISO/IEC 42001:2023 Artificial intelligence: Management system. Read 8 Nov 2026.
  13. COPC Inc.. Key updates at a glance: CX Standard Release 8.0. Read 8 Nov 2026.
  14. Forrester. Total Economic Impact methodology policy. Read 8 Nov 2026.
  15. Forrester. A sample commissioned Total Economic Impact study (July 2022). Read 8 Nov 2026.
  16. Forrester. Forrester Wave methodology. Read 8 Nov 2026.
  17. G2. Community Guidelines. Read 8 Nov 2026.
  18. Zendesk. Zendesk QA. Published 6 Dec 2024. Read 8 Nov 2026.
  19. Observe.AI. Interaction Intelligence and Auto QA. Read 8 Nov 2026.
  20. Braintrust. Evaluate. Read 8 Nov 2026.
  21. LangChain (LangSmith). Evaluation concepts. Read 8 Nov 2026.
  22. Langfuse. Evaluation overview. Read 8 Nov 2026.
  23. Arize. Evaluators. Read 8 Nov 2026.
  24. Intercom. Fin AI Agent outcomes. Updated 23 Sept 2026. Read 8 Nov 2026.
  25. Zendesk. About automated resolution tiers. Read 8 Nov 2026.
  26. Cambridge University Press. Trustworthy Online Controlled Experiments (Kohavi, Tang and Xu, 2020). Read 8 Nov 2026.