AI agent benchmarks vs. production verification

  • Comparison
  • Published
  • Reviewed
  • By

Public agent benchmarks test language models and agent setups on fixed tasks with simulated users in sandboxed environments, so researchers can compare models under identical conditions. They do not measure any vendor's deployed agent, and a benchmark score says little about whether a particular agent's claimed outcomes held up for a particular company's customers.

The benchmarks, from their papers#

BenchmarkWhat it testsHow success is judgedSource
tau-benchConversations between an LLM-simulated user and an agent with domain tools and policies (airline, retail)The final database state against an annotated goal; introduces pass^k for reliability over repeated trialsYao et al., 17 June 2024. The repository says its tasks are no longer updated and points to a successor (Sierra Research)
tau2-benchA dual-control setting where agent and user both act through tools (telecom)Task completion in the shared environmentBarres et al., 9 June 2025
CRMArena-ProLLM agents on 19 expert-validated tasks across sales, service and configure-price-quote, B2B and B2C, multi-turnTask success, plus confidentiality awarenessHuang et al., 24 May 2025

Side by side#

Public benchmarkProduction verification (Attify)
SubjectA model or agent setupOne vendor's agent at one customer (and pooled across its customers)
UsersSimulated by a modelThe customer's real customers
TasksPublic, fixedWhatever customers actually asked
OutcomeGoal state reached in the sandboxThe claimed outcome held in the customer's records: no repeat contact or handoff for support; for meetings and appointments, booked, or held and kept, in the unit the vendor claims
Who runs itResearchers; anyone can rerun itAttify, under a published standard, on the customer's own records, or a vendor's stream the customer reconciled

What a benchmark cannot show#

Tasks are public, so a system can be tuned to them. The simulated user is a model, not a customer. And the benchmark compares models or scaffolds, not the configured product a vendor sells, with its integrations and policies.

Using both#

Benchmarks are useful for developers choosing a model and for tracking reliability over repeated trials. Production verification has limits of its own: it measures one deployed agent, only after deployment, and a vendor's score describes the customers that vendor invited. For a buying decision, pair them with an evaluation of the product and a pilot, then verify in production.

Questions

What is pass^k?

A reliability metric introduced with tau-bench: the chance an agent succeeds on all of k repeated trials of the same task. An agent that solves a task once may fail it the next time.

Is there a benchmark of audited outcomes by job type?

The Attify Index is designed to be one: pooled audited outcomes per job type from contributing buyers' own systems, naming no company. An edition publishes only with at least five contributing companies and none holding more than half of the claims.

Sources

  1. arXiv. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao, Shinn, Razavi, Narasimhan). Published 17 Jun 2024. Read 8 Nov 2026.
  2. Sierra Research. tau-bench repository. Read 8 Nov 2026.
  3. arXiv. tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al.). Published 9 Jun 2025. Read 8 Nov 2026.
  4. arXiv. CRMArena-Pro (Huang et al.). Published 24 May 2025. Read 8 Nov 2026.