# AI agent benchmarks vs. production verification

> Public agent benchmarks such as tau-bench, tau2-bench and CRMArena-Pro test models and agent setups on simulated users and sandboxes. They do not measure a vendor's deployment.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/compare/agent-benchmarks

Public agent benchmarks test language models and agent setups on fixed tasks with simulated users in sandboxed environments, so researchers can compare models under identical conditions. They do not measure any vendor's deployed agent, and a benchmark score says little about whether a particular agent's claimed outcomes held up for a particular company's customers.

## The benchmarks, from their papers

| Benchmark | What it tests | How success is judged | Source |
|---|---|---|---|
| tau-bench | Conversations between an LLM-simulated user and an agent with domain tools and policies (airline, retail) | The final database state against an annotated goal; introduces pass^k for reliability over repeated trials | [Yao et al., 17 June 2024](https://arxiv.org/abs/2406.12045). The repository says its tasks are no longer updated and points to a successor ([Sierra Research](https://github.com/sierra-research/tau-bench)) |
| tau2-bench | A dual-control setting where agent and user both act through tools (telecom) | Task completion in the shared environment | [Barres et al., 9 June 2025](https://arxiv.org/abs/2506.07982) |
| CRMArena-Pro | LLM agents on 19 expert-validated tasks across sales, service and configure-price-quote, B2B and B2C, multi-turn | Task success, plus confidentiality awareness | [Huang et al., 24 May 2025](https://arxiv.org/abs/2505.18878) |

## Side by side

| | Public benchmark | Production verification (Attify) |
|---|---|---|
| Subject | A model or agent setup | One vendor's agent at one customer (and pooled across its customers) |
| Users | Simulated by a model | The customer's real customers |
| Tasks | Public, fixed | Whatever customers actually asked |
| Outcome | Goal state reached in the sandbox | The claimed outcome held in the customer's records: no repeat contact or handoff for support; for meetings and appointments, booked, or held and kept, in the unit the vendor claims |
| Who runs it | Researchers; anyone can rerun it | Attify, under a published standard, on the customer's own records, or a vendor's stream the customer reconciled |

## What a benchmark cannot show

Tasks are public, so a system can be tuned to them. The simulated user is a model, not a customer. And the benchmark compares models or scaffolds, not the configured product a vendor sells, with its integrations and policies.

## Using both

Benchmarks are useful for developers choosing a model and for tracking reliability over repeated trials. Production verification has limits of its own: it measures one deployed agent, only after deployment, and a vendor's score describes the customers that vendor invited. For a buying decision, pair them with an evaluation of the product and a [pilot](https://attify.app/guides/evaluate-ai-agent-pilot), then verify in production.

## Questions

### What is pass^k?

A reliability metric introduced with tau-bench: the chance an agent succeeds on all of k repeated trials of the same task. An agent that solves a task once may fail it the next time.

### Is there a benchmark of audited outcomes by job type?

The Attify Index is designed to be one: pooled audited outcomes per job type from contributing buyers' own systems, naming no company. An edition publishes only with at least five contributing companies and none holding more than half of the claims.

## Sources

- arXiv: [tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao, Shinn, Razavi, Narasimhan)](https://arxiv.org/abs/2406.12045), read 2026-11-08
- Sierra Research: [tau-bench repository](https://github.com/sierra-research/tau-bench), read 2026-11-08
- arXiv: [tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al.)](https://arxiv.org/abs/2506.07982), read 2026-11-08
- arXiv: [CRMArena-Pro (Huang et al.)](https://arxiv.org/abs/2505.18878), read 2026-11-08
