# Pre-purchase AI agent evaluations vs. production verification

> Evaluations such as G2 Agent Evaluations and Vals AI test agents on private tasks in a controlled environment before purchase. Production verification checks a deployed agent's real outcomes.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/compare/pre-purchase-evaluations

A pre-purchase evaluation tests AI agents on a fixed, usually private, set of tasks in a controlled environment, so buyers can compare products before choosing one. Production verification measures what one deployed agent actually did for a customer, in that customer's own records, after the fact.

## What the evaluations say about themselves

**G2 Agent Evaluations** (in beta; documentation updated 8 September 2026). G2 configures each product as a customer would and runs it in a simulated support company it builds, with its own policies, customer and order data and 38 business tools, against a private set of 50 tasks. Scores combine a task success score with LLM-judged accuracy, policy compliance and relevance, and a deterministic completeness check. Results appear on leaderboards and are refreshed several times a year. Evaluations are free to vendors during the beta, and G2 says it plans paid evaluations. G2 states that results reflect a controlled environment and may not match every real deployment ([G2 documentation](https://documentation.g2.com/docs/agent-evaluations), [methodology](https://ai.g2.com/evaluations/methodology)).

**Vals AI** builds benchmarks for professional work (law, finance, coding, tax and others) with privately held test sets, and publishes scores with an update date per benchmark ([Vals AI](https://www.vals.ai/about), [benchmarks](https://www.vals.ai/benchmarks)). No customer-support or sales benchmark appeared on its benchmarks page when it was read.

## Side by side

| | Pre-purchase evaluation | Production verification (Attify) |
|---|---|---|
| Question it answers | How does this product do on a fixed test, compared with others? | What share of this deployed agent's claimed outcomes held up? |
| When | Before purchase | After deployment, continuously |
| Data | The evaluator's tasks and simulated environment | The customer's own help desk, CRM, calendar or booking system |
| Configuration | Set up by the evaluator | The customer's real configuration, integrations and policies |
| Outcome | Task success in the test | Did it happen, did it stick, did a human have to redo it |
| Refresh | Several times a year (G2) | Daily on the rules tier; judged scores every 30 days |

## What an evaluation cannot show

By their own description, these evaluations test a configured copy on the evaluator's tasks. They do not observe how the agent does on a buyer's own customers and integrations, whether outcomes held after the conversation, or whether a person had to finish the work. A dated test result also says nothing about the weeks after the vendor changes the product.

## Using both

Use evaluations to build a shortlist: production verification starts only after deployment, and a vendor's published score describes the customers that vendor invited, not yours. Run a [pilot](https://attify.app/guides/evaluate-ai-agent-pilot) on your own traffic. Verify in production once you deploy ([methodology](https://attify.app/methodology)).

## Questions

### Should I use an evaluation or a verification?

Both, at different times. An evaluation helps choose a shortlist before you buy. Verification measures what the chosen agent does with your customers after you deploy it.

### Do evaluations use real customer data?

G2's customer experience evaluation runs in a simulated company G2 builds, with its own policies, customer and order data and business tools. Vals AI keeps its test sets private.

## Sources

- G2: [Agent Evaluations](https://documentation.g2.com/docs/agent-evaluations), read 2026-11-08
- G2: [Customer Experience AI Agent Evaluation methodology](https://ai.g2.com/evaluations/methodology), read 2026-11-08
- Vals AI: [About](https://www.vals.ai/about), read 2026-11-08
- Vals AI: [Benchmarks](https://www.vals.ai/benchmarks), read 2026-11-08
