Pre-purchase AI agent evaluations vs. production verification
A pre-purchase evaluation tests AI agents on a fixed, usually private, set of tasks in a controlled environment, so buyers can compare products before choosing one. Production verification measures what one deployed agent actually did for a customer, in that customer's own records, after the fact.
What the evaluations say about themselves#
G2 Agent Evaluations (in beta; documentation updated 8 September 2026). G2 configures each product as a customer would and runs it in a simulated support company it builds, with its own policies, customer and order data and 38 business tools, against a private set of 50 tasks. Scores combine a task success score with LLM-judged accuracy, policy compliance and relevance, and a deterministic completeness check. Results appear on leaderboards and are refreshed several times a year. Evaluations are free to vendors during the beta, and G2 says it plans paid evaluations. G2 states that results reflect a controlled environment and may not match every real deployment (G2 documentation, methodology).
Vals AI builds benchmarks for professional work (law, finance, coding, tax and others) with privately held test sets, and publishes scores with an update date per benchmark (Vals AI, benchmarks). No customer-support or sales benchmark appeared on its benchmarks page when it was read.
Side by side#
What an evaluation cannot show#
By their own description, these evaluations test a configured copy on the evaluator's tasks. They do not observe how the agent does on a buyer's own customers and integrations, whether outcomes held after the conversation, or whether a person had to finish the work. A dated test result also says nothing about the weeks after the vendor changes the product.
Using both#
Use evaluations to build a shortlist: production verification starts only after deployment, and a vendor's published score describes the customers that vendor invited, not yours. Run a pilot on your own traffic. Verify in production once you deploy (methodology).
Questions
Should I use an evaluation or a verification?
- Both, at different times. An evaluation helps choose a shortlist before you buy. Verification measures what the chosen agent does with your customers after you deploy it.
Do evaluations use real customer data?
- G2's customer experience evaluation runs in a simulated company G2 builds, with its own policies, customer and order data and business tools. Vals AI keeps its test sets private.