# AI agent pilots and holdout tests vs. standing verification

> A pilot with a control group shows whether an AI agent beats your current process for a few weeks. Standing verification keeps measuring its outcomes after the pilot ends.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/compare/pilots-and-holdouts

A pilot with a randomized control group is the strongest way to learn whether an AI agent beats your current process: same period, same kind of traffic, one difference. It answers that question for the pilot's weeks and configuration; standing verification keeps measuring the deployed agent's outcomes, month after month, under the same published rule.

## What makes a pilot trustworthy

Controlled experiments have an established method: choose the metric before starting, randomize by the right unit (the customer, not the message), check that the split is balanced, and watch for carryover between groups ([Kohavi, Tang and Xu, 2020](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B518F59)). For AI agents, add one thing: let outcomes settle (7 days for support, a week after a meeting, 48 hours after an appointment) before counting. Steps: [how to run an AI agent pilot](https://attify.app/guides/evaluate-ai-agent-pilot).

## Side by side

| | Pilot or holdout | Standing verification (Attify) |
|---|---|---|
| Question | Is it better than what we do now? | What share of its claimed outcomes hold up, and is that changing? |
| Period | Weeks | Continuous |
| Rule | Yours, set before the pilot | A published standard, the same for every company |
| Who checks | Your analysts | Attify, on your systems |
| Comparable with others | No | Within a job type, through the Index and published verifications |

## Using both

Run the pilot on the same rule you will use in production, so the pilot result and the first months of verification read the same way.

## Questions

### Why measure after a successful pilot?

Pilots are short, may run on a configuration tuned for them, and end. Products, prompts, traffic and contact mix change after it. A standing measure catches drift.

### Can Attify measure my pilot?

Yes: connect your systems before the pilot starts, and the audited share, failure reasons and cost per audited outcome are measured under the published standard from day one.

## Sources

- Cambridge University Press: [Trustworthy Online Controlled Experiments (Kohavi, Tang and Xu, 2020)](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B518F59), read 2026-11-08
