How to run an AI agent pilot and decide on evidence
Before the pilot starts, write down how an outcome is counted, what share must pass for a "go", and which traffic is the control. Then run the agent, measure its outcomes in your own systems after they have had time to settle, and compare with the control under the same rule.
Before it starts#
- The outcome rule. Use a published one rather than the vendor's dashboard: for support, no repeat contact within 7 days, no handoff, and a read confirmation on a sample (standard); for meetings and appointments, the checks in the sales and bookings standards.
- The pass mark. The audited share, and the cost per audited outcome, at which you would go ahead. Work it out from your own costs with the break-even calculator.
- The control. Randomly assign part of the traffic to the current process for the same period, so seasonality and mix affect both. Randomize by customer, not by message, so one customer does not see both. Controlled experiments have a well-documented method, including choosing the randomization unit and checking the split is trustworthy (Kohavi, Tang and Xu, 2020).
- The sample. Enough outcomes for the interval you need (audit sample size).
- The data. Make sure your systems record who did what: the AI's own user or integration in your help desk, CRM or booking system.
During#
- Keep the configuration stable, or log every change with its date.
- Do not let anyone pick which conversations or calls go to the AI.
- Keep the vendor's dashboard for operations, not for the decision.
After#
- Wait for outcomes to settle: 7 days after the last support conversation; a week after the last meeting date; 48 hours after the last appointment.
- Measure both arms under the same rule, in your own systems.
- Report: the audited share for each arm with 95% intervals, the difference with its interval, why claims failed, and the cost per audited outcome for each arm.
- Decide against the pass mark you wrote down. If the interval of the difference includes zero, the pilot did not show a difference.
Scorecard#
Do it with Attify#
Connect your help desk, CRM or booking system and Attify measures every month under the published standard, with the reasons claims failed and the cost per audited outcome. See for buyers.
Questions
How long should a pilot run?
- Long enough to reach the sample you need, plus the time outcomes take to settle: 7 days after the last conversation for support, the meeting's date plus a week for sales, 48 hours after the appointment for bookings. For most teams that is 30 days of traffic and a week or two of follow-up.
Should the vendor tune the agent during the pilot?
- Agree up front. If the vendor tunes during the pilot, measure only the period after the last change, and expect production to differ from the tuned pilot.
What is a good pass mark?
- One you set before seeing results, from your own economics: the audited share at which the cost per audited outcome beats what the same outcome costs you today. The break-even calculator does the arithmetic.