# Why vendor-reported AI agent metrics differ from what happened

> Vendors count outcomes at the earliest moment, treat silence as success, use short windows and their own graders. Each choice is reasonable alone; together they open a gap.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/guides/why-vendor-ai-metrics-differ-from-reality

A vendor counts its outcomes under its own rule, and the published rules count an outcome at the earliest moment they can: when the customer goes quiet, when a meeting is booked, when the agent's own grader says the call went well. Each choice is defensible on its own; together they count outcomes that later fail, which is why an audited rate checked afterwards in the customer's records comes out lower.

## Seven reasons, each from how the rules are written

1. **Counting at the earliest moment.** Support resolutions are counted when the customer stops replying; meetings and appointments when they are booked. What happens next (a repeat contact, a no-show) is outside the count.
2. **Silence as success.** A customer who leaves after an answer may have been helped or may have given up. Intercom counts the first case as an assumed resolution ([Intercom](https://www.intercom.com/help/en/articles/8205718-fin-ai-agent-outcomes)); several vendors count after 72 hours of quiet ([comparison](https://attify.app/compare/support-ai-agent-outcome-definitions)).
3. **Short windows.** Among support vendors whose documentation states a window, none waits 7 days. A customer who tries again on day four is outside a 72-hour window.
4. **Some handoffs do not count.** HubSpot counts only handoffs the visitor starts; a manual assignment does not void a resolution ([HubSpot](https://knowledge.hubspot.com/customer-agent/analyze-your-customer-agents-performance)). Intercom's article says a teammate joining after Fin answered can still leave it resolved.
5. **The denominator.** Excluding short calls, "excused" calls or conversations with no request changes a rate without changing a single outcome.
6. **The vendor grades itself.** Voice platforms' success flags and some resolution classifiers are an LLM reading the transcript against a prompt the vendor or customer wrote. That is the vendor's verdict, not independent evidence.
7. **The best customers.** Case studies and headline rates can come from the strongest deployments, without the period, sample or baseline.

## Forecasts are not measurements

An analyst forecast is not a measurement, even when it is quoted as one. Gartner's June 2025 release, for example, predicted that over 40% of agentic AI projects will be canceled by the end of 2027 ([Gartner, 25 June 2025](https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027)). It is a prediction, labelled as one. A measurement states its rule, data, period and interval.

## How to close the gap

Test the vendor's own claim, as set, in your own systems, and ask what happened next. Attify's standards do this for [support](https://attify.app/standards/support-resolution), [sales meetings](https://attify.app/standards/sales-meetings) and [bookings](https://attify.app/standards/bookings), and every result splits the failed claims by the first check they failed (a repeat contact, a handoff, a no-show and so on), in shares that add up.

## Questions

### Are vendors lying about their numbers?

This page does not say so. A number can follow the vendor's published rule and still differ from what happened: the gap comes from what the rule counts and when, which is why the rule matters more than the number.

### Is a forecast like 'AI will resolve 80% of issues' a measurement?

No. A forecast is an analyst's prediction about the future. A measurement states its rule, its data, its period and its uncertainty.

## Sources

- Intercom: [Fin AI Agent outcomes](https://www.intercom.com/help/en/articles/8205718-fin-ai-agent-outcomes), read 2026-11-08
- HubSpot: [Analyze your customer agent's performance](https://knowledge.hubspot.com/customer-agent/analyze-your-customer-agents-performance), read 2026-11-08
- Gartner: [Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027](https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027), read 2026-11-08
