Why vendor-reported AI agent metrics differ from what happened

  • Guide
  • Published
  • Reviewed
  • By

A vendor counts its outcomes under its own rule, and the published rules count an outcome at the earliest moment they can: when the customer goes quiet, when a meeting is booked, when the agent's own grader says the call went well. Each choice is defensible on its own; together they count outcomes that later fail, which is why an audited rate checked afterwards in the customer's records comes out lower.

Seven reasons, each from how the rules are written#

  1. Counting at the earliest moment. Support resolutions are counted when the customer stops replying; meetings and appointments when they are booked. What happens next (a repeat contact, a no-show) is outside the count.
  2. Silence as success. A customer who leaves after an answer may have been helped or may have given up. Intercom counts the first case as an assumed resolution (Intercom); several vendors count after 72 hours of quiet (comparison).
  3. Short windows. Among support vendors whose documentation states a window, none waits 7 days. A customer who tries again on day four is outside a 72-hour window.
  4. Some handoffs do not count. HubSpot counts only handoffs the visitor starts; a manual assignment does not void a resolution (HubSpot). Intercom's article says a teammate joining after Fin answered can still leave it resolved.
  5. The denominator. Excluding short calls, "excused" calls or conversations with no request changes a rate without changing a single outcome.
  6. The vendor grades itself. Voice platforms' success flags and some resolution classifiers are an LLM reading the transcript against a prompt the vendor or customer wrote. That is the vendor's verdict, not independent evidence.
  7. The best customers. Case studies and headline rates can come from the strongest deployments, without the period, sample or baseline.

Forecasts are not measurements#

An analyst forecast is not a measurement, even when it is quoted as one. Gartner's June 2025 release, for example, predicted that over 40% of agentic AI projects will be canceled by the end of 2027 (Gartner, 25 June 2025). It is a prediction, labelled as one. A measurement states its rule, data, period and interval.

How to close the gap#

Test the vendor's own claim, as set, in your own systems, and ask what happened next. Attify's standards do this for support, sales meetings and bookings, and every result splits the failed claims by the first check they failed (a repeat contact, a handoff, a no-show and so on), in shares that add up.

Questions

Are vendors lying about their numbers?

This page does not say so. A number can follow the vendor's published rule and still differ from what happened: the gap comes from what the rule counts and when, which is why the rule matters more than the number.

Is a forecast like 'AI will resolve 80% of issues' a measurement?

No. A forecast is an analyst's prediction about the future. A measurement states its rule, its data, its period and its uncertainty.

Sources

  1. Intercom. Fin AI Agent outcomes. Updated 23 Sept 2026. Read 8 Nov 2026.
  2. HubSpot. Analyze your customer agent's performance. Updated 25 Sept 2026. Read 8 Nov 2026.
  3. Gartner. Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. Published 25 Jun 2025. Read 8 Nov 2026.