How to measure your AI support agent's real resolution rate

Take the conversations your AI agent counted as resolved and check three things in your own help desk: whether the same customer contacted you again within 7 days, whether a person stepped in or the AI sent the customer elsewhere, and, on a random sample, whether the answer actually solved the request. The share of all eligible conversations that passes all three is your audited resolution rate; put it next to the vendor's rate, counted on the same conversations.

This is the method of the Attify Audited Resolution Standard (v0.11), written as steps you can follow with an export or a connected help desk.

1. Collect a whole period#

  • Every conversation that started in a 30-day period, from your help desk, not the vendor's dashboard.
  • For each: who wrote each reply (the AI, a person, an automation), the customer's id, timestamps, and the vendor's resolution flag where your help desk stores it (for example Intercom's resolution state on Fin conversations).
  • Wait until 7 days after the period ends, so every conversation has its full follow-up.

Do not filter. Whatever you leave out (a channel, a team, a week) can move the number.

2. Use the vendor's claim as it was set#

Test the vendor's own flag, exactly as set. Each vendor counts differently: Intercom counts assumed and confirmed resolutions (Intercom), and HubSpot evaluates 72 hours after the last visitor response (HubSpot); see how support AI vendors define a resolution. If your data has no flag, count a conversation as claimed when support replied, the last reply was not a redirect, and the customer sent nothing for 24 hours. That is the most generous rule a careful vendor would use.

3. Take out conversations with nothing to resolve#

Spam, conversations support never answered, incomplete threads, and conversations with no request in them (a "thanks", a greeting). The last group needs reading: estimate its share from a random sample and remove it from both the claimed and the audited count.

4. Check each claim, in order#

CheckA claim fails whenWhere to look
Repeat contactThe same customer opened a new conversation, wrote again, or the ticket reopened, within 7 days of the final support replyEvery conversation in the help desk for that customer id
HandoffA person replied after the AI, or the final reply redirected the customer ("DM us", "call this number", a bare link)Who wrote each reply; the last reply's text
Not resolvedReading the conversation, the request was not resolvedA random sample of 200 of the claims that passed the first two checks

Count each failure at the first check it fails, so the reasons add up to 100% of the claims.

5. Compute the rates#

  • Claimed rate = claims ÷ eligible conversations with a request.
  • Audited rate = (claims that passed checks 1 and 2) × (share of the sample read as resolved) ÷ the same denominator.
  • Gap = claimed rate − audited rate, in percentage points.
  • Give the sample's share a 95% interval (see the audit sample size calculator).

Worked example#

Synthetic example, for the arithmetic. A store with 3,000 support conversations a month on Zendesk; its AI agent handled 2,100. After removing 100 with no request, 2,000 are eligible. The vendor's flag marks 1,400 resolved.

ConversationsShare of claims
Claimed resolved1,400100.0%
Failed: repeat contact within 7 days21015.0%
Failed: handoff14010.0%
Passed both, read in a sample of 200: 168 resolved (84.0%)1,050 × 0.84 = 882 audited63.0%
Failed: not resolved (the other 16.0% of 1,050)16812.0%

Claimed rate 1,400 ÷ 2,000 = 70.0%. Audited rate 882 ÷ 2,000 = 44.1%. Gap 25.9 points: 10.5 from repeat contacts, 7.0 from handoffs, 8.4 from conversations where the issue was not solved.

6. Read it in context#

  • Windows. Recompute with 24-hour, 3-day and 14-day repeat-contact windows. If the audited rate barely moves, the window is not driving your result.
  • Reasons. Split by contact reason (orders, billing, account access, technical). Resolution differs between reasons, so a change in your mix moves the rate.
  • People. Your own team's conversations in the same period are a useful baseline, measured the same way.

Do it with Attify#

  • Free check: upload an export at /check/support-resolution for the rules tier (repeat contact and handoff) on your own data, labelled self-reported.
  • Monthly, from your help desk: connect Intercom, Zendesk, HubSpot, Salesforce, Gorgias or Freshdesk, and the buyer dashboard gives each month's audited rate on the rules tier (repeat contact and handoff); paid plans add the reasons claims failed, the cost per audited outcome and judged audits within the plan's monthly limit.

Questions

Our AI resolution rate is 15% after two months. Is that normal?

There is no independent benchmark to compare it with yet, and the number depends on the vendor's rule and your contact reasons. Check the claims first: if most of the 15% holds up for 7 days with no handoff, the agent is doing less but doing it properly.

How can I tell customers who were helped from customers who gave up?

Silence alone cannot tell you. Look at what the same customer did next: a new conversation, an email or a reopened ticket within 7 days is a failed resolution. For the rest, read a random sample of conversations.

Customers come back by email after an AI chat. Does that count?

It should, if your help desk links the email to the same customer. Count a repeat contact on any channel the help desk holds, not just the chat thread.

Can I do this from an export instead of connecting my help desk?

Yes, for your own check. An export is self-reported data: fine for your decision, but it cannot support a vendor's audited score.

How long should I wait before measuring?

At least 7 days after the end of the period, so every conversation has had its full repeat-contact window.

Sources

  1. Attify. Attify Audited Resolution Standard v0.11. Published 9 Nov 2026. Read 9 Nov 2026.
  2. Intercom. Fin AI Agent outcomes. Updated 23 Sept 2026. Read 8 Nov 2026.
  3. HubSpot. Analyze your customer agent's performance. Updated 25 Sept 2026. Read 8 Nov 2026.