In short
- Measure problems solved (resolution), not just chats that did not reach a person (containment).
- Watch the handover rate and read why conversations hand over.
- Ask customers, and count repeat contacts about the same thing.
- Note your starting point before launch, or you will not know if it worked.
Resolution, containment and deflection
These three words are often used as if they meant the same thing. They do not.
- Containment: conversations that ended without reaching a person. It says nothing about whether the customer got what they needed.
- Deflection: contacts you think the bot prevented. It needs an estimate of what would have happened without it.
- Resolution: conversations where the customer’s problem was actually solved. This is the one that matters.
Watch how platforms define resolution, especially if you pay for it. Intercom’s Fin counts an “assumed resolution” when the customer does not reply for 24 hours after an answer (opens in a new tab), and charges per outcome. A customer who gave up looks the same as one who was helped.
Know where you start
Self-service is harder than it looks. In Gartner’s global research, only 14% of customer service issues were fully resolved in self-service (opens in a new tab). Before launch, note how many contacts you get each week, how long replies take, and the top questions. Those are the numbers the bot should move.
The measures worth watching
| Measure | How to get it | What it tells you |
|---|---|---|
| Resolution rate | Ask “Did that answer your question?” and count yes | Whether the bot is helping |
| Handover rate and reasons | Count handovers and tag why | Gaps in answers, or jobs a bot should not do |
| Repeat contacts | Same customer, same topic, within 7 days | Answers that looked right but were not |
| Missed questions | Questions the bot could not match | What to write next |
| Cost per resolved conversation | Monthly running cost divided by resolutions | Whether it pays for itself |
Read the conversations
Numbers show where to look; conversations show what is wrong. Read a sample every week for the first month, then every month. Look for wrong answers, awkward wording, people repeating themselves and handovers that came too late.
Most businesses already check their AI. 81% of UK businesses using AI apply at least some human checking to its output (opens in a new tab). For a bot, that checking is the monthly review.
Does it pay for itself?
Returns are not automatic. Gartner reported in July 2026 that only 24% of customer service leaders it surveyed showed positive financial returns from AI (opens in a new tab). Work out the time the bot saves, what that time is worth, and what the bot costs to run and look after. The ROI calculator does this from your own numbers, and the designer’s measure step records your targets in the plan.