Measure · 7 min read

Measuring a chatbot

Chatbot dashboards are full of numbers that look good and mean little. This guide explains the measures that show whether a bot is actually helping customers and your team.

Updated 15 November 2026 · Sources at the end

In short

  • Measure problems solved (resolution), not just chats that did not reach a person (containment).
  • Watch the handover rate and read why conversations hand over.
  • Ask customers, and count repeat contacts about the same thing.
  • Note your starting point before launch, or you will not know if it worked.

Resolution, containment and deflection

These three words are often used as if they meant the same thing. They do not.

  • Containment: conversations that ended without reaching a person. It says nothing about whether the customer got what they needed.
  • Deflection: contacts you think the bot prevented. It needs an estimate of what would have happened without it.
  • Resolution: conversations where the customer’s problem was actually solved. This is the one that matters.

Watch how platforms define resolution, especially if you pay for it. Intercom’s Fin counts an “assumed resolution” when the customer does not reply for 24 hours after an answer (opens in a new tab), and charges per outcome. A customer who gave up looks the same as one who was helped.

Know where you start

Self-service is harder than it looks. In Gartner’s global research, only 14% of customer service issues were fully resolved in self-service (opens in a new tab). Before launch, note how many contacts you get each week, how long replies take, and the top questions. Those are the numbers the bot should move.

The measures worth watching

Five measures for a small business bot
MeasureHow to get itWhat it tells you
Resolution rateAsk “Did that answer your question?” and count yesWhether the bot is helping
Handover rate and reasonsCount handovers and tag whyGaps in answers, or jobs a bot should not do
Repeat contactsSame customer, same topic, within 7 daysAnswers that looked right but were not
Missed questionsQuestions the bot could not matchWhat to write next
Cost per resolved conversationMonthly running cost divided by resolutionsWhether it pays for itself

Read the conversations

Numbers show where to look; conversations show what is wrong. Read a sample every week for the first month, then every month. Look for wrong answers, awkward wording, people repeating themselves and handovers that came too late.

Most businesses already check their AI. 81% of UK businesses using AI apply at least some human checking to its output (opens in a new tab). For a bot, that checking is the monthly review.

Does it pay for itself?

Returns are not automatic. Gartner reported in July 2026 that only 24% of customer service leaders it surveyed showed positive financial returns from AI (opens in a new tab). Work out the time the bot saves, what that time is worth, and what the bot costs to run and look after. The ROI calculator does this from your own numbers, and the designer’s measure step records your targets in the plan.

Questions people ask

What is a good resolution rate?

There is no reliable independent benchmark, and vendors define resolution differently. Set your own target from your starting point and the jobs the bot does, and improve it month by month.

Is a high handover rate bad?

Not necessarily. A bot that hands over complaints and sensitive matters quickly is doing its job. Look at the reasons: handovers because an answer was missing can be fixed.

How often should we review the bot?

Weekly for the first month, then monthly, and after every change to prices, policies or the bot itself.

Design your bot in an afternoon

Free, no sign-up. Pick its jobs, write its answers, set the handoff, then test it in your browser and export the plan.