How to evaluate chatbot performance
A chatbot gets credit for leads you'd have had anyway
A customer who was always going to call you might happen to open the chat first. A dashboard that counts every lead the chatbot touched gives it credit for that customer too.
The honest way to measure it is the way medicine tests a treatment: a control group. Hold a share of visitors back from seeing the assistant, and compare how often each group gets in touch. The difference is what the assistant actually adds.
Advertisers learned this the expensive way. When Gordon and colleagues compared 15 randomized ad experiments at Facebook with the usual non-experimental estimates, the estimates often failed to match what the experiments found, even after adjusting for detailed data about each user.
There's a catch at small-business traffic. You'll look at the numbers every week, and a standard A/B test that's checked that often starts finding differences that aren't there. Results uses an always-valid confidence interval instead, a method from a 2017 paper by Johari, Koomen, Pekelis and Walsh built for tests people check whenever they like. It only says the assistant helps when that would still hold had you checked every day.
Sources: Johari, Koomen, Pekelis and Walsh, "Peeking at A/B Tests: Why it matters, and what to do about it", KDD 2017; Johari, Pekelis and Walsh, "Always Valid Inference: Bringing Sequential Analysis to A/B Testing", arXiv; Gordon, Zettelmeyer, Bhargava and Chapsky, "A Comparison of Approaches to Advertising Measurement", Marketing Science, 2019
How the measurement works
Measured against visitors who never saw it
Choose how many to hold back
Turn measurement on and hold back 10%, 20% or 30% of new visitors. They never see the chat bubble; everyone else does.
Count who gets in touch
A lead from the chat counts, and so can a contact form or a booking from your own systems, reported for either group.
Read the difference
Results shows the share of each group that got in touch and, once there's enough traffic, how many more customers the assistant brought in, with a range. Enter what a customer is worth to you and it shows that difference in money too.
What it shows
Chatbot metrics that mean something
Customers it added
An estimate of the extra customers who got in touch because the assistant was there, with the lowest and highest it could plausibly be.
What that is worth, in your numbers
Enter what a new customer is worth to you and, if you like, how many inquiries in ten become customers. Results turns the measured difference into money for the same 30 days. It leads with the low end of the range and writes out the sum, so you can check it on a calculator.
No early verdicts
It says "too early to say" until each group has at least 30 visitors. After that, its range stays honest however often you look, with a one-in-twenty chance of calling a difference that isn't there.
Where its answers came from
Every answer to a real customer in the last 30 days, split three ways: from your facts and your own answers, quoted from your pages or documents, or handed to you instead of guessed. This part is on every plan.
Tested before customers met it
How your assistant did on its last test run: answers from facts, and handoffs it should have made and did.
Your other conversions count
Report a form submission from your website, or send outcomes from your own systems with a key, so a customer who didn't use the chat still counts.
What it won't do
What Results doesn't claim
It measures carefully, which means it won't tell you more than the numbers support.
- It doesn't guess what a customer is worth. There's no money figure until you enter your own number and the verdict is clear, and the figure covers the 30 days measured, not a forecast for the year. It doesn't subtract what the assistant costs.
- It needs traffic. A site with a handful of visitors a month won't get a verdict for a long time.
- Measurement is off until you turn it on, and holding visitors back means they don't see the assistant.
- It measures the chat widget on your website, not the shareable chat page.
Questions
Chatbot analytics questions
How do you evaluate chatbot performance?
Compare what happens with the chatbot against what happens without it. Ours holds back a share of visitors and compares how often each group gets in touch.
How do you calculate chatbot ROI?
Work out the extra customers it brings in, multiply by what a customer is worth to you, and subtract its cost. Results does the first two once you enter what a customer is worth: it shows the low end of the measured range in money, with the sum written out. The cost is on the pricing page.
What chatbot metrics should you track?
The ones that show value: extra customers compared with visitors who didn't see it, how many of its answers were backed by your facts, and how many questions it had to hand over.
Why hold visitors back?
Without a group that didn't see the assistant there's nothing to compare against, and every lead it touched looks like its doing.
How long before I see a result?
It depends on your traffic. It needs at least 30 visitors in each group before it gives any verdict, and small differences take longer to confirm.
Keep exploring
Other features
- Answers from your websiteReplies from your own pages and documents, and says when it doesn't know.
- Lead alertsNew leads emailed to you in seconds, with where they came from.
- BookingsYour booking page or Google Calendar, right in the chat.
- Handoff to youPasses the conversation to you when a person should answer.
- Test and correctCheck its answers before customers see them.
- Chat widget and pageA chat bubble for your site and a link to share anywhere.