How to evaluate chatbot performance
A chatbot gets credit for leads you'd have had anyway
A customer who was always going to call you might happen to open the chat first. A dashboard that counts every lead the chatbot touched gives it credit for that customer too.
The honest way to measure it is the way medicine tests a treatment: a control group. Hold a share of visitors back from seeing the assistant, and compare how often each group gets in touch. The difference is what the assistant actually adds.
How we measure chatbot ROI
Measured against visitors who never saw it
Choose how many to hold back
Turn measurement on and hold back 10%, 20% or 30% of new visitors. They never see the chat bubble; everyone else does.
Count who gets in touch
A lead from the chat counts, and so can a contact form or a booking from your own systems, reported for either group.
Read the difference
Results shows the share of each group that got in touch and, once there's enough traffic, how many more customers the assistant brought in, with a range.
What it shows
Chatbot metrics that mean something
Customers it added
An estimate of the extra customers who got in touch because the assistant was there, with the lowest and highest it could plausibly be.
No early verdicts
It says "too early to say" until each group has at least 30 visitors, and uses a method that stays valid however often you look.
How its answers were backed
Of its answers to real customers in the last 30 days, how many came from your facts, partly, or from nothing, and how many were handed over.
Tested before customers met it
How your assistant did on its last test run: answers from facts, and handoffs it should have made and did.
Your other conversions count
Report a form submission from your website, or send outcomes from your own systems with a key, so a customer who didn't use the chat still counts.
What it won't do
What Results doesn't claim
It measures carefully, which means it won't tell you more than the numbers support.
- It counts customers, not money. It doesn't calculate revenue or ROI in dollars.
- It needs traffic. A site with a handful of visitors a month won't get a verdict for a long time.
- Measurement is off until you turn it on, and holding visitors back means they don't see the assistant.
- It measures the chat widget on your website, not the shareable chat page.
Questions
Chatbot analytics questions
How do you evaluate chatbot performance?
Compare what happens with the chatbot against what happens without it. Ours holds back a share of visitors and compares how often each group gets in touch.
How do you calculate chatbot ROI?
Work out the extra customers it brings in, multiply by what a customer is worth to you, and subtract its cost. Results gives you the extra customers; the value of a customer is yours to supply.
What chatbot metrics should you track?
The ones that show value: extra customers compared with visitors who didn't see it, how many of its answers were backed by your facts, and how many questions it had to hand over.
Why hold visitors back?
Without a group that didn't see the assistant there's nothing to compare against, and every lead it touched looks like its doing.
How long before I see a result?
It depends on your traffic. It needs at least 30 visitors in each group before it gives any verdict, and small differences take longer to confirm.
Keep exploring
Other features
- Answers from your websiteReplies from your own pages, and says when it doesn't know.
- Lead alertsNew leads sent to your phone and inbox in seconds.
- BookingsYour booking page or Google Calendar, right in the chat.
- Handoff to youPasses the conversation to you when a person should answer.
- Test and correctCheck its answers before customers see them.
- Chat widget and pageA chat bubble for your site and a link to share anywhere.