Find false positives in your message feedback and conversation logs. Edit the relevant intent to avoid that false positive happening again. Create a new intent, or edit an existing one, to address the customer's actual question.
What is a false positive?
A false positive occurs when the chatbot matches the customer's query with an incorrect intent, resulting in the customer receiving the wrong answer to their question.
Finding false positives
Managing false positives has to be a manual process. If the chatbot could identify something as not being a good match, it wouldn't have matched it in the first place.
The Conversations page in Chatbot Studio lets you review all conversations conducted by the chatbot on a message by message basis, and decide if there are any false positives. It is unlikely that you would have time to review every conversation, so you can apply filters to target particular conversations.
There are several ways you might identify potential false positives.
Unexpectedly common intents
Not all intents will trigger with the same likelihood. Some should only trigger in very specific circumstances. If your Analytics are showing that in intent like that is triggering often, you can use the Conversations page to examine conversations where that intent was used.
For example, in this case we weren't expecting the Published novels intent to trigger often, so we are looking at the conversations where it did trigger, to see if there are false positives.
Transferred Conversations
If you have conversations which are being transferred from the chatbot to an agent, that may be because the chatbot presented the customer with an answer that didn't fit their question. You could use the Conversations page to review all conversations that triggered the intent to transfer the conversation, and try to establish why the customer may have asked to speak to a person.
Likewise, if you have intents which gather information and then send a ticket to support, you might want to review the conversation to see if that could have been avoided.
If conversations were transferred to an agent, consider asking agents to inform you when they think the customer's query should have been answered by the chatbot. Review those conversations in the Conversations page. This can help identify both unanswered questions and false positives.
Conversation feedback
If you have an intent that uses the Feedback action, customers may be giving you valuable information about which conversations contain false positives. In the Conversations page, use the Feedback drop-down list to find conversations with low ratings. Conversations with false positives are less likely to receive a perfect rating.
Message feedback
If you are using the dedicated chatbot widget, as opposed to the more common webchat widget, customers can vote on each reply they receive. They can click the thumbs up icon on replies they like and thumbs down on replies they don't like.
This gives you a quick way to pinpoint chatbot replies that might be unsatisfactory. You can filter these in the Conversations page by clicking the Activities drop-down list and selecting Like or Dislike. False positives are more likely to result in a dislike vote.
Keep in mind a dislike doesn't necessarily mean there was a false positive, the customer just may not have liked the answer they got!
Managing false positives
When you identify a false positive, it is important to both track that it happened, and to take steps to correct it. As the process continues, the aim it to have fewer false positives in the future.
Marking false positives
Understanding how common false positives are for your bot is an important metric for how well your bot's training is going. When you identify a false positive, it is important to flag the result as such, so that you can track their frequency in your analytics.
When you identify a false positive reply in a conversation:
- Click the Mark bot reply as False Positive link under the reply.
The reply will be marked with the false positive icon . This will now be counted as a false positive in the Analytics page.
On the Analytics page, the Messages graph will show the percentage of messages you have marked as false positives (among other information).
Preventing similar false positives
Once a false positive has occurred, it is not possible to fix that occasion of it. But having identified the false positive we can prevent the same question generating the same wrong response in the future. We can do this by one or both of:
- Preventing the question matching with the wrong intent in the future.
- Finding or creating an intent that correctly answers the question and matching with that.
Preventing future matches
Perhaps the question matched with an intent that would have been correct except that it was asked outside office hours, or because the customer had asked a previous question which made this reply incorrect.
You can prevent an intent matching to a query by adding necessary conditions.
Go to the intent that was incorrectly matched:
- Click Add necessary condition.
- In the Select rule dialog, choose the rule you want to apply.
- Complete any required details for that necessary condition
- Make any other changes to the intent as required
- Click the Save button.
In this example, the customer now will not match the intent if they have visited the "Stories" intent.
For a fuller explanation see How do Necessary Conditions work?
Linking to an intent with the correct reply
While using necessary conditions to prevent false positives is good, converting a false positive to an unanswered question is not as good as providing the correct answer.
If you have an existing intent which answers the question, edit the intent and add the question that created the false positive as another query for that intent.
- Click the add new query link.
- Type the question that generated the false positive, perhaps editing it first.
- If you have less than ten query variants for this intent, you can click the Generate queries button
to use AI to suggest some variants of the query.
- Make any other changes needed to the intent.
- Click the Save button.
If you do not have an existing intent, you should create a new intent with the question which created a false positive as a query. For more details see What is an intent?
Reducing the impact of false positives
Careful building and training of your chatbot, including following the suggestions in this article, can reduce the number of false positives you receive. Even so, false positives will happen.
Knowing that, you should also consider how to reduce their impact when they occur. This should include:
- Clearly identifying the chatbot as a machine.
- Creating realistic expectations for the customer.
- Providing a way for customers to get their answer even if they get a false positive (transfers/ticket creation or links to related resources, for example).