How to Recognise and Fix False Positives in Your Chatbot

Check your message feedback and conversation logs for false positives, edit the relevant state so the chatbot won't return a false positive and create a new state to address the customer's actual question or connect it to an existing state.

A false positive is where a chatbot misunderstands your customer's intent and gives the wrong answer to the customer's question. 

False positives happen. In your chatbot building strategy, your GetJenny customer success manager will have addressed how to build a chatbot to minimise impact from false positives or wrong answers. These would include:

1. Clearly identifying the chatbot as a machine.
2. Creating realistic expectations for the customer.
3. Providing a way for customers to get their answer even if they get a false positive (transfers/ticket creation or links to related resources, for example).

Finding False Positives

By definition, a false positive cannot be precisely identified by machine but JennyStudio has a number of ways to find and address them. 

Message Feedback

Your customer may have the option to give a thumbs up/down to each chatbot reply. This gives you a quick way to pinpoint chatbot replies that might be unsatisfactory. You can filter these in the Conversations log by choosing "Like" or "Dislike" under the "Activities" filter. Then you will get a list of conversations where negative feedback on chatbot answers have occurred and you can check them for false positives. 

Filtering messages with negative user feedback for review. Source: JennyStudio

Transferred Conversations

Check conversations that have gone through your defined transfers, like passing the conversation to a live agent or opening a ticket in your CRM. You can do this in JennyStudio. Alternatively:

Filter these conversations as they happen by asking agents to flag messages that were transferred but your agent thinks should have be answered by the chatbot. This method can be used to identify new issues as well as false positives where the chatbot gave a wrong answer and the customer wanted to talk to a human to clarify. 

Chatbot Analytics: Mark Answer as a False Positive. 

You can mark false positive messages within JennyStudio. It's useful to add this step into your chatbot maintenance process because false positives can then be tracked easily in your analytics dashboard. 

Marking a chatbot reply as a false positive and reviewing false positives over time in your AI chatbot analytics. Source: JennyStudio

Removing the Possibility of False Positives 

Your state generally covers a topic paired with a customer intent. Generally false positives happen because the topic is broadly covered but doesn't cater for specific intents. This can be addressed using "Queries" and "Necessary Conditions". 

Queries

Try to have five queries related to your state context and intent. More is better. This gives the AI varied amount of information across each of these queries to calculate a certainty score that this answer is the best one for your customer based on its similarity of the customer's question to the queries you put in. 

Setting queries for the chatbot to analyse against a customer's query for a particular answer. Source: JennyStudio 

Necessary Conditions

Use the Necessary Condition feature to ensure customers get the right answer, even if their question matches the query. For example, you can make sure the customer gets a certain answer based on:

• keywords they use
• the page they're on 
• a previous question they asked in the same conversation
• the time of day they're asking the question

The factors in Necessary Condition play a defining part to whether your chatbot will give a particular answer even if it is certain that it should answer that way based on its evaluation of the queries.

Create or Connect Existing States to Address the False Positive

Using the Queries and the Necessary Conditions, you can alter your existing states related to the topic so that the customer gets the right answer based on their context. Alternatively, you might see a need to create a new state to address the customer's question in more detail.