1 of 41

UTBI^* Support chatbot

Team Irish Talking Tree

Yu Shen Lu

Yan He

Haozhe Huang

Chen Cui

1

* IRDG-BI

2 of 41

What is UTBI?

University of Toronto Business Intelligence

Helps UofT departments make data-driven decisions

UTBI provides access to a data analytics platform -> Tableau

Offers technical support to ~1500 Tableau users

2

3 of 41

Problem description

“On average I spend 10 hours a week helping people with email and chat requests”

“70-80% of Tableau questions can be resolved using online resources”

Goals:

Help UTBI staff to save time
Make it easier for UTBI's users to get answers

3

4 of 41

Our Solution

4

Projected savings of

over $10,000 per year

~63%

User requests

Projected to resolve

5 of 41

5

Privacy is our priority

Generative models can behave in unpredictable ways

They might synthesize answers that leak sensitive information

6 of 41

6

User Request Statistics

From 1800 emails

(~250 requests)

7 of 41

Solution Pipeline

7

Initial Filtering

Quick Answer

User

Request

Redirect to human

Resolved

8 of 41

8

9 of 41

Solution Pipeline

9

Initial

Filtering

UTBI

Knowledge Base

Search

Quick Answer

Resolved

User

Request

Redirect to human

Resolved

10 of 41

UTBI Knowledge Base Search

10

How to check Tableau version

How to connect to VPN

How to use VPN on mobile

Where to submit request form

Answer

How to set up remote access on my phone?

Microsoft QnA Maker

User Request

How do I set up a server

Answer

11 of 41

Building the UTBI Knowledge Base

UTBI Tableau FAQ
Tableau "interview questions"
Instructions for various admin requests

Q

A

12 of 41

Solution Pipeline

12

Initial

Filtering

UTBI

Knowledge Base Search

Tableau

Site

Search

Quick Answer

Resolved

User

Request

13 of 41

On-demand search

13

14 of 41

Solution Pipeline

14

Initial

Filtering

UTBI

Knowledge Base Search

Tableau

Site

Search

Redirect to human

Quick Answer

Resolved

User

Request

15 of 41

Logging user responses

15

User ID	Timestamp	Question	Module	Helpful?
0135	2020-11-30 11:12:04	why can't I access the tableau server?	QnA	Yes
0246	2020-11-30 11:16:08	how do I create dual axis chart?	QnA	No
0246	2020-11-30 11:16:17	how do I create dual axis chart?	Site Search	Yes
0187	2020-11-30 11:20:42	This error message keeps popping up…	Site Search	No

Did I help with your question?

👍

👎

16 of 41

DEMO

17 of 41

17

Performance

Affordability

Classification-based Solution

Human Agent

Our Solution

18 of 41

Conclusion

18

~63%

…requests automated

>$10,000

Savings at a cost of $1200

Estimated

19 of 41

Appendix

19

20 of 41

Problem description – UTBI users

- Users struggle to locate the resources they need

Support team already overwhelmed by the repetitive requests
No dedicated support agent
Real-time assistance not available to everyone

20

Tina

Our proposed solution is an automated chatbot that can respond to both administrative and technical inquiries, such as 'how do I set up the vpn'

With a chatbot, we aim to reduce the time that UTBI spends answering common requests through email and chat

Another objective of the chatbot is to provide informative answers in a friendly manner, to ensure good user experience.

UTBI's users need resources from all over the place (multiple levels of ticketing system, FAQ, online resources

Customers and employees expect to easily get what they need, whenever they need it. But they don’t have the patience to navigate portals and they struggle to find what they need in online documentation, requiring them to request assistance. In the end, it’s not a good experience for anyone because your human agents are continuously overloaded, answering repetitive and routine requests. And your employees and customers are frustrated because they have to wait for a response to their service request. It’s become increasingly clear that businesses must embrace intelligent automation to drive new levels of operational efficiency while simultaneously improving employee and customer satisfaction

21 of 41

Percentage of Test Questions Resolved

21

40%

63%

22 of 41

Comparison with alternative product

22

	Our chatbot
Method	Question-answer matching + On-demand search + human agent	Question-answer matching + human agent
Pricing	Price based on server & usage Approx. $1200 per year (base on current request volume)	Proprietary service with unknown pricing
Knowledge Base	Maintained by UTBI	Maintained by ServiceNow

23 of 41

Looking Ahead

Chatbot will be deployed by client

After deployment, UTBI can collect feedback from users

UTBI will maintain and update the knowledge base

23

24 of 41

Data Preprocessing and Augmentation

Anonymized email and removed email-specific format from data

We only have <250 email chains
Augment data with

Back Translation
Bert-based contextual word insertion

We have ~700 emails after augmentation

24

25 of 41

First attempt: classification

25

Intent classifier

Redirect to human

Resolved

User

Request

26 of 41

Simple Models

Linear classification

Naïve Bayes
Linear Regression

Rule-based classification

Keyword Classifier

26

27 of 41

Universal Sentence Encoder (Cer et al. 2018)

27

Converts sentences to vectors
Captures contextual meaning in sentences

Classification

User queries

Universal

Sentence

Encoder

Sentence Vectors

28 of 41

On-demand search

Because Tableau documentation is extremely large and constantly updating

28

29 of 41

Performance Evaluation

29

30 Evaluation Questions

Rephrased Email Requests

30 of 41

Quick access buttons

30

31 of 41

Building the QnA Maker Knowledge Base

31

UTBI FAQ Page

Tableau concepts ("interview questions")

Instructions for admin related questions

32 of 41

UTBI Knowledge Base Search

32

33 of 41

On-demand search

Because Tableau documentation is extremely large and constantly updating

33

34 of 41

On-demand search

Tableau vs Google

34

35 of 41

Redirect to support team

35

36 of 41

Performance Evaluation after Deployment

User statistics

Chatbot usage
Percentage of questions resolved
Percentage of returning users

Updating QnA Maker knowledge base

36

37 of 41

Formulating answers for chatbot is difficult

We want the chatbot to display pre-written answers if possible

Generative models trained on user questions may synthesize answers that can leak sensitive information

37

38 of 41

Limitations

User input should be formulated in the format of a single question

Unable to process background information
Unable to parse multiple messages
Unable to process attachments

38

39 of 41

QnA Maker [1]

Information Retrieval

Word level semantic matching
Sentence level semantic matching
TF-IDF to disambiguate UTBI specific data
Gradient-boosted decision tree

Other features

Chit chat
Active learning

39

40 of 41

Definition of F1 Score

40

41 of 41

41