Towards Exploiting Background Knowledge for Building Conversation Systems

Nikita Moghe, Siddhartha Arora, Suman Banerjee, Mitesh M. Khapra

1

Indian Institute of Technology Madras

Robert Bosch Centre for Data Sciences and AI, IITM

2

Courtesy: Google Images (Cortana, Allo, Echo, Siri, Ask Jenn)

3

Deep Learning for Conversational AI

Rise of data driven systems

First Chatbot

Template/Rule based Systems

Natural Language Generation

Natural Language Understanding

Dialog State Tracking

Utterance

Response

Rules

Slot-Filling

Rules

Data

Data

​

​

Data Hungry

End to End Systems

Data

​

Weizenbaum.1966

Ritter et al., 2011; Vinyals and Le, 2015; Lowe et al., 2015

Perez-Marin and Pascual-Nieto, 2011; Shawar and Atwell, 2007b; Williams et al., 2013

Aust et al., 1995; McGlashan et al., 1992; Simpson and Eraser, 1993

Data Data Everywhere..

4

Based on Movie Scripts

​

Crawled from websites

​

60+ such datasets for dialog

~1.3M

chats

~930K

chats

~1.7B

comments

A Survey of Available Corpora for Building Data-Driven Dialogue Systems, Serban et al., arxiv 2015

Based on Human-Human Spoken Interaction

Based on Human Machine Interaction

Logos: https://ubuntuforums.org/, https://twitter.com/, https://www.reddit.com/

Dialog as a Seq2Seq Problem!

5

Human: Please suggest a movie ?

​

Bot: Sure! Check out Titanic

​

​

please suggest a movie

sure check out <UNK>

Encoder

Decoder

But...

Humans rely on background knowledge to converse

​

Vinyals et al., ICML 2015

Has this never been tried before ?

6

Not really...

Linux

Man Pages

Goal Oriented Dialog

Ubuntu Corpus

But the resources and chats are not tightly coupled!

Open Domain Dialog

Alexa Proceedings 2017

Lowe et al., 2015

Key Contribution

7

Domain specific conversation systems with alternate responses explicitly obtained from specific

background knowledge

Dataset Creation

8

The 4C’s of Dataset Creation

9

Crowdsource

Crawl

Check

Curate

9071 chats

~90K

Utterances

9278

resources

15.29

Avg words/turn

153.07

Avg words/chat

10

Movie : Spider-Man

Popular and Diverse Movie List

921 Movies

IMDb 250

​

Top Ten Movies by Genre

​

1001 Movies you must watch before you die!

Curate

11

.. spiders through

genetic manipulation. While Peter is taking photographs of Mary Jane for the school newspaper, ...

I thought the movie very engrossing. Director Sam Raimi kept the action quotient high but also emphasized the human element of the story

Crazy attention to detail.

It was too heavily reliant on light-hearted humor.

​

Movie : Spider-Man

Plot

Review

Comments

Wikipedia for Plots

Collected Reviews using IMDb Most Popular Reviews

Crawl

Curate

Official Reddit Pages for Comments

Facts

Wikipedia Infoboxes for Facts

Background Knowledge

12

​

​

​

.. spiders through

genetic manipulation. While Peter is taking photographs of Mary Jane for the school newspaper, ...

I thought the movie very engrossing. Director Sam Raimi kept the action quotient high but also emphasized the human element of the story

Crazy attention to detail.

It was too heavily reliant on light-hearted humor

Movie : Spider-Man

Plot

Review

Comments

S1(N): Which is your favourite character in this?

​

S2(C): My favorite character was played by Tobey Maguire.

Crowdsource

Crawl

Curate

Chat opening statements

Which is your favourite scene in the movie ?

​

Which is your favourite character in this ?

​

What do you think about the movie ?

9 per movie

Facts

My favorite character was played by Tobey Maguire.

13

S1(N): Which is your favourite character in this?

​

S2(C): My favorite character was played by Tobey Maguire.

​

​

.. spiders through

genetic manipulation. While Peter is taking photographs of Mary Jane for the school newspaper, ...

I thought the movie very engrossing. Director Sam Raimi kept the action quotient high but also emphasized the human element of the story

Movie : Spider-Man

Plot

Review

Comments

Crowdsource

Crawl

Curate

Facts

Crazy attention to detail.

It was too heavily reliant on light-hearted humor

My favorite character was played by Tobey Maguire.

Crowdsourcing platforms are meant for Atomic Tasks!

​

But dialog requires two people

Same worker plays the role of Speaker 1 and Speaker 2

to complete the chat

Self-Chats

Krause et al. Alexa Proceedings 2017

14

S1(N): Which is your favourite character in this?

​

S2(C): My favorite character was played by Tobey Maguire.

​

S1(N): I thought he did an excellent job as Peter Parker, I didn’t see what it was that turned him into Spider-Man though.

.. spiders through

genetic manipulation. While Peter is taking photographs of Mary Jane for the school newspaper, ...

I thought the movie very engrossing. Director Sam Raimi kept the action quotient high but also emphasized the human element of the story

Movie : Spider-Man

Plot

Review

Comments

Crowdsource

Crawl

Curate

Facts

Crazy attention to detail.

It was too heavily reliant on light-hearted humor

My favorite character was played by Tobey Maguire.

Self-Chats

Speaker 1 and Speaker 2 are played by the same person.

​

Speaker 1 is free to talk about anything

​

15

S1(N): Which is your favourite character in this?

​

S2(C): My favorite character was played by Tobey Maguire.

​

S1(N): I thought he did an excellent job as Peter Parker, I didn’t see what it was that turned him into Spider-Man though.

​

S2(P): Well this happens while Peter is taking photographs of Mary Jane for the school newspaper, one of these new spiders lands on his hand and bites him.

​

​

.. spiders through

genetic manipulation. While Peter is taking photographs of Mary Jane for the school newspaper, ...

I thought the movie very engrossing. Director Sam Raimi kept the action quotient high but also emphasized the human element of the story

Movie : Spider-Man

Plot

Review

Comments

Crowdsource

Crawl

Curate

Facts

Crazy attention to detail.

It was too heavily reliant on light-hearted humor

My favorite character was played by Tobey Maguire.

Self-Chats

Speaker 1 and Speaker 2 are played by the person.

​

Speaker 1 is free to talk about anything

​

Speaker 2 has to reply using background knowledge

Specifically, Speaker 2 selects a contiguous span of words from the resource and appends suitable words

16

S1(N): Which is your favourite character in this?

​

S2(C): My favorite character was played by Tobey Maguire.

​

S1(N): I thought he did an excellent job as Peter Parker, I didn’t see what it was that turned him into Spider-Man though.

​

S2(P): Well this happens while Peter is taking photographs of Mary Jane for the school newspaper, one of these new spiders lands on his hand and bites him.

.. spiders through

genetic manipulation. While Peter is taking photographs of Mary Jane for the school newspaper, ...

I thought the movie very engrossing. Director Sam Raimi kept the action quotient high but also emphasized the human element of the story

Movie : Spider-Man

Plot

Review

Comments

AMT workers are notorious!

Check for:

​

• Incoherence

​

• Simulated Patterns

​

• Incompleteness

​

• Digression

Crowdsource

Crawl

Check

Curate

Facts

Crazy attention to detail.

It was too heavily reliant on light-hearted humor

My favorite character was played by Tobey Maguire.

Amazon Mechanical Turk https://www.mturk.com

17

S1(N): Which is your favourite character in this?

​

S2(C): My favorite character was played by Tobey Maguire.

​

S1(N): I thought he did an excellent job as Peter Parker, I didn’t see what it was that turned him into Spider-Man though.

​

S2(P): Well this happens while Peter is taking photographs of Mary Jane for the school newspaper, one of these new spiders lands on his hand and bites him.

​

S1 (N): I see. I was very excited to see this film and it did not disappoint!

​

S2(R): I agree, I thoroughly enjoyed “Spider-Man”

​

S1(N): I loved that they stayed pretty true to the comic.

​

S2(C): Yeah, it was a really great comic book adaptation

​

S1(N): The movie is a great life lesson on balancing

power.

​

S2(F): That is my most favorite line in the movie, ‘With

great power comes great responsibility.’

​

I thoroughly enjoyed“Spider-Man” which I saw in a screening. I thought the movie very engrossing. Director Sam Raimi kept the action quotient high, but also emphasized the human element of the story. The casting was perfect. Tobey Maguire was very believable as the gawky teenager in the early part of the film and then, after his run-in with the radioactive

Peter’s science class takes a field trip

to a genetics laboratory at Columbia University. The lab works on spiders and has even managed to create new species of spiders through

genetic manipulation. While Peter is taking photographs of Mary Jane for the school newspaper, one of these new spiders lands on his hand and bites him

​

Plot

Review

Crazy attention to detail. My favorite character was played by Tobey Maguire. I can’t get over the ”I’m gonna kill you dead” line.

No spoilers, but it does start to take itself more seriously towards the finale. It was too heavily

reliant on constant light-hearted humor. How ever the constant joking around kinda was low. A really great comic book adaptation.

​

Comments

Fact Table

Movie : Spider-Man

Multi-Reference Test Set

18

S1(N): I thought he did an excellent job as Peter Parker, I didn’t see what it was that turned him into Spider-Man though.

​

S2(P): Well this happens while Peter is taking photographs of Mary Jane for the school newspaper, one of these new spiders lands on his hand and bites him.

​

S1 (N): I see. I was very excited to see this film and it did not disappoint!

​

​

I thoroughly enjoyed“Spider-Man” which I saw in a screening. I thought the movie very engrossing. Director Sam Raimi kept the action quotient high, but also emphasized the human element of the story. The casting was perfect. Tobey Maguire was very believable as the gawky teenager in the early part of the film and then, after his run-in with the radioactive

Review

S2 (R): I agree. I thoroughly enjoyed Spider-Man

S2 (R): Also, The casting was perfect

S2 (R): I think so too! Director Sam Raimi kept the action quotient high

78.04% of test set

Several Responses can be correct for a given context

But wait… Is this Natural?

​

19

500 randomly chosen chats

​

Evaluated by three in-house annotators per chat

How can self-chat be a dialog?

What if the conversation digressed ?

Does this copy-paste make any sense ?

Methods

20

21

Generation Based Models

Copy-or-Generate Models

Span Prediction Models

Hierarchical Recurrent

Encoder Decoder (HRED)

​

(No Background Knowledge)

Get to the Point (GTTP)

​

Document <-> Resource

Summary <-> Response

​

​

BiDirectional

Attention Flow (BiDAF)

​

Question <-> Context

Document <-> Resource

Serban et al. AAAI 2016; See et al. ACL 2017; Seo et al., ICLR 2017

​

Challenges

22

S1 (N): I see. I was very excited to see this film and it did not disappoint!

S2(R): I agree, I thoroughly enjoyed “Spider-Man”

S1(N): I loved that they stayed pretty true to the comic

​

S1 (N): I see. I was very excited to see this film and it did not disappoint!

S2(R): I agree, I thoroughly enjoyed “Spider-Man”

S1(N): I loved that they stayed pretty true to the comic

​

S1 (N): I see. I was very excited to see this film and it did not disappoint!

S2(R): I agree, I thoroughly enjoyed “Spider-Man”

S1(N): I loved that they stayed pretty true to the comic

​

Oracle

Mixed-Long

Mixed-Short

BiDirectional Attention Flow fails beyond 256 resource words!

Average combined resource length ~ 900 words

Results

​

23

Results

​

24

Background knowledge helps! But correct background knowledge helps better!

​

Results

​

25

Span prediction models perform better

Results

​

26

BiDAF can chose to ignore noise. GTTP has no such distinction

Results

​

27

On Multi Reference Test Set (78.04%)

​

What next?

Finding the right resource is important

• Two stage model : Predict the resource and generate

Cross attention mechanisms are useful

• Building scalable cross attention mechanisms

Generation models are more human-like

• Adding the power of cross attention mechanisms to generation models

​

​

​

28

Code and data : https://github.com/nikitacs16/Holl-E

29

Siddhartha Arora

Suman Banerjee

Mitesh M. Khapra

Microsoft Research India

(Student Travel Grant)

TextKernel

(EMNLP Student Travel Scholarship)

Thank You!

Questions/Suggestions

30

Human Evaluation of Responses

31