1 of 50

Generation QA

Jordan Boyd-Graber

2 of 50

3 of 50

4 of 50

5 of 50

6 of 50

7 of 50

8 of 50

At 6 feet and 165 pounds, Iverson is the shortest and lightest player to win the MVP award.

9 of 50

10 of 50

Open Book

  • More interpretable / trustworthy
  • Less prone to hallucination
  • Can better answer long tail

Closed Book

  • Space efficient
  • Light transformations

11 of 50

Bär

Bärli

Zwingli

Zwing

12 of 50

Multihop Question 1

The Maniac then addresses the audience, showing what the scenario entails. When Feletti leaves them, the four policemen die in the resulting explosion. However, the Maniac then offers the second result: sticking to the rule of law, Feletti releases them, but is chained to the window by the policemen when they realize that Feletti knows what they did. The Maniac then leaves the audience to decide which ending they prefer.

13 of 50

Open Book

  • More interpretable / trustworthy
  • Less prone to hallucination
  • Can better answer long tail

Closed Book

  • Space efficient
  • Typos make eval hard
  • Light transformations
  • Longer, conversational

14 of 50

15 of 50

16 of 50

Conversational QA

Jordan Boyd-Graber

17 of 50

18 of 50

19 of 50

http://sequential.qanta.org/

20 of 50

21 of 50

QB Link Dataset

Only twenty­ one million units in this system will ever be created.

Leadin

Q1

Name this digital payment system whose transactions are recorded on a “block chain”.

​

Q2

It was invented by this person, who, according to a dubious Newsweek cover story, is a 64­ year­ old Japanese­ American man who lives in California.

This online drugs marketplace was started in 2011 by Ross Ulbricht using the pseudonym Dread

Pirate Roberts.

Q3

Bitcoin

Satoshi Nakamoto

Silk Road

22 of 50

23 of 50

http://canard.qanta.org/

24 of 50

25 of 50

26 of 50

27 of 50

28 of 50

29 of 50

30 of 50

What’s the capital of Georgia?

Atlanta

And what’s the population of that city?

Half a million

What’s the capital of Georgia?

Tbilisi

No, I meant the American state.

Sorry! In that case it’s Georgia.

Reference

System

31 of 50

What’s the capital of Georgia?

Atlanta

And what’s the population of that city?

Half a million

What’s the capital of Georgia?

Do you mean the country or the American state?

The country.

Tbilisi

Reference

System

32 of 50

33 of 50

34 of 50

Long Form QA

Jordan Boyd-Graber

35 of 50

36 of 50

37 of 50

38 of 50

39 of 50

40 of 50

Bilingual Evaluation Understudy

From Philipp Koehn

41 of 50

BLEU

42 of 50

43 of 50

Reference:

Comparison:

The children annoyed the parents

Parents annoy the children

The children

children annoyed

annoyed the

the parents

ROUGE-2: 1/4

44 of 50

Reference:

Comparison:

Parents annoy the tired children

Parents do annoy the children daily

Comparison length

Reference length

Weight parameter (usually 1)

= 4/5

= 4/6

45 of 50

​

Hurdles to Progress in Long-form Question Answering (NAACL 2021)

Kalpesh Krishna 1

Mohit Iyyer 1

Aurko Roy 2

1

2

46 of 50

The ELI5 train/valid splits have significant question overlap

47 of 50

Retrievals don’t impact model’s generations

48 of 50

Tricky evaluation, with trivial systems scoring high ROUGE-L

49 of 50

50 of 50