Parametric and semi-parametric methods for knowledge acquisition from text

Yury Zemlyanskiy

December 9, 2021

1

Motivation

Knowledge acquisition = (extract + store + retrieve) knowledge

2

Motivation

Modern NLP neural models do not have a separate memory and store knowledge in the parameters ⟹ unreliable (Cao et al., 2021) and require retraining to incorporate new information

3

processing

Textual data

Neural NLP model

Cao et al., Knowledgeable or educated guess? revisiting language models as knowledge bases. ACL/IJCNLP 2021

Key insight: Memory

Key hypothesis of this thesis: we can improve model’s knowledge acquisition abilities by adding a dedicated memory component.

4

Memory

The exact kind of memory depends on a task at hand.

processing

Textual data

Neural NLP model

What kind of memory?

Task: Knowledge base completion for a fixed set of entities based on relevant documents.

Parametric memory for aggregated information on each entity to reliably extract knowledge about the entity

5

Do people generally like this movie?�

Is it “surreal”?�

Are there “multiple storylines”?

compress

What kind of memory?

Task: Question-answering over books or entire Wikipedia (open-domain)

Semi-parametric memory for individual entity-centric facts from text,

amendable without re-training

​

6

What kind of memory?

Task: Structured prediction (semantic parsing or machine translation) in low resource scenario

​

Set reminder to send happy b-day text to Clark on Friday

SetReminder(

Message(� text= happy b-day,

whom= Clark),

time= on Friday)

Semi-parametric memory of the individual training samples

Add another reminder for next week’s exam.

SetReminder(GetTodo(time= for next week, what=exam))

Send a happy birthday message to dad

Message(whom= dad, text= happy birthday)

7

Neural net

compress

Outline

1. Parametric entity memory
• Zemlyanskiy et al., DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections, EACL’21 (qualification exam)
2. Semi-parametric entity memory
• Zemlyanskiy et al., ReadTwice: Reading Very Large Documents with Memories, NAACL’21 (qualification exam)
• de Jong and Zemlyanskiy et al., Mention Memory: incorporating textual knowledge into Transformers through entity mention attention, in submission to ICLR’22
3. Semi-parametric task-specific memory
• Planned work

8

• Motivation
• Completed work
• Parametric entity memory
• Semi-parametric entity memory
• Intro: memory
• Memory of large documents
• Memory of Wikipedia
• Planned work
• Task-specific memory
• Conclusion

9

Parametric entity memory

1. Zemlyanskiy et al., DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections, EACL 2021

• Motivation
• Completed work
• Parametric entity memory
• Semi-parametric entity memory
• Intro: memory
• Memory of large documents
• Memory of Wikipedia
• Planned work
• Task-specific memory
• Conclusion

10

Motivation

Task: Predict movie attributes based on information in movies reviews.

1. Train neural model on reviews data jointly with leanable entity embeddings
2. Fine-tune and apply the model on tag prediction task.

11

Train

MovieLens tag

nonlinear

multiple storylines

Japan

imdb top 250

surreal

cerebral

Fine-tune

Contributions

• DOCENT -- new method to learn entity embeddings
• Model joint distribution of entities and related text instead of conditional distribution of entity given text�
• Experiments: MovieLens tag prediction
• Entity memory matters -- outperform baselines without entity-specific parameters
• DOCENT matters -- outperforms other methods to learn entity embedding
• Generalizes well to unseens tags (treating them as arbitrary text)

12

Semi-parametric entity memory

• Zemlyanskiy et al., ReadTwice: Reading Very Large Documents with Memories, NAACL’21�
• de Jong and Zemlyanskiy et al., Mention Memory: incorporating textual knowledge into Transformers through entity mention attention, in submission to ICLR’22

• Motivation
• Completed work
• Parametric entity memory
• Semi-parametric entity memory
• Intro: memory
• Memory of large documents
• Memory of Wikipedia
• Planned work
• Task-specific memory
• Conclusion

13

Intro: memory�

• Motivation
• Completed work
• Parametric entity memory
• Semi-parametric entity memory
• Intro: memory
• Memory of large documents
• Memory of Wikipedia
• Planned work
• Task-specific memory
• Conclusion

14

Text as a virtual knowledge base

What is known about Perseus?

[Perseus] was the son of Zeus and Danaë, ...

… after [Perseus] married Andromeda founded a city and called it Amandra …

[Perseus] was a Greek hero who killed Medusa ...

Heracles was 4 generations later than [Perseus] ...

text as a virtual knowledge base�(Dhingra et al., 2020)

regular knowledge base

15

Mention Encodings

• A passage that contains entity mention describes� ⟹ a property of the entity� ⟹ or its relation to other entities
• Train a Mention Encoder model to capture this semantic information from entity mentions into high-dimensional mention encodings

​

MentionEncoder(“[Perseus] was a Greek hero who bravely killed Medusa.”) ∈ R^d

16

Memory based on mentions

Collect mention encodings from the entire document into a Memory Table.

17

[Perseus] was a Greek hero who bravely ...�Perseus was a [Greek] hero who bravely ...�...�[H. Simpson] is a character from US sitcom�H. Simpson is a character from [US] sitcom

...

MentionEncoder

All mentions in a document or a corpus

Memory Table:

number of mentions x d

Why memory table?

Memory table as knowledge representation has several advantages

• Mention encodings could be fine-grained enough to be useful
• Easy to add / remove / modify memory without re-training the model
• Easy to integrate into a neural network using attention
• grounded -- easier to inspect which information was used by the model

18

Contributions

Both: new method to train Mention Encoder using only textual data�(and entity annotations)

Memory

Memory

ReadTwice:

MentionMemory:

Use memory to help a neural network process long documents

Use memory to capture world knowledge about entities in a format compatible with a neural network

​

19

Memory of large documents

• Zemlyanskiy et al., ReadTwice: Reading Very Large Documents with Memories, NAACL’21

• Motivation
• Completed work
• Parametric entity memory
• Semi-parametric entity memory
• Intro: memory
• Memory of large documents
• Memory of Wikipedia
• Planned work
• Task-specific memory
• Conclusion

20

Motivation: question answering

Task: Question Answering that require assimilating information from different sections of large inputs such as books or article collections.

Who does Aragorn take as his wife?

21

Existing methods: BERT

Goal: Output vector for every word representing word’s meaning within a sentence.

22

Frodo

couldn’t

throw

Transformer model

The

Ring

into

the

fire

Existing methods: Transfer learning

23

[MASK]

killed

Medusa

Transformer model

Perseus

Transfer

Pre-training stage:�Masked Language Model (MLM) task

Use pre-trained model as initialization

Fine-tuning stage:�Any downstream task of interest

Question Answering (QA)

Text Classification

Named Entity Recognition

...

BERT’s limitations

• The length of the input is limited (~ 512 tokens)
• due to high computational costs of the Transformer’s attention mechanism.�
• harder to understand text without context

Inside the Sammath Naur, Frodo struggled to throw the Ring into the Cracks of Doom.

• BERT struggles with long document (books) QA

Who does Aragorn take as his wife?�BERT: Éowyn ❌

24

Motivation: long-range dependencies

Long-range dependencies are common and modeling them is crucial

Frodo couldn’t throw [MASK] into the fire ⇒ BERT ⇒ ???

but

Frodo couldn’t throw [MASK] into the fire � + The Ring slowly gained control over Frodo ⇒ BERT ⇒ The Ring

25

Motivation: long-range dependencies

Mention encodings as a way to model long range dependencies

Frodo couldn’t throw [MASK] into the fire ⇒ BERT ⇒ ???

but

Frodo couldn’t throw [MASK] into the fire � + MentionEncoding(“The Ring slowly gained control over [Frodo]”)� BERT ⇒ The Ring

26

ReadTwice method

​

• Split long text into shorter segments
• 1^st pass: Process segments independently with BERT model
• Extract entity mention information into a memory table.
• 2^nd pass: Process segments independently now using with compressed information from other segments as an external memory

27

ReadTwice: 1^st read

Frodo couldn’t throw [MASK] into the fire

The Ring slowly gained control over Frodo

28

BERT₁

BERT₁

[Frodo] couldn’t throw [MASK] into the fire

[The Ring] slowly gained control over Frodo

The Ring slowly gained control over [Frodo]

ReadTwice: 1^st read

Frodo couldn’t throw [MASK] into the fire

The Ring slowly gained control over Frodo

29

BERT₁

BERT₁

[Frodo] couldn’t throw [MASK] into the fire

[The Ring] slowly gained control over Frodo

The Ring slowly gained control over [Frodo]

Memory Table: number of mentions x 768

ReadTwice: 2^nd read

Frodo couldn’t throw [MASK] into the fire

The Ring slowly gained control over Frodo

30

Memory Table

Attention Over Memory Table

BERT₁

BERT₁

Frodo

...

throw

The

Ring

slowly

...

over

into

the

fire

Frodo

ReadTwice: 2^nd read

Frodo couldn’t throw [MASK] into the fire

The Ring slowly gained control over Frodo

31

Memory Table

Attention Over Memory Table

BERT₂

BERT₂

BERT₁

BERT₁

Frodo

...

fire

The

...

Frodo

How to encode a mention?

32

[The

Ring]

gained

BERT₁

control

over

[Frodo]

Linear

Linear

[The Ring]

[Frodo]

Pre-training

The model is trained end-to-end on masked language model task

• Mention encoder learns to produce memories that are “useful” for the second read
• Batch has 128 segments per document
• segments = consecutive spans of 512 tokens
• ⟹ model can process documents up to 65k (≈ 128×512) tokens

33

Evaluation: QA over books

Dataset

• NarrativeQA
• Input: entire books or movie scripts
• Average of 62,000 words per document

​

�

“Who does Aragorn take as his wife?”

Baselines

• RoBERTa
• a version of BERT model
• process passages up to 512 tokens
• ETC
• a modern long-range version of BERT
• process passages up to 5k tokens

34

Results: NarrativeQA

ReadTwice: New state-of-the-art for NarrativeQA

35

Results on dev / test sets

Insight: Inter-segment memory matters

ReadTwice: It’s important to attend memories from other segments

36

Memory of Wikipedia

• de Jong and Zemlyanskiy et al., Mention Memory: incorporating textual knowledge into Transformers through entity mention attention, in submission to ICLR’22

• Motivation
• Completed work
• Parametric entity memory
• Semi-parametric entity memory
• Intro: memory
• Memory of large documents
• Memory of Wikipedia
• Planned work
• Task-specific memory
• Conclusion

37

Motivation: open domain QA

38

What is the nationality of the hero killed Medusa?�BERT: England ❌

Motivation

Goal: incorporate information from the English Wikipedia into BERT model

​

Existing solutions:

• Large LM: store information in parameters� ❌ Compute-inefficient, frequent factual mistakes
• Retrieve-and-read: retrieve and encode text� ❌ Limited text in BERT, apply entire BERT model for every retrieved passage

​

Proposal: Mention Memory -- retrieve and integrate mention encoding

​

39

TOME: Transformer over Memory

40

TOMEBlock

1. For each mention in input passage
1. Extract mention encoding
2. Retrieve top-K relevant mention encodings from memory
3. Combine with current input mention representations
2. Apply L Transformer layers

​

High-level approach: 2 training phases

• Phase 1: Mention encoder pre-training
• Pre-train mention encoder model on masked language model (MLM) task
• Generate memory table by encoding named entity mentions in Wikipedia.
• Freeze the memory table�
• Phase 2: TOME model pre-training
• Perform second MLM pre-training with attention over frozen memory

41

Train Mention Encoder

Generate Mention Memory by encoding mentions in Wikipedia

Train TOME model

Pre-training mention encoder

Challenge: How to train Mention Encoder to produce useful representations?

​

• Smaller-scale mention memory task with backpropagation
• Train Batch-TOME with in-batch memory
• Auxiliary tasks for Batch-TOME: (masked) entity coreference resolution

42

Batch: Related Wiki articles

Mention Encoder

Batch-TOME

In-batch memory

Objectives

Pre-training: insights

• Batch-TOME and TOME trained on just MLM learn to attend to memories of the same entity: 55% and 41% average attention score.
• Expected: mentions from the same entity often contain mutually relevant information

​

Batch-TOME and TOME attend to relevant mentions w/o any supervision

43

How good is the Mention Encoder?

Task: Classify a mention in a passage into several categories.

Dataset: Ultra Fine Entity Typing

“[We] want to find the truth, regardless of what the truth is, ….,” chief Yemeni investigator, minister of sea and transport Said Yafaai, told the reporter … �⟹ detective, enforcement, government, authority, investigator, law_enforcement, police, agency, administration

Labels (total 10331):

• Coarse-grained (9): person, group, organization, location, entity, time, object, event, place
• Fine-grained (121): medicine, institution, subway, soldier, province, ...
• Ultra-fine-grained (10201): front_bench, collective_agreement, business_administration, ...

44

Evaluation: Ultra Fine Entity Typing

MentionEncoder: outperforms BERT in capturing mention-level information

45

Evaluation: MentionMemory + TOME

Open-domain boolean QA / claim-verification tasks: FEVER, HoVeR

• FEVER Example
• Claim: Nikolaj Coster-Waldau worked with the Fox Broadcasting Company
• Evidence: Nikolaj Coster-Waldau played a detective in the short-lived Fox television series New Amsterdam (2008)
• HoVeR Example
• Claim: Patrick Carpentier currently drives a Ford Fusion, introduced for model year 2006, in the NASCAR
• Evidence 1: Ford Fusion is manufactured and marketed by Ford. Introduced for the 2006 model year, …
• Evidence 2: Patrick Carpentier competed in the NASCAR Sprint Cup Series, driving the Ford Fusion.

46

Evaluation: open domain QA

47

TOME-1 and TOME-2 have 1/2 TOMEBlocks (same total #parameters as BERT)

HoVer example: simple retrieval

48

Evaluation: open domain entity QA

49

• QA with entity as an answer: “What is the nationality of the hero who killed Medusa”
• Predict entity answer by retrieving mention of that entity in final prediction layer

Insight: memory size matters

• TOME is pre-trained with ~38M mentions
• More memory we use during fine-tuning ⟹ the better the performance.

Memory size matters

50

Insights: Mention Encoder pre-training

51

Auxiliary loss on mention encoder training is important to generate better memory

Insight: memory is amendable

52

• Sample answer entities for questions from TriviaQA and ComplexWebQuestions dev sets.
• TOME-1-unseen: exclude mentions of the entities from the Mention Memory
• during TOME pre-training
• during TOME fine-tuning on TriviaQA and ComplexWebQuestions
• Evaluate TOME-unseen on the sampled questions using the full memory for evaluation only

Memory can be modified on the fly without re-training.

Contributions

• ReadTwice project:
• ReadTwice method -- use memory to improve processing of large documents
• Achieve state-of-the-art results on a challenging NarrativeQA benchmark
• MentionMemory project:
• Use memory to represent world knowledge about entities for another model to use
• Developed pre-training procedure and auxiliary loss to generate high quality memory
• Experiments to investigate memory properties
• Mention encoder:
• can be trained using only textual data
• can be used for other mention-oriented tasks (like mention typing)

53

Task-specific memory

• Planned work

• Motivation
• Completed work
• Parametric entity memory
• Semi-parametric entity memory
• Intro: memory
• Memory of large documents
• Memory of Wikipedia
• Planned work
• Task-specific memory
• Conclusion

54

Motivation: Structured prediction

Semantic parsing

Machine Translation

Set reminder to send happy b-day text to Clark on Friday

​

Reminder(

Message(text= happy b-day, whom= Clark),

time= on Friday)

Low resource setting (<10K training samples) is common and challenging

55

Intuition

Intuition: A new input sample contains partial similarities with some train samples.

Add another reminder for next week’s exam.

SetReminder(� GetTodo(� time= for next week,� what= exam))

Send a happy birthday message to dad

Message(� whom= dad,� text= happy birthday)

Set reminder to send happy b-day text to Clark on Friday

SetReminder(� Message(� text= happy b-day,

whom= Clark),

time= on Friday)

56

Train sample

Train sample

New sample

Key hypothesis: memory

Add another reminder for next week’s exam.

SetReminder(GetTodo(time= for next week, what=exam))

Semi-parametric memory: encode training data in an explicitly decomposed way:

memories of individual samples or their parts

Send a happy birthday message to dad

Message(whom= dad, text= happy birthday)

57

MemoryEncoder

Memory of

training sample #1

Memory ∈ R^d of

training sample #2

Key hypothesis: memory

Hypothesis: Improve prediction accuracy by allowing the model to retrieve and re-combine multiple training samples from memory

• Make the model better at compositional generalization
• Better memoize in-frequent patterns and sub-labels

58

Method

The model is similar to the TOME model.

59

Send

...

dad

Initial Transformer model

i-th Transformer block

Decoder model

N blocks

Related work

• Large sequence-to-sequence models: store information in parameters� ❌ Challenging to memoize infrequent patterns� ❌ Fails to generalize to unseen combinations

​

• Retrieve-and-read: retrieve similar training samples and process with input� ❌ Limited in number of retrievals� ❌ Cannot recover from a retrieval mistake� ❌ Performs retrieval for an entire input

60

Plan / Expected Contribution

• Start with the semantic parsing task
• Datasets: TOP, TOPv2, MTOP and SB-TOP
• Step 1: Exploratory experiments with oracle retrieval
• Provide the model memories of training samples that we know are relevant.
• Step 2: Joint model and retrieval training
• Follow ups: machine translation task
• Memory = a neural network version of phrase table in statistical phrase-based systems.

61

Conclusion

• Motivation
• Completed work
• Parametric entity memory
• Semi-parametric entity memory
• Intro: memory
• Memory of large documents
• Memory of Wikipedia
• Planned work
• Task-specific memory
• Conclusion

62

Conclusion

We propose and study several methods for knowledge acquisition from text.

We consider several challenging knowledge-intensive tasks arising in natural language processing.

We improve the knowledge acquisition abilities of modern neural networks by adding a dedicated memory component.

63

Timeline

• Step 1: Exploratory experiments
• Step 2: Joint model and retrieval training
• Follow ups: machine translation task

64

Summer 2022

Spring 2022

Defense

July/August

Step 1

Follow ups

Step 2

Acknowledgements

Many thanks to my advisor Fei Sha and my co-authors

Docent: Sudeep Gandhe, Ruining He, Bhargav Kanagal, Anirudh Ravula, Juraj Gottweis, Fei Sha, Ilya Eckstein

ReadTwice: Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, Fei Sha

Mention Memory: Michiel de Jong, Nicholas FitzGerald, William Cohen, Fei Sha

​

65

Thank you!

Questions?

66

Evaluation: MovieLens tag prediction

Attention over memory

Differentiable equivalent of key-value memory lookup for neural networks.

68

key₁

key₂

...

key_N

value₁

value₂

...

value_N

query

query, key_i ∈ ℝ^K; value_i ∈ ℝ^V

Attention over memory

Differentiable equivalent of key-value memory lookup for neural networks.

69

key₁

key₂

...

key_N

value₁

value₂

...

value_N

query

attn₁

∑_i attn_i·value_i

attn₂

...

attn_N

Transformer

70

Which

hero

killed

Medusa

?

Multi-head self-attention layer

MLP

MLP

MLP

MLP

MLP

Word embeddings

Input text

“Hidden” states

2-layer neural network

Contextualized word embeddings

BERT

Goal: Output vector for every word representing word’s meaning within a sentence.

71

Astronaut

Neil

Armstrong

Transformer model

put

money

in

bank

account

BERT

Idea: train the model to produce "general" embeddings via Mask Language Model task.

72

Astronaut

[MASK]

Armstrong

Transformer model

put

[MASK]

in

bank

account

Neil

money

Motivation: question answering

Google was founded in 1998 by Larry Page and Sergey Brin while they were Ph.D. students at Stanford University in California. Together they own about 14 percent of its shares and control 56 percent of the stockholder voting power through supervoting stock. They incorporated Google as a privately held company on September 4, 1998. An initial public offering (IPO) took place on August 19, 2004, and Google moved to its headquarters in Mountain View, California, nicknamed the Googleplex. In August 2015, Google announced plans to reorganize its various interests as a conglomerate called Alphabet Inc. Sundar Pichai was appointed CEO of Google, replacing Larry Page who became the CEO of Alphabet.

Questions based on the passage:

What is IPO?�initial public offering

Where is headquarter of Google?�Mountain View, California

Who is the current CEO?�Sundar Pichai

Challenging for modern neural networks when the document is very long (e.g., entire book).

73

Insight: Entity memory matters

Memory based on entity mentions outperforms other memories.

74

Evaluation: open domain entity QA

75

• Predict entity answer by retrieving mention of that entity in final prediction layer
• 84% and 94% of questions are answerable by MM, rest marked as wrong

HoVer example: multiple retrieval

76

The End

77