CAN�A PRETRAINED NEURAL �LANGUAGE MODEL ��STILL BENEFIT FROM ��

Jakob Prange

​

11th May, 2023

Some upper and some lower bounds

�LINGUISTIC SYMBOL STRUCTURE�?

A SHORT HISTORY LESSON

Syntax trees, Logic, Formal Grammars��Connectionism

Deep Learning

Transformers, BERT

GPT-3, ChatGPT, GPT-4

Rule-based systems, chart-parsers, …

“Golden Age of symbolic linguistics”

Markov models, SVMs, CRFs

LSTMs, Convolutional nets

Models are starting to get more powerful but�also less interpretable

“Rediscover the classical NLP pipeline”?

Models seem to really “get” meaning and remember things

Jakob Prange

2

11th May, 2023

Statistical �Machine Learning

2017, 2019

1990s

early 2000s

1950s

DEEP LEARNING �FROM A LINGUISTIC-COGNITIVE PERSPECTIVE

Successes and Questions

• Big wins in task-specific performance
• Continuous criticism, skepticism, scientific curiosity about linguistic-cognitive capabilities
• Many major technological novelties originated in linguistic-cognitive theories �(LSTM, attention, …)

Cognitive interpretability and (im)plausibility

• 5-10 years ago: trying to analyze the behavior of individual neurons
• BERTology (last 5 years): analyzing the behavior of network layers�🡪”functional connectivity”

​

• The better the task performance, the worse the correlation with human judgments

Jakob Prange

3

11th May, 2023

OVERVIEW

• Linguistics & Neuro-symbolic LM
• Upper bounds (and their problems)
• Lower bounds (and the unexpected difficulty of reaching them)
• Is it worth it?

Jakob Prange

4

11th May, 2023

NEURO-SYMBOLIC LANGUAGE MODELS

Jakob Prange

5

11th May, 2023

Alice did not know what to say

Jakob Prange

6

11th May, 2023

AMR

UCCA

PTB

UD

Syntax

Semantics

Dependencies

Constituencies

[Nivre+ 2016, 2020]

[Marcus+ 1993; Hovy+ 2006]

[Banarescu+ 2013]

[Abend & Rappoport 2013]

Alice did not know what to say

SLR

Symbolic/Structured Linguistic Representation

​

WHOA THERE, NOT SO FAST!

Jakob Prange

7

11th May, 2023

Grammar Rules

S → NP VP

VP → (AUX) (NEG) V

VP → (AUX) (NEG) V VP

VP → NP AUX V

NP → N

Syntactic Constituent Tree

from the Penn Treebank (PTB)

Semantic Argument-structure Graph

in the UCCA Framework

Cognitive Concepts

Scenes

Process vs State

Shared Argumenthood

SLICING UP LINGUISTIC GRAPHS

Jakob Prange

8

11th May, 2023

SLICING UP LINGUISTIC GRAPHS

Jakob Prange

9

11th May, 2023

SLICING UP LINGUISTIC GRAPHS

Jakob Prange

10

11th May, 2023

ENCODING GRAPHS AS VECTORS

Jakob Prange

11

11th May, 2023

"High resolution"

"Low resolution"

Edge labels + anchor token embeddings

Structural relations

R-GCN [Schlichtkrull+ 2018; Kipf & Welling 2017]

Inefficient in the incremental setting

ENCODING GRAPHS AS VECTORS

Jakob Prange

12

11th May, 2023

Parent

Sibling

Lookups

NP-SUBJ

VP

…

…

NEURO-SYMBOLIC LM

Jakob Prange

13

11th May, 2023

…

…

Parent

Sibling

Flatten &�Feed-forward

NP-SUBJ

VP

Lookups

logits_Symbolic

GPT-2

logits_Neural

BUT WHERE DOES THE SYMBOL STRUCTURE COME FROM?

Jakob Prange

14

11th May, 2023

OVERVIEW

• Linguistics & Neuro-symbolic LM
• Upper bounds (and their problems)
• Lower bounds (and the unexpected difficulty of reaching them)
• Is it worth it?

Jakob Prange

15

11th May, 2023

UPPER BOUNDS

(We don’t care where it comes from, we just assume we have it)

Jakob Prange

16

11th May, 2023

Published in�NAACL-HLT 2022, Seattle

EXPERIMENTAL SETUP

​

• Annotated text corpus

​

​

• Use ground truth graphs at training and test time
• Controls for parser performance
• Upper bound for LM performance potential

17

11th May, 2023

Jakob Prange

EXPERIMENTAL SETUP

• 7 sentence-structure SLRs with different properties
• Semantic constituencies: 2 (EDS, PTG)
• Syntactic constituencies: 2 (PTB with 2 different label sets)
• Semantic dependencies: 2 (DM, PSD)
• Syntactic dependencies: 1 (UD)

​

​

​

​

​

​

• All annotated over the same text corpus 🡪 Relatively fair comparison �(the subset of English WSJ used in the MRP shared tasks)

18

w₁

w₂

w₃

w₁

w₂

w₃

11th May, 2023

Jakob Prange

Alice did not know what to say

Jakob Prange

19

11th May, 2023

AMR

UCCA

PTB

UD

Syntax

Semantics

Dependencies

Constituencies

[Nivre+ 2016, 2020]

[Marcus+ 1993; Hovy+ 2006]

[Banarescu+ 2013]

[Abend & Rappoport 2013]

Alice did not know what to say

RESULTS IN PERPLEXITY (LOWER IS BETTER)

Jakob Prange

20

11th May, 2023

GRAPH-TO-VECTOR: R-GCN ENCODING VS CONCATENATION

Jakob Prange

21

11th May, 2023

PROBLEMS WITH UPPER BOUND ASSUMPTIONS

Jakob Prange

22

11th May, 2023

• Availability

• Annotatability

• Predictability

• Richness

LOWER BOUNDS

(What if we jointly predict words and structure – how good does our structure parser need to be?)

Jakob Prange

23

11th May, 2023

To appear in *SEM 2023, Toronto

A JOINT MODEL

Jakob Prange

24

11th May, 2023

A JOINT MODEL

Jakob Prange

25

11th May, 2023

¯\_(ツ)_/¯

TWO “ADVERSARIAL” APPROACHES

Reduce Signal

Obscure Signal

Jakob Prange

26

11th May, 2023

⤮

REDUCE SIGNAL

Jakob Prange

27

11th May, 2023

REDUCE SIGNAL

Jakob Prange

28

11th May, 2023

¯\_(ツ)_/¯

OBSCURE SIGNAL

Jakob Prange

29

11th May, 2023

¯\_(ツ)_/¯

SIGNAL/NOISE DISTRIBUTIONS MATTER!

Jakob Prange

30

11th May, 2023

IS IT WORTH IT?

If you want to learn about language and build cognitively plausible models: Maybe.

Jakob Prange

31

11th May, 2023

¯\_(ツ)_/¯

If you have the resources to build the next ChatGPT and want to earn a lot of money: Probably not.

THANK YOU!

jakob.prange@polyu.edu.hk

Jakob Prange

32

11th May, 2023

NAACL-HLT 2022

*SEM 2023