1 of 34

​

Lexical Semantic Change

​

​

​

Deep Learning & Digital Humanities, WS 2021/22

Mark Baierl, Yi Li

Literature:

  • Challenges for computational lexical semantic change
  • Explaining and Improving BERT Performance on Lexical Semantic Change Detection
  • Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media

WS 2021/22 | Fachbereich Informatik | Deep Learning and Digital Humanities (DLDH) 1

​

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 1

2 of 34

Contents

Semantic Change

01

Introduction to Lexical Semantic Change (LSC)

02

Data for LSC

03

Models of meaning & Meaning Change

04

Evaluation

05

Related fields and applications

06

Conclusion

07

Auxiliary papers

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 2

3 of 34

What is Lexical Semantic Change?

01 Introduction to Lexical Semantic Change (LSC)

gay

rock

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 3

4 of 34

02 Data for LSC

Newspaper

Google N-grams

Others

Optimal resource depends on task.

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 4

5 of 34

02 Data for LSC

Problems of Data

  1. English
  2. Artifact
  3. Metadata

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 5

6 of 34

03 Models of meaning & Meaning Change

Levels of word meaning

Computational models of meaning

Two parts:

Levels of word meaning

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 6

7 of 34

03 Models of meaning & Meaning Change

Levels of word meaning

  1. Language-specific semantic
  2. Language-specific lexical
  3. Language-external knowledge

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 7

8 of 34

03 Models of meaning & Meaning Change

Computational models of meaning

  1. Single representation per word
  2. Split word into semantic areas
  3. Every occurrence separately

Different uses and advantages.

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 8

9 of 34

03 Models of meaning & Meaning Change

Computational models of meaning

Problems

  • Preprocessing
  • Distributional hypothesis

How does one accurately portray a word’s senses?

Lexical Meaning vs. Cultural and Topical Information

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 9

10 of 34

03 Models of meaning & Meaning Change

Post-Hoc clustering

by @bryanMMathers

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 10

11 of 34

04 Evaluation

SemEval-2020 Task 1

on Unsupervised Lexical Semantic Change Detection.

provided the first larger-scale, openly available dataset

with high-quality, hand-labeled judgements.

  • a list of target words
  • a set of usages per target word
  • a Word Usage Graph (WUG)

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 11

12 of 34

04 Evaluation

The definition of LSC:

a binary classification & a ranking task Schlechtweg et al. (2020)

  • modelling of words
  • detecting sense changes
  • generalizing across many more time points
  • disambiguating instances of words in the text

Many questions could be asked.

How to evaluate?

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 12

13 of 34

04 Evaluation

Types of evaluations

  1. on ground-truth data → two strategies
  2. on simulated LSC data
  • use a set of predetermined words and with an expected pattern of change
  • evaluate the output of the algorithm

​

Pros.++ Cons.--

​

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 13

14 of 34

04 Evaluation

Simulating LSC in existing data

  • difficult to obtain ground-truth data
  • need sufficient expert annotators
  • need large amounts of text with annotation

How it works? Positive and negative sides?

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 14

15 of 34

04 Evaluation

Quantifying meaning and meaning change

To obtain high-quality and ground-truth data for LSC

Execute manual annotation

Choose the relevant level of meaning

Quantify meaning distinctions and the change of a word

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 15

16 of 34

04 Evaluation

Quantifying meaning and meaning change

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 16

17 of 34

05 Related fields and applications

Two disciplines:

human activities

language

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 17

18 of 34

Conclusions

  • Several data-limitations
    • Limited Languages
    • Data needs precise qualities (+metadata)
    • Difficult to rely on data for other NLP fields
  • importance of what meaning is is undervalued
  • Need for large-scale evaluation frameworks

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 18

19 of 34

Lexical Semantic Change Discovery

  • Application
  • build clusters of usages with same sense
  • Visualise annotated data intuitively

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 19

20 of 34

Data

  • German data set provided by the SemEval-2020
    • Contains a diachronic corpus pair (newspaper)
    • two time periods to be compared
    • a set of carefully selected target words
    • binary and graded gold data
  • Corpora
  • Target Words (48)

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 20

21 of 34

Models

  1. Type-based models
    • Assume word has one sense
    • still dominant in LSCD

​

  • Token-based models
    • Can handle multiple senses
    • Probably more suitable for LSCD

​

​

​

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 21

22 of 34

Annotation

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 22

23 of 34

Explaining and Improving BERT Performance on Lexical Semantic Change Detection

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 23

24 of 34

Explaining and Improving BERT Performance on Lexical Semantic Change Detection

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 24

25 of 34

Explaining and Improving BERT Performance on Lexical Semantic Change Detection

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 25

26 of 34

Explaining and Improving BERT Performance on Lexical Semantic Change Detection

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 26

27 of 34

Explaining and Improving BERT Performance on Lexical Semantic Change Detection

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 27

28 of 34

Explaining and Improving BERT Performance on Lexical Semantic Change Detection

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 28

29 of 34

Temporal Adaptation of BERT

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 29

30 of 34

Temporal Adaptation of BERT

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 30

31 of 34

Temporal Adaptation of BERT

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 31

32 of 34

Temporal Adaptation of BERT

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 32

33 of 34

Temporal Adaptation of BERT

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 33

34 of 34

Temporal Adaptation of BERT

Jan 2022 | DLDH 2021 | Mark Baierl, Yi Li 34