1 of 55

Computational models for

Lexical Semantic Change

Francesco Periti, francesco.periti@unimi.it

LOT Winter School 2024

2 of 55

CIAO!

Francesco Periti

3^rdyear - PhD Student

Natural Language Processing, Distributional Semantic Models, Lexical Semantic Change

francesco.periti@unimi.it

2

LOT Winter School 2024

3 of 55

Contextualized

embedding

models

References:�

A Survey on Contextualised Semantic Shift Detection. �Contextualized embeddings for semantic change detection: Lessons learned.

3

LOT Winter School 2024

4 of 55

Problem definition

4

manufacture

From to make by hand

To to make by machine

gay

From cheerful

To homosexual

Lexical Semantic Change

LOT Winter School 2024

Let’s start by refreshing the problem. Let’s do it in a different way. Consider a large, digitized, diachronic corpus that contains documents from various time periods. Formally, we can view the set of documents from each time period as a subcorpus of the entire diachronic corpus. For example, here we have three distinct subcorpora (i.e. C1, C2, C3) associated to specific time periods, and we define the diachronic corpus (i.e. C) as the union of these subcorpora. Note that these time periods are just examples; we can choose to split the diachronic corpus in different ways, depending on the time granularity we are interested to analyze. ��The goal is to identify words that exhibit semantic change in the diachronic corpus across different time periods. For instance, 'manufacturer' has changed from meaning 'to make by hand' to meaning 'to make by machine' during the industrial revolution. Similarly, the word 'gay' has changed in the last century, shifting from meaning 'cheerful' to primarily meaning 'homosexual' . Of course, we can consider different kinds of changes.

This is the general problem, but it is too broad and complex to be addressed in this way, by considering multiple time periods at once. Thus, generally, the focus is narrowed to detecting semantic changes in two time periods only (i.e. a single time interval). It still can be easily extended to scenarios involving multiple time periods.

5 of 55

Static vs. Contextualized embeddings

Quantifying lexical semantic change

5

plane

trained / pre-trained��Word2Vec

FastText

GloVe

pre-trained / fine-tuned��BERT - mBERT

RoBERTa - XLM-R

XL-LEXEME

[...] plane [...]

word-level approaches word usage-level approaches

LOT Winter School 2024

When using contextualized embeddings to address the LSC problem, it is essential to distinguish them from the previous static embeddings. Static embedding models such as Word2Vec, FastText, and GloVe encode a word in a single static embedding that retains the dominant semantic of that word. For instance the word 'plane' has a single vector, and other vectors are associated to other words. The closer the vectors, the more semantically similar are the words. Approaches to quantify lexical semantic change with static embeddings operate at a word-level granularity.

In contrast, contextualized embedding models like BERT, mBERT, or XL-LEXEME generate a distinct embedding for each word occurrence based on the context in which the word appears. For instance, we have multiple vectors for the word plane. The closer the vectors of the word 'plane,' the more probable it is that the associated word occurrences denote the same meaning (e.g. airplane, coordinate plane, or inclined plane). Approaches to quantify lexical semantic change with contextualized embeddings operate at a word usage-level granularity. This way, we have a multitude of options for measuring lexical semantic change with contextualized embeddings.

6 of 55

Word-usage selection

6

target word

- If a particle of mass m is placed on a smooth inclined plane and re- leased, it will slide down the slope.

- [...]

Word usages

- Euclidean planes are Euclidean spaces of dimension two

- [...]

- The plane flew above the clouds.

- [...]

LOT Winter School 2024

7 of 55

Word-usage selection

7

def word_usage_selection(corpus: list, target: str):

indexes_target_token = list() # position of word in sentence

word_usages = list() # sentences

for sentence in corpus:

sentence_lemma = lemmatize(sentence) # spacy (See spacy.io/api/token)

for token in sentence_lemma:

if token.lemma_ == target:

start = token.idx

end = start + len(token.text)

indexes_target_token.append(f'{start}:{end}')

word_usages.append(sentence)

# break

df = pd.DataFrame() # pandas

df['indexes_target_token'] = indexes_target_token

df['context'] = word_usages

return df

target word

LOT Winter School 2024

either or

How can we do it? That’s more complex than it seems. Here, there is a Python pseudocode. It works, but it has some limitations that we will discuss further later. Given a corpus and a target, we want to retrieve all occurrences of a word in the corpus in each of its forms. Specifically, we want to know exactly the position of the word in the context where it occurs.

For simplicity, we can consider a corpus split into a list of sentences. For each sentence, we lemmatize the sentence. There are several options to perform lemmatization. Here, we are considering the spaCy library, which is well-known and generally adopted.

SpaCy tokenizes and lemmatizes our sentence. Then, for each token in the sentence, if its lemma is equal to our target, then we have found a new occurrence. SpaCy enables us to access the position of the word in context. Then, we store in two separate lists both the sentence and the indexes to access the word in the sentence.

Why this break? Because a sentence could potentially contains more occurrence of the same word. We can decide if consider all the occurrences or to approximate and take only the first one.

Anyway, here we are storing this data in a pandas DataFrame for simplicity

8 of 55

Embedding extraction

8

target word

bert

[ "[CLS]", "the", "plane", "flew", "above", "the", "clouds", ".", "[SEP]" ]

The plane flew above the clouds.

word usage

LOT Winter School 2024

[ ⚪, ⚪, 🔵, ⚪, ⚪, ⚪, ⚪, ⚪, ⚪ ]

9 of 55

Embedding extraction

9

def embedding_extraction(df: pd.DataFrame):

embeddings = list()

for _, row in df.iterrows():

start, end = row["indexes_target_token"].split(':')

start, end = int(start), int(end)

left_context = row['context'][:start]� word = row['context'][start:end]

right_context = row['context'][end:]�

left_tokens = ['[CLS]'] + tokenize(left_context)

word_tokens = tokenize(word)

right_tokens = tokenize(right_context) + ['[SEP]']

# start and end in terms of tokens

start, end = len(left_tokens), len(left_tokens) + len(word_tokens)

# tokenization and embeddings

encoded_input = tokenizer(row['context']) # transformers

output = model(**encoded_input)

embedding = output.last_hidden_state[0, start:end, :].mean(axis=0)

embeddings.append(embedding)

return np.array(embeddings)

target word

bert

word usage

LOT Winter School 2024

The plane flew above the clouds.

index: 4:9

[CLS], the, plane, flew, above, the, clouds, ., [SEP]

index: 2:3

Let’s focus on this pseudocode to understand how to do it. We have this function called "embedding extraction" that takes as input the previous pandas DataFrame we created. For each row of the dataframe, we want to store embeddings of the corresponding word usage. Then, we take the position of the word in context and split the context into three parts: left context, word usage, and right context.

Next, we tokenize each part separately. Remember to add special tokens at the beginning and at the end. This way, we can shift from the position of the word in the context in terms of characters to the position of the word in the context in terms of tokens generated by the model tokenizer. These indices will allow us to access the embeddings of the exact subwords of our target.

After tokenizing the context, we generate the embeddings and extract the embeddings of the word we are interested in. If the word is split into multiple subwords, we take the average embeddings, following the typical procedure adopted in the state of the art. The embeddings are then stored.

Finally, we return an array of word embeddings.

10 of 55

Embedding aggregation by averaging

10

target word

bert

word usages

word prototype

plane

aircraft

airplane

slope

ramp

incline

facet

surface

sheet

LOT Winter School 2024

Once we have collected the embeddings of each target word, we can optionally aggregate this set of embeddings to reduce the number of embeddings to store in memory. A first choice is to aggregate them by averaging. Let’s consider this figure. Let’s assume we have some embeddings related to different meanings… Some related to plane as facet/surface/sheet. Other related to plane as incline/ramp/slope. Other related to airplane and aircraft. Each of these embeddings has a different position in the embedding space more or less close. Then an option is to consider the centroid of this cloud of representations. This representation is typically called ‘word prototype’ and represents the dominant or general meaning of the word.

We have two set of word usages and corresponding embeddings, one for the subcorpus Ci and another for the subcorpus Cj. By aggregating the embeddings for Ci and for Cj, we obtain two word prototypes.

11 of 55

Embedding aggregation by clustering + averaging

11

target word

bert

plane

landing

boarding

aviation

plane

infinity

parallel lines

surface

plane

inclined

inclined by 60°

word usages

LOT Winter School 2024

inclined by 20°

sense prototypes

Another option is to group embeddings according to the meaning of each word usage. For example, we can group word usages according to three different meanings of the word "plane": one related to airplanes, one to inclined planes, and one to parallel lines. Of course, we don’t know this information in advance. Therefore we need to run a clustering activity to infer the meanings of a word. After clustering, the embeddings of each cluster can be aggregated by averaging, thus creating sense prototypes. The idea is to encode the dominant/general meaning of that group of embeddings in the centroid. For example, we can have occurrences of ‘white plane’, ‘blue plane’, ‘ryanair plane’. We can compact all these representation into a single representation of ‘plane’. This is the theoretical goal, even we will see later that in practice is very hard to obtain this situation.��We create sense prototypes for each time periods. In this example, we have two sense prototypes for the sense of an inclined plane: one for time Ci and one for Cj. And only one sense prototype for the sense of airplane, as there is no embeddings from Cj that falls into that cluster.

12 of 55

Shift assessment

12

LOT Winter School 2024

target word

bert

word usages

A survey

13 of 55

Overall pipeline

13

- If a particle of mass m is placed on a smooth inclined plane and re- leased, it will slide down the slope.

- [...]

Word usage

selection

Embedding extraction

Embedding

aggregation

[⚪, ⚪, 🔵, ⚪, ⚪, ⚪]

prototypes

Shift

assessment

LOT Winter School 2024

14 of 55

A classification of LSC approaches

14

LOT Winter School 2024

Meaning representation

Time-awareness

Learning modality

How the meaning(s) of a

word is represented

How the time of the text is considered

How external knowledge is exploited

form-based

sense-based

time-oblivious

time-aware

supervised

unsupervised

A survey

The advent of word embeddings has pushed the modeling of semantic change. More and more researchers and practitioners have used contextualized models to address the problem. This has led to a plenty of experiments because, as a matter of fact, the behavior of contextualized embeddings is still not completely understood. While surveys on static word embeddings and other standard techniques were already available, a recent survey discusses the use of contextualized embeddings for semantic change. As a summary of this survey, I am going to present the classification framework proposed to distinguish different approaches and the state-of-the-art solutions.

In general, we can categorize approaches to semantic change that use contextualized embeddings according to three dimensions of analysis: meaning representation, time awareness, and learning modality. Each of these dimensions has two sub-categories that I will present one by one.

Meaning representation concerns how the meaning(s) of a word is represented. Approaches can be form-based or sense-based. �Time awareness concerns how the time of a text is considered. Approaches can be time-oblivious or time-aware. �Learning modality concerns how external knowledge is exploited. Approaches can be supervised or unsupervised.

15 of 55

Meaning representation

form-based approaches

15

LOT Winter School 2024

form-based

sense-based

○ multiple word senses

[...]

Average Pairwise Distance (APD)

Inverted similarity over word prototype (PRT)

APD

PRT

○ degree of polysemy

○ dominant sense

Let's start with meaning representation. Approaches are form-based when they address the problem of semantic change by modeling the degree of polysemy of a word or its dominant meaning. In particular, the most representative approaches are APD and PRT, respectively. The first one estimates the degree of polysemy by computing the average distance between all possible embedding distances. This means that we have collected a set of word usages and corresponding embeddings for the periods C1 and the same for C2. Now, we compute the distances between each embedding of C1 and the embedding of C2, for instance, embedding 1 and embedding 1, …. Then we take the average. On the other hand, PRT simply models semantic change as the cosine distance between word prototypes, i.e., the centroid embeddings for Phi1 and Phi2. Why Inverted? Because some works propose to use the reciprocal of cosine similarity.

16 of 55

Meaning representation

form-based approaches

16

LOT Winter School 2024

def apd(embeddings_t1: np.array, embeddings_t2: np.array, metric: str='cosine'):

pairwise_distances = cdist(embeddings_t1, embeddings_t2, metric=metric) # scipy

return np.mean(pairwise_distances) # numpy

def prt(embeddings_t1: np.array, embeddings_t2: np.array):

word_prototype_t1 = embeddings_t1.mean(axis=0)

word_prototype_t2 = embeddings_t2.mean(axis=0)

return cosine(word_prototype_t1, word_prototype_t2)

17 of 55

Meaning representation

sense-based approaches via clustering

17

LOT Winter School 2024

form-based

sense-based

○ degree of polysemy

○ dominant sense

○ multiple word senses

–

K-Mean

Affinity Propagation

Although powerful, form-based approaches do not fully leverage the potential of contextualized models. Indeed, they do not aim to model the different meanings a word can have. For instance, PRT is a very naive method that resembles the use of static embeddings. In this approach, all the different embeddings of a word are compacted into a single centroid embedding.

In contrast to form-based approaches, sense-based approaches attempt to distinguish the various meanings of a word. This is typically achieved by employing a clustering algorithm over all the embeddings Phi1 and Phi2, considered as a whole. The idea is that each cluster denotes a specific word meaning. Specifically, we can have three types of clusters: clusters containing document/embeddings from the first time period only, clusters containing embeddings from the second time period only, and clusters containing embeddings from both time periods.

The goal is then to compare the distribution of embeddings within each cluster to understand which meaning has changed, gained, or lost.

18 of 55

Meaning representation

sense-based approaches via clustering

18

LOT Winter School 2024

form-based

sense-based

○ degree of polysemy

○ dominant sense

○ multiple word senses

Average Pairwise Distance between sense Prototypes (APDP)

Jensen Shannon Divergence (JSD)

[ 3 , 2 , 0 ]�

[ 0 , 2 , 3 ]

APDP

JSD

Two typical and effective approaches are the use of Average Pairwise Distance between sense prototypes (APDP) and Jensen Shannon Divergence (JSD). The first one leverages the sense prototypes for capturing the shifts. Basically, first we compute the sense prototypes by averaging the embeddings in each clusters, then apply the Average Pairwise Distance between a reduced set of embeddings (i.e., the sense prototypes). That is, we have to compute the distances between each pair of sense prototypes and take the average.

On the other hand, JSD is a symmetric extension of Kullback-Liebler divergence to estimate how much the distribution of embeddings for C1 is different from the one of C2, and vice versa. Considering the previous example, we know that in cluster 1, there are three embeddings from C1, but none from C2. We also know that cluster 2 contains two embeddings from C1 and two embeddings from C2 respectively. Cluster 3 contains only three embeddings from the C2.

19 of 55

Meaning representation

sense-based approaches via clustering

19

LOT Winter School 2024

def clustering_jsd(embeddings_t1: np.array, embeddings_t2: np.array):

embeddings = np.concatenate([embeddings_t1, embeddings_t2], axis=0)

# cluster labels

L = clustering(embeddings) # clustering embeddings as a whole - sklearn

L1, L2 = L[:embeddings_t1.shape[0]], L[embeddings_t1.shape[0]:]

# time-specific distributions

L1_dist, L2_dist = time_specific_distributions(L1, L2)�

return jensenshannon(L1_dist, L2_dist) # scipy

def clustering_apdp(embeddings_t1: np.array, embeddings_t2: np.array, metric: str='cosine'):

embeddings = np.concatenate([embeddings_t1, embeddings_t2], axis=0) �

# cluster labels

L = clustering(embeddings) # clustering embeddings as a whole

L1, L2 = L[:embeddings_t1.shape[0]], L[embeddings_t1.shape[0]:]� unique_L1, unique_L2 = np.unique(L1), np.unique(L2)

sense_prototype_t1 = np.array([embeddings_t1[L1 == label].mean(axis=0) for label in unique_L1])

sense_prototype_t2 = np.array([embeddings_t2[L2 == label].mean(axis=0) for label in unique_L2])

� return apd(sense_prototype_t1 , sense_prototype_t2, metric)

This is a naive implementation. It’s more a pseudocode that requires adaptation based on specific choices. For example, the kind of clustering we want to use and the clustering parameters we want to adopt. Consider JSD as an example. [Don’t worry about the implementation of JSD, we can leverage implementation from python libraries.] Initially, we concatenate the embeddings from both time periods into a single array, treating the embeddings collectively. A single clustering activity is then performed on this array. Numerous clustering algorithms, ranging from K-Means to DBSCAN and HDBSCAN, have been proposed. Affinity Propagation, however, is often the preferred choice due to the high computational cost associated with DBSCAN, HDBSCAN, and other hierarchical algorithms.

While simpler algorithms like K-Means are valuable, they require making unrealistic assumptions about the number of meanings associated with a word in practice. Do you all know K-Means? Basically, it partitions a dataset into a predetermined number of distinct clusters based on the similarity of data points. Techniques to estimate a reasonable number of clusters exists, but they often lack scalability. We will discuss it later. Affinity Propagation emerges as a valuable alternative since the number of clusters is determined without the need to preset it in advance.

It's worth noting that the scikit-learn (sklearn) library provides numerous implementations of clustering algorithms. Following the clustering process, a set of labels is obtained, associating each embedding with a specific cluster ID. These labels are then utilized to compute time-specific distributions for C1 and C2, ultimately leading to the computation of JSD.

Similarly, for APDP, we cluster the embeddings, assign labels, and for each label, compute the sense prototypes for C1 and C2. Subsequently, the previously implemented APD is utilized to compute semantic changes based on these sense prototypes.

20 of 55

Time awareness

time-oblivious approaches

20

LOT Winter School 2024

time-oblivious

time-aware

○ contextualization

○ diachronic data

○ time embedding

you are a cheerful and lively person.

Thou art a gay and jovial fellow.

1. The context is always time-specific.

2. The model is trained on diachronic data.

1890

1990

Modern

Historical

We previously mention how this dimension relates to the consideration of time in a text. To illustrate, let's consider the same sentence from two distinct time periods. “Thou art a gay and jovial fellow.” - “you are a cheerful and lively person”��Time-oblivious approaches are built upon one of two strong assumptions: �1. The assumption that the context is always time-specific and since contextualized models can infer word meanings based on the context, the model should be able to contextualize new meanings correctly. While this assumption may seem ideal, it generally proves untrue because the contextualization capability of the model is limited to the word meanings it was trained on. �2. The assumption that the model is trained on diachronic data, allowing it to generalize meanings over time. ��New BERT models can be trained or fine-tuned on historical data, in order to make BERT completely understand even the past meanings.��The approaches that we have seen so far can be used according to this assumption.

21 of 55

Time awareness

time-aware approaches

21

LOT Winter School 2024

time-oblivious

time-aware

○ contextualization

○ diachronic data

○ time embedding

you are a cheerful and lively person.

Thou art a gay and jovial fellow.

The model cannot generalize across time

1890

1990

Þu eart glæd and wynsum gefera

1000

22 of 55

Time awareness

time-aware approaches

22

LOT Winter School 2024

time-oblivious

time-aware

○ contextualization

○ stability

○ time embedding

Temporal Referencing Time Masking

Thou art a gay_[1890] and jovial fellow.�

Gay_[2020] pride radiates love and acceptance.

Temporal Attention

Dynamic Contextualized Word Embeddings

How can we provide time information to the models? There are several alternatives. I will discuss only some. A basic strategies that have been employed, even for static word embeddings is ‘temporal referencing’. Temporal referencing involves a specific processing and fine-tuning. Initially, we preprocess our data by adding a special tag to each occurrence of a target word we wish to investigate. This tag indicates the time period of the sentence. For instance, if we want to examine occurrences of the word ‘gay’ in both 1890 and 2020, we add the tags '1890' and '2020' to distinguish the time of each occurrence. Consequently, in this example, we introduce the tags 'Gay_1890' and 'Gay_2020.' Subsequently, we incorporate these tokens, such as 'gay_[1890]' and 'gay_[2020],' into the model's vocabulary. We then train or fine-tune the model to acquire new representations (i.e., embeddings) for these specific tokens. We use the embeddings of these time-specific tokens for capturing semantic change.��Time masking is a similar strategy aimed at addressing some of the drawbacks associated with temporal references. These drawbacks include the need to add a tag for each target we want to investigate, resulting in the addition of numerous tokens to our vocabulary, considering that each target may have different forms. In time masking, a special token is placed at the beginning of the sentence, irrespective of the token we aim to investigate. This enables the model to learn to represent words in a sentence based on the time periods associated with the sentence. Alternatively, other approaches involve modifying the basic architecture in a more complex manner to inject time directly into the attention mechanism or into the embeddings. Additional details can be found in the corresponding papers

23 of 55

Time awareness

time-aware approaches

23

LOT Winter School 2024

Time-referencing�corpus1 = temporal_referencing(corpus1, target, tag)

corpus2 = temporal_referencing(corpus2, target, tag)

new_tokens_with_tag = extract_new_tokens(corpus1) + extract_new_tokens(corpus2)

add_token_to_vocab(model, new_tokens_with_tag )

fine_tuned(model)��embeddings = embedding_extraction(model)�semantic_change_approach(embeddings)

Time-masking�corpus1 = temporal_referencing(corpus1, tag)

corpus2 = temporal_referencing(corpus2, tag)

new_tokens_with_tag = [1, 2]

add_token_to_vocab(model, new_tokens_with_tag )

fine_tuned(model)��embeddings = embedding_extraction(model)�semantic_change_approach(embeddings)

24 of 55

Learning modality

supervised

24

LOT Winter School 2024

supervised

unsupervised

○ lexicographic supervision

○ manual supervision

○ only text

Gloss Reader

WordNet

Oxford Dictionary

Diachronic Senses

XL-LEXEME

WiC benchmarks

Deep Mistake

Inclined planes simplify lifting on slopes.

The plane is inclined at a slight angle.

Passengers quickly boarded the plane

[...] plane [...]

Word-in-Context (WiC)

According to this dimension, approaches can be classified into supervised and unsupervised categories. Supervised approaches utilize various forms of supervision to model semantic change. This supervision may be i) lexicographic, involving the use of thesauri, vocabularies, dictionaries, or ii) manual supervision, such as datasets containing examples of semantic change with labels derived from annotators. �

GlossReader is a model that exploits glosses from WordNet to embed not only words in the embedding space but also sense definitions. The sense of a word is then inferred by considering the closer sense embeddings to a specific word occurrence embedding. Interestingly, the paper explores the transfer learning capability of contextualized models. A multilingual model is trained on English data and then applied to Russian data. Refer to the paper for further details. ��Alternatively, instead of relying on WordNet, which is a synchronic collection of word senses, one can choose to exploit diachronic lexicographic supervision. For example, the Oxford Dictionary is employed for English. For each meaning, the dictionary contains word usage examples. A strategy is to consider the embeddings of those word usages as 'sense' indicators. And then proceed by computing distances between sense indicators of a particular time periods and word embeddings. Of course, few diachronic resources are available. Few resource languages cannot leverage this approaches.��DeepMistake and XL-LEXEME are crucial approaches, with XL-LEXEME achieving state-of-the-art results in evaluations across all available benchmarks. Both models are trained using the contrastive loss over datasets of Word-in-Context. These datasets consist of pairs of sentences where a word is used in two different contexts. The label can be True or False, depending on whether the word has the same meaning or a different meaning in the contexts. The contrastive loss is employed to fine-tune the model, minimizing the distances between word occurrences related to similar meanings and maximizing the distances between word occurrences related to different meanings.

25 of 55

Learning modality

supervised

25

LOT Winter School 2024

Inclined <t> planes </t> simplify lifting on slopes.

The <t> planes </t> is inclined at a slight angle.

The <t> plane </t> is inclined at a slight angle.

Passengers quickly boarded the <t> plane </t>

[...] plane [...]

Word-in-Context (WiC)

https://www.sbert.net/examples/applications/cross-encoder/README.html

26 of 55

Learning modality

unsupervised

26

LOT Winter School 2024

supervised

unsupervised

○ lexicographic supervision

○ manual supervision

○ only text

APD

PRT

Clustering + APDP

Clustering + JSD

…

XL-LEXEME

BERT

mBERT

XLM-R

pre-trained model + LSC approach

27 of 55

Scalability and interpretability

issues

References:�

A Survey on Contextualised Semantic Shift Detection. �Scalable and Interpretable Semantic Change Detection

27

LOT Winter School 2024

28 of 55

Scalability

Memory consumption

28

The plane flew above the clouds.

[ "[CLS]", "the", "plane", "flew", "above", "the", "clouds", ".", "[SEP]" ]

[ ⚪, ⚪, 🔵, ⚪, ⚪, ⚪, ⚪, ⚪, ⚪ ]

An embedding contains 768 floats

A survey

Solution:

Process one target word at a time
Dimensionality reduction of the embeddings
Random sampling the occurrences

LOT Winter School 2024

A floating-point in Python requires 8 B

A word appearing 500.000 times

requires 3Gb + overhead

29 of 55

Scalability

Computation time

29

A vocabulary may contain more than 500.000 words�Clustering and dimensionality reduction take time

A survey

Solution:

We need a GPU
Select a small set of target words
Random sampling the occurrences

LOT Winter School 2024

30 of 55

Interpretability

form-based approaches

30

A survey

–

form-based approaches are not interpretable

Solution:

Quantify semantic change across the entire vocabulary.
Use to select a small set of targets

LOT Winter School 2024

31 of 55

Interpretability

word meaning representation

31

A survey

–

Changes in contextual variance

Clusters of word meanings are clusters of “sense nodules” - i.e., lumps of meaning with greater stability under contextual changes (Cruse, 2000)

A word may change its context without changing its meaning

Solution:�

? ? ?

LOT Winter School 2024

Then, let’s consider sense-based approaches. We said that the intuition is that each cluster denotes a specific word meaning. Unfortunately this is only ideal. More and more work studied the generated clusters, concluding that they represents more ‘sense nodules’ than word meanings. Where sense-nodules are defined as ‘lumps of meanings with greater stability under contextual changes’.��For example, existing work reported to obtained too much clusters by using AffinityPropagation. Let’s consider the word ‘plane’ and the meaning of ‘airplane’. This an extreme case, that acts only as an example, but it could be the case that three different clusters are generated for ‘green plane’, ‘organge plane’, and ‘red plane’ if the model recognize these three bigram as important pattern.��Then what’s the solution? Unfortunately, there is no a real solution yet

32 of 55

Interpretability

word meaning description

32

A survey

list of keywords

close reading

random sampling

Solution:�

generating natural language definitions of contextualised word usage

Definitions

LOT Winter School 2024

we cannot assume that our clustering activity performed perfectly. Let’s then assume it worked somewhat well. We theoretically interpret the results, but can also do it in practice? A cluster may contains hundred of sentences. Then for understand that a cluster is related to a specific meaning we should check the sentences occurring inside that cluster through close reading. This is very time demanding as clusters may contains some noise, multiple instances, and a word can be associated to multiple clusters.

Some solution have been proposed. For example, we can exploit standard approaches from Information Retrieval to collect a list of keywords representing each clusters. This way, we can infer the meaning directly from the keywords.��Another solution is to randomly sample a small number of sentences from each clusters and base our meaning inference on them.��Those are basic approaches but we should be aware they are very limited. Especially when meanings are wrongly clustered.

A new solution that is emerging along with the recently available LLMs consists of leveraging their generative capability to derived word definitions for particular word occurrences. For example, we can prompt ChatGPT by asking to generate a definition for a word in context.

As a matter of fact, this new approaches open a new set of possibilities for modeling semantic change.

33 of 55

Lexical Semantic Change over multiple time periods

References:�

Studying Word Meaning Evolution through Incremental Semantic Shift Detection: A Case Study of Italian Parliamentary Speeches

33

LOT Winter School 2024

34 of 55

Word meaning evolution

APD

34

A survey

. . .

1750

1800

1900

1950

2024

. . .

1850

LOT Winter School 2024

LSC

35 of 55

Word meaning evolution

APD

35

36 of 55

Word meaning evolution

PRT

36

A survey

. . .

1750

1800

1900

1950

2024

. . .

1850

LOT Winter School 2024

LSC

Evolving

Average

37 of 55

Word meaning evolution

Clustering

37

A survey

. . .

1750

LSC

1800

LSC

1900

1950

2024

. . .

LSC

. . .

1850

LSC

LOT Winter School 2024

38 of 55

Word meaning evolution

Clustering alignment

38

–

LOT Winter School 2024

1750

1800

1850

39 of 55

Word meaning evolution

Clustering alignment

39

–

LOT Winter School 2024

1750

1800

40 of 55

Word meaning evolution

Massive clustering

40

. . .

1750

1800

1900

1950

2024

. . .

1850

Semantic change in interaction

LOT Winter School 2024

41 of 55

Word meaning evolution

Evolutionary clustering

41

WiDiD

introduction

In review 🤞

Incremental Affinity Propagation based on Cluster Consolidation and Stratification

1750

1800

What is Done is Done (WiDiD)

and cannot be changed

LOT Winter School 2024

clustering

evaluation

WiDiD

evaluation

A superior alternative, from my perspective, is to employ an evolutionary clustering algorithm that allows us to model word meanings in a diachronic manner. Why is this a better alternative? Firstly, because it can be utilized for both static and dynamic corpora. Secondly, it represents a scalable option, reducing time and memory consumption.

It is necessary to distinguish between two types of evolutionary clustering. Some algorithms integrate new documents into the clustering, treating them as new evidence that can alter the previous clustering results for improvement. Other algorithms freeze the existing clustering and create new clusters based on it. We are interested in using the latter for the problem of lexical semantic change because we aim to avoid altering past representations based on future evidence. The goal is to individually model the future through the lens of the past, but not vice versa. Then the ideas is that what is done in the past cannot be changed.

This concept forms the basis of a clustering algorithm I recently introduced, known as WiDiD. WiDiD is straightforward. Essentially, it involves conducting a single clustering activity for each time period, such as one for 1750 and another for 1800. Instead of utilizing a clustering algorithm each time, we consider the sense prototypes of previous clusters as additional embeddings for the next iteration. These sense prototypes are then clustered with the embeddings of the subsequent time period. We can expand the sense prototypes for incorporating all the embeddings from earlier time periods, which will be still grouped together.

Different situations may arise. For instance, a previously created cluster can be integrated with new embeddings, or it remains unintegrated, forming a singleton cluster. Another possibility is the merger of two sense prototypes into a single cluster, which is subsequently integrated with further embeddings from new time periods. This approach allows us to study the historical evolution of word meanings while maintaining scalability, as we consider embeddings from specific time periods along with sense prototypes from previous time periods.

42 of 55

Word meaning evolution

WiDiD drawbacks

42

Drawbacks

From evolutionary clustering to online evolutionary clustering

A sense prototype is considered as a singular word usage instance.

Word can lose meanings

Change is gradual

Assign higher importance to sense prototypes.

Solutions

Eliminate aging clusters that are no longer integrated

LOT Winter School 2024

In review 🤞

Incremental Affinity Propagation based on Cluster Consolidation and Stratification

clustering

evaluation

WiDiD

evaluation

WiDiD

introduction

43 of 55

Word meaning evolution

Cluster Monitoring

43

LOT Winter School 2024

Time-series

Degree of

Semantic Change

Time periods

44 of 55

CIAO!

Francesco Periti

3^rdyear - PhD Student

Natural Language Processing, Distributional Semantic Models, Lexical Semantic Change

francesco.periti@unimi.it

44

LOT Winter School 2024

45 of 55

Problem definition

45

gay

From cheerful

To homosexual

Lexical Semantic Change

LOT Winter School 2024

monitoring

Let’s start by refreshing the problem. Let’s do it in a different way. Consider a large, digitized, diachronic corpus that contains documents from various time periods. Formally, we can view the set of documents from each time period as a subcorpus of the entire diachronic corpus. For example, here we have three distinct subcorpora (i.e. C1, C2, C3) associated to specific time periods, and we define the diachronic corpus (i.e. C) as the union of these subcorpora. Note that these time periods are just examples; we can choose to split the diachronic corpus in different ways, depending on the time granularity we are interested to analyze. ��The goal is to identify words that exhibit semantic change in the diachronic corpus across different time periods. For instance, 'manufacturer' has changed from meaning 'to make by hand' to meaning 'to make by machine' during the industrial revolution. Similarly, the word 'gay' has changed in the last century, shifting from meaning 'cheerful' to primarily meaning 'homosexual' . Of course, we can consider different kinds of changes.

This is the general problem, but it is too broad and complex to be addressed in this way, by considering multiple time periods at once. Thus, generally, the focus is narrowed to detecting semantic changes in two time periods only (i.e. a single time interval). It still can be easily extended to scenarios involving multiple time periods.

46 of 55

46

LOT Winter School 2024

Graded Change Detection

over time

Let’s start by refreshing the problem. Let’s do it in a different way. Consider a large, digitized, diachronic corpus that contains documents from various time periods. Formally, we can view the set of documents from each time period as a subcorpus of the entire diachronic corpus. For example, here we have three distinct subcorpora (i.e. C1, C2, C3) associated to specific time periods, and we define the diachronic corpus (i.e. C) as the union of these subcorpora. Note that these time periods are just examples; we can choose to split the diachronic corpus in different ways, depending on the time granularity we are interested to analyze. ��The goal is to identify words that exhibit semantic change in the diachronic corpus across different time periods. For instance, 'manufacturer' has changed from meaning 'to make by hand' to meaning 'to make by machine' during the industrial revolution. Similarly, the word 'gay' has changed in the last century, shifting from meaning 'cheerful' to primarily meaning 'homosexual' . Of course, we can consider different kinds of changes.

This is the general problem, but it is too broad and complex to be addressed in this way, by considering multiple time periods at once. Thus, generally, the focus is narrowed to detecting semantic changes in two time periods only (i.e. a single time interval). It still can be easily extended to scenarios involving multiple time periods.

47 of 55

47

LOT Winter School 2024

Monitoring

Graded Change Detection

over time

Change Point

Detection

Let’s start by refreshing the problem. Let’s do it in a different way. Consider a large, digitized, diachronic corpus that contains documents from various time periods. Formally, we can view the set of documents from each time period as a subcorpus of the entire diachronic corpus. For example, here we have three distinct subcorpora (i.e. C1, C2, C3) associated to specific time periods, and we define the diachronic corpus (i.e. C) as the union of these subcorpora. Note that these time periods are just examples; we can choose to split the diachronic corpus in different ways, depending on the time granularity we are interested to analyze. ��The goal is to identify words that exhibit semantic change in the diachronic corpus across different time periods. For instance, 'manufacturer' has changed from meaning 'to make by hand' to meaning 'to make by machine' during the industrial revolution. Similarly, the word 'gay' has changed in the last century, shifting from meaning 'cheerful' to primarily meaning 'homosexual' . Of course, we can consider different kinds of changes.

This is the general problem, but it is too broad and complex to be addressed in this way, by considering multiple time periods at once. Thus, generally, the focus is narrowed to detecting semantic changes in two time periods only (i.e. a single time interval). It still can be easily extended to scenarios involving multiple time periods.

48 of 55

48

LOT Winter School 2024

monitoring

Change Point Detection

Let’s start by refreshing the problem. Let’s do it in a different way. Consider a large, digitized, diachronic corpus that contains documents from various time periods. Formally, we can view the set of documents from each time period as a subcorpus of the entire diachronic corpus. For example, here we have three distinct subcorpora (i.e. C1, C2, C3) associated to specific time periods, and we define the diachronic corpus (i.e. C) as the union of these subcorpora. Note that these time periods are just examples; we can choose to split the diachronic corpus in different ways, depending on the time granularity we are interested to analyze. ��The goal is to identify words that exhibit semantic change in the diachronic corpus across different time periods. For instance, 'manufacturer' has changed from meaning 'to make by hand' to meaning 'to make by machine' during the industrial revolution. Similarly, the word 'gay' has changed in the last century, shifting from meaning 'cheerful' to primarily meaning 'homosexual' . Of course, we can consider different kinds of changes.

This is the general problem, but it is too broad and complex to be addressed in this way, by considering multiple time periods at once. Thus, generally, the focus is narrowed to detecting semantic changes in two time periods only (i.e. a single time interval). It still can be easily extended to scenarios involving multiple time periods.

49 of 55

49

LOT Winter School 2024

Change Point Detection

Word Sense Induction

gay

From cheerful

To homosexual

Interpretation

Let’s start by refreshing the problem. Let’s do it in a different way. Consider a large, digitized, diachronic corpus that contains documents from various time periods. Formally, we can view the set of documents from each time period as a subcorpus of the entire diachronic corpus. For example, here we have three distinct subcorpora (i.e. C1, C2, C3) associated to specific time periods, and we define the diachronic corpus (i.e. C) as the union of these subcorpora. Note that these time periods are just examples; we can choose to split the diachronic corpus in different ways, depending on the time granularity we are interested to analyze. ��The goal is to identify words that exhibit semantic change in the diachronic corpus across different time periods. For instance, 'manufacturer' has changed from meaning 'to make by hand' to meaning 'to make by machine' during the industrial revolution. Similarly, the word 'gay' has changed in the last century, shifting from meaning 'cheerful' to primarily meaning 'homosexual' . Of course, we can consider different kinds of changes.

This is the general problem, but it is too broad and complex to be addressed in this way, by considering multiple time periods at once. Thus, generally, the focus is narrowed to detecting semantic changes in two time periods only (i.e. a single time interval). It still can be easily extended to scenarios involving multiple time periods.

50 of 55

50

⬖

alignment

(C)

alignment

⬖

. . .

embeddings

clustering

(A)

(B)

Clustering over consecutive time intervals

⬖

Let’s start by refreshing the problem. Let’s do it in a different way. Consider a large, digitized, diachronic corpus that contains documents from various time periods. Formally, we can view the set of documents from each time period as a subcorpus of the entire diachronic corpus. For example, here we have three distinct subcorpora (i.e. C1, C2, C3) associated to specific time periods, and we define the diachronic corpus (i.e. C) as the union of these subcorpora. Note that these time periods are just examples; we can choose to split the diachronic corpus in different ways, depending on the time granularity we are interested to analyze. ��The goal is to identify words that exhibit semantic change in the diachronic corpus across different time periods. For instance, 'manufacturer' has changed from meaning 'to make by hand' to meaning 'to make by machine' during the industrial revolution. Similarly, the word 'gay' has changed in the last century, shifting from meaning 'cheerful' to primarily meaning 'homosexual' . Of course, we can consider different kinds of changes.

This is the general problem, but it is too broad and complex to be addressed in this way, by considering multiple time periods at once. Thus, generally, the focus is narrowed to detecting semantic changes in two time periods only (i.e. a single time interval). It still can be easily extended to scenarios involving multiple time periods.

51 of 55

51

alignment

⬖

clustering

alignment

(A)

(D)

(E)

alignment

clustering

(B)

(C)

Clustering over consecutive time periods

⬖

Let’s start by refreshing the problem. Let’s do it in a different way. Consider a large, digitized, diachronic corpus that contains documents from various time periods. Formally, we can view the set of documents from each time period as a subcorpus of the entire diachronic corpus. For example, here we have three distinct subcorpora (i.e. C1, C2, C3) associated to specific time periods, and we define the diachronic corpus (i.e. C) as the union of these subcorpora. Note that these time periods are just examples; we can choose to split the diachronic corpus in different ways, depending on the time granularity we are interested to analyze. ��The goal is to identify words that exhibit semantic change in the diachronic corpus across different time periods. For instance, 'manufacturer' has changed from meaning 'to make by hand' to meaning 'to make by machine' during the industrial revolution. Similarly, the word 'gay' has changed in the last century, shifting from meaning 'cheerful' to primarily meaning 'homosexual' . Of course, we can consider different kinds of changes.

This is the general problem, but it is too broad and complex to be addressed in this way, by considering multiple time periods at once. Thus, generally, the focus is narrowed to detecting semantic changes in two time periods only (i.e. a single time interval). It still can be easily extended to scenarios involving multiple time periods.

52 of 55

52

clustering

(A)

Clustering over all time periods

. . .

embeddings

⬖

Let’s start by refreshing the problem. Let’s do it in a different way. Consider a large, digitized, diachronic corpus that contains documents from various time periods. Formally, we can view the set of documents from each time period as a subcorpus of the entire diachronic corpus. For example, here we have three distinct subcorpora (i.e. C1, C2, C3) associated to specific time periods, and we define the diachronic corpus (i.e. C) as the union of these subcorpora. Note that these time periods are just examples; we can choose to split the diachronic corpus in different ways, depending on the time granularity we are interested to analyze. ��The goal is to identify words that exhibit semantic change in the diachronic corpus across different time periods. For instance, 'manufacturer' has changed from meaning 'to make by hand' to meaning 'to make by machine' during the industrial revolution. Similarly, the word 'gay' has changed in the last century, shifting from meaning 'cheerful' to primarily meaning 'homosexual' . Of course, we can consider different kinds of changes.

This is the general problem, but it is too broad and complex to be addressed in this way, by considering multiple time periods at once. Thus, generally, the focus is narrowed to detecting semantic changes in two time periods only (i.e. a single time interval). It still can be easily extended to scenarios involving multiple time periods.

53 of 55

53

change point detection

(A)

clustering

(B)

Clustering over specific time intervals/periods

54 of 55

54

update

⬖

update

⬖

clustering

incremental

clustering

update

(A)

(C)

update

incremental

clustering

(B)

Incremental clustering over consecutive time periods

⬖

55 of 55

55

(A)