1 of 23

embed(

training word,

context word,

continuous year)

Deep Neural Models of Semantic Shift

Alex Rosenfeld, Katrin Erk

CS 6742 Spring 2024 | Presented by B.W. | Thursday, Feb 15, 2024

2 of 23

Background & Contribution

3 of 23

4 of 23

5 of 23

eval(model, input, gold)

6 of 23

eval(model, input, )

synthetic

gold

7 of 23

Models

8 of 23

SGNS Models

9 of 23

Previous Models

as baselines

LargeBin: Binning by Decade. Time is partitioned into decades and an SGNS model is trained on each decade’s worth of data.
SmallBinPreInit: Preinitialization. Time is partitioned into years and an SGNS model is trained on each year’s worth of data, but preinitializing the model for a given time bin with the vectors of the preceding time bin.
SmallBinReg: Prior and Transition Probabilities. Apply a logarithm in constructing the estimate, which transforms the joint probability into the SGNS loss function with four regularizers, which

discourage the vectors from growing too large
discourage a vector variable from deviating from the previous bin’s vectors

10 of 23

Proposed Model

DiffTime

11 of 23

Proposed Model

DiffTime

timevec(t) encodes temporal information as a vector
M_W encodes lexical information as a matrix.
The target vector for w at time t, use_W(w, t), is found by combining Trans_w and timevec(t).
Context version use_C (c, t) is the same except that it has its own embedding layer.

Trans = transition probabilities

The ﬁrst component of our model is a two-layer feed-forward neural network with tanh activation functions. These layers take a time t as input and produces a time embedding timevec(t) as output of those layers:
The second component incorporates wordspeciﬁc information into our model. For use W (w, t), each target word w has a target vector representation w.~ The vector w~ is then transformed into a linear transformation Trans w , which in the third component is applied to the time embedding timevec(t). We do this via a modiﬁed linear layer where the weights are a three dimensional tensor, the biases are a matrix and the output is a matrix:
The third component combines the wordindependent time embedding timevec(t) and the time-independent linear transformation Transw together to produce the ﬁnal result.

12 of 23

Proposed Model

DiffTime

Trans = transition probabilities

The ﬁrst component of our model is a two-layer feed-forward neural network with tanh activation functions. These layers take a time t as input and produces a time embedding timevec(t) as output of those layers:
The second component incorporates wordspeciﬁc information into our model. For use W (w, t), each target word w has a target vector representation w.~ The vector w~ is then transformed into a linear transformation Trans w , which in the third component is applied to the time embedding timevec(t). We do this via a modiﬁed linear layer where the weights are a three dimensional tensor, the biases are a matrix and the output is a matrix:
The third component combines the wordindependent time embedding timevec(t) and the time-independent linear transformation Transw together to produce the ﬁnal result.

13 of 23

Data

Google Books ngram corpus - English fictions

(training word, context word, year)

(training word, context word, year bin)

(target word, context word, year, frequency)

14 of 23

Evaluations

15 of 23

Existing Evaluation

Synchronic Accuracy

16 of 23

Existing Evaluation

Synchronic Accuracy on the MEN word similarity task

17 of 23

Evaluation

New Synthetic Task

banana◦lobster

18 of 23

Evaluation

New Synthetic Task

banana◦lobster

19 of 23

Evaluation

New Synthetic Task

20 of 23

Evaluation

New Synthetic Task

21 of 23

Measure of Word Changes

22 of 23

Extra Find

Automatic extraction of time periods

23 of 23

Closing Words