1 of 29

Lecture 11 : Word Representation

2 of 29

How to represent a word? Distributional Hypothesis
Sparse vs Dense Vectors
Reasoning with word vectors

3 of 29

How to represent a word?

In traditional NLP / IR, words are treated as discrete symbols.

Vector dimension = number of words in vocabulary (e.g., 500,000)

4 of 29

Problems with words as discrete symbols

Example: In web search, if user searches for “Baltimore motel”, we would like

to match documents containing “Baltimore hotel”. But

The vectors are orthogonal, and there is no natural notion of similarity between

one-hot vectors!

How can we define the word meaning that encodes

this notion of similarity?

Can we use shorter vectors?

5 of 29

Ludwig Wittgenstein

"The meaning of a word is its use in the language"

Source: https://web.stanford.edu/~jurafsky/slp3.

6 of 29

Distributional Hypothesis: Let's define words by their usages

One way to define "usage":

words are defined by their environments (the words around them)

Zellig Harris (1954):

If A and B have almost identical environments we say that they are synonyms.

Source: https://web.stanford.edu/~jurafsky/slp3.

7 of 29

What does recent English borrowing ongchoi mean?

Suppose you see these sentences:

Ongchoi is delicious sautéed with garlic.
Ongchoi is superb over rice
Ongchoi leaves with salty sauces

And you've also seen these:

…spinach sautéed with garlic over rice
Chard stems and leaves are delicious
Collard greens and other salty leafy greens

Conclusion:

Ongchoi is a leafy green like spinach, chard, or collard greens

We could conclude this based on words like "leaves" and "delicious" and

"sauteed"

Source: https://web.stanford.edu/~jurafsky/slp3.

8 of 29

Ongchoi: Ipomoea aquatica "Water Spinach"

Yamaguchi, Wikimedia Commons, public domain

空心菜

kangkong

rau muống

…

Source: https://web.stanford.edu/~jurafsky/slp3.

9 of 29

Defining meaning as a point in space based on

distribution

Each word = a vector

Similar words are "nearby in semantic space"

We build this space automatically by seeing which words are nearby in text

Source: https://web.stanford.edu/~jurafsky/slp3.

10 of 29

We define meaning of a word as a vector

Called an "embedding" because it's embedded into a space

The standard way to represent meaning in NLP

Modern NLP algorithms use embeddings as the representation of word meaning

Fine-grained model of meaning for similarity

Source: https://web.stanford.edu/~jurafsky/slp3.

11 of 29

Intuition: why vectors?

Consider classical algorithms:

With words, a feature is a word identity

Feature 5: 'The previous word was "terrible"'
requires exact same word to be in training and test (dreadful?)

With embeddings:

Feature is a word vector
'The previous word was vector [35,22,17…]
Now in the test set we might see a similar vector [34,21,14]
We can generalize to similar but unseen words!!!

Source: https://web.stanford.edu/~jurafsky/slp3.

12 of 29

Encoding similarity: Distributional Vectors

13 of 29

Let’s take an example

14 of 29

Distributional Vectors: Computing Similarity

15 of 29

Context Weighting via association measures

16 of 29

Pointwise Mutual Information (PMI)

Statistically measure whether two words co-occur frequently (relative to their global frequencies)

17 of 29

Even simpler: Can use documents as context

18 of 29

Context weighting: documents as context

19 of 29

Sparse versus dense vectors

One-hot (or distributional) vectors (includes tf-idf, PMI, etc.)

long (length |V|= 20,000 to 50,000)
sparse (most elements are zero)

Alternative: learn vectors which are

short (length 50-1000)
dense (most elements are non-zero)

Source: https://web.stanford.edu/~jurafsky/slp3.

20 of 29

Sparse versus dense vectors

Why dense vectors?

Short vectors may be easier to use as features in machine learning (fewer weights to tune)
Dense vectors may generalize better than explicit counts
Dense vectors may do better at capturing synonymy:

car and automobile are synonyms; but are distinct dimensions

a word with car as a neighbor and a word with automobile as neighbor should be similar, but aren't

In practice, they work better

Source: https://web.stanford.edu/~jurafsky/slp3.

21 of 29

Common methods for getting short dense vectors

Singular Value Decomposition (SVD)

A special case of this is called LSA – Latent Semantic Analysis

“Neural Language Model”-inspired models

Word2vec (skipgram, CBOW), GloVe

Alternative to these "static embeddings":

Contextual Embeddings (ELMo, BERT)
Compute distinct embeddings for a word in its context
Separate embeddings for each token of a word

Source: https://web.stanford.edu/~jurafsky/slp3.

22 of 29

Word2vec

Instead of counting how often each word w occurs near "apricot"

Train a classifier on a binary prediction task:

Is w likely to show up near "apricot"?

We don’t actually care about this task

But we'll take the learned classifier weights as the word embeddings

Big idea: self-supervision:

A word c that occurs near apricot in the corpus acts as

the gold "correct answer" for supervised learning

No need for human labels
Bengio et al. (2003); Collobert et al. (2011)

Source: https://web.stanford.edu/~jurafsky/slp3.

The intuition of word2vec is that instead of counting how often each word w oc- curs near, say, apricot, we’ll instead train a classifier on a binary prediction task: “Is word w likely to show up near apricot?” We don’t actually care about this prediction task; instead we’ll take the learned classifier weights as the word embeddings. The revolutionary intuition here is that we can just use running text as implicitly supervised training data for such a classifier; a word c that occurs near the target word apricot acts as gold ‘correct answer’ to the question “Is word c likely to show up near apricot?” This method, often called self-supervision, avoids the need for any sort of hand-labeled supervision signal. This idea was first proposed in the task of neural language modeling, when Bengio et al. (2003) and Collobert et al. (2011) showed that a neural language model (a neural network that learned to predict the next word from prior words) could just use the next word in running text as its supervision signal, and could be used to learn an embedding representation for each word as part of doing this prediction task. word2vec is a much simpler model than the neural network language model, in two ways. First, word2vec simplifies the task (making it binary classification instead of word pre- diction). Second, word2vec simplifies the architecture (training a logistic regression classifier instead of a multi-layer neural network with hidden layers that demand more sophisticated training algorithms)

23 of 29

Distributional Representation

Take a vector with several hundred dimensions (say 1000).
Each word is represented by a distribution of weights across those elements.
So instead of a one-to-one mapping between an element in the vector and a word, the representation of a word is spread across all of the elements in the vector, and
Each element in the vector contributes to the definition of many words.

24 of 29

Reasoning with Word Vectors

It has been found that the learned word representations in fact capture meaningful syntactic and semantic regularities in a very simple way.
Specifically, the regularities are observed as constant vector offsets between pairs of words sharing a particular relationship.

25 of 29

Reasoning with Word Vectors

Perhaps more surprisingly, we find that this is also the case for a variety of semantic relations.

A simple vector offset method based on cosine distance shows the relation.

26 of 29

Vector Offset for Relations

Singular - Plural

Gender

27 of 29

Analogy Testing

28 of 29

Daniel Jurafsky and James H. Martin. 2024. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, 3rd edition. Online manuscript released August 20, 2024. [Chapter 6]

1 of 29

2 of 29

3 of 29

4 of 29

5 of 29

6 of 29

7 of 29

8 of 29

9 of 29

10 of 29

11 of 29

12 of 29

13 of 29

14 of 29

15 of 29

16 of 29

17 of 29

18 of 29

19 of 29

20 of 29

21 of 29

22 of 29

23 of 29

24 of 29

25 of 29

26 of 29

27 of 29

28 of 29

29 of 29