Lecture 13 : Learning Word Representation - II

• Skip-gram: Negative Sampling, Hierarchical Softmax
• Glove Word Vectors

​

Issues with word2Vec

Negative Sampling: Formulation

Negative Sampling: Objective function

Negative Sampling: Objective function

Skip-Gram Training data

​

…lemon, a [tablespoon of apricot jam, a] pinch…

c1 c2 [target] c3 c4

​

​

7

Skip-gram classifier – Intuition

8

�embedding similarity high ⇒ probability high too

Skip-gram model

9

Given a target word w & its context window of L words c_{1:L}, assigns a probability based on how similar this context window is to the target word

​

The probability is based on applying the sigmoid function to the dot product of the embeddings of the target word with each context word

​

​

Skip-gram learning algorithm

10

Given a large corpus, positive and negative examples, & randomly initialized embeddings

Maximize the emb. similarity of the real target-context word pairs (w,c_pos) drawn from the corpus

​

Maximize the emb. distance between target noncontext word pairs (w,c_neg) drawn from the negative examples

​

k is a hyperparameter �∈ {2,...,5} for larger corpora

Skip-gram learning algorithm

– Stochastic gradient descent

11

Hierarchical Softmax: Formulation

12

Hierarchical Softmax: Formulation

13

Hierarchical Softmax: Formulation

14

Hierarchical Softmax: Try this Problem

15

Suppose you are using hierarchical softmax to compute the probability of the context word given the center word. Suppose, at a given time during training, the center word embeddings be [0.5 0.3 -0.4 0.2], and the path to the context word goes via three nodes starting from root (left, right, right) having embeddings as [0.3 0.1 -0.1 0.05], [-0.4 0.5 0.3 -0.6] and [0.5 0.3 -0.8 -0.9], respectively. What will be the probability of the context word given the center word?

Two sets of embeddings

Skip-Gram with negative sampling (SGNS) learns two sets of embeddings

Target embeddings matrix W

Context embedding matrix C

It's common to just add them together, representing word i as the vector w_i + c_i

Summary: How to learn word2vec (skip-gram) embeddings

Start with V random d-dimensional vectors as initial embeddings

Train a classifier based on embedding similarity

• Take a corpus and take pairs of words that co-occur as positive examples
• Take pairs of words that don't co-occur as negative examples
• Train the classifier to distinguish these by slowly adjusting all the embeddings to improve the classifier performance
• Throw away the classifier code and keep the embeddings.

Why not capture co-occurrence directly?

Co-occurrence vectors

Glove Formulation

GloVe Formulation

log( X_i ) can be absorbed in bias

but it weights all co-occurrences equally

f(x) should be relatively small for large values of x, so that frequent co-occurrences are not over-weighted.

Symmetric formulation

Connection with Skip-gram

Let Q_ij be the probability that word j appears in the context of word i.

The global objective function can be written as

weighted cross-entropy resembles the least square formulation

Daniel Jurafsky and James H. Martin. 2024. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, 3rd edition. Online manuscript released August 20, 2024. [Chapter 6]

​

​

​

​