1 of 19

Distributional Semantics
Word2Vec

Word Embeddings

CS6501: Vision and Language

2 of 19

What we see

How to represent a word?

dog

cat

person

1

2

3

Problem: distance between words using one-hot encodings always the same

Idea: Instead of one-hot-encoding use a histogram of commonly co-occurring words.

CS6501: Vision and Language

3 of 19

What we see

Distributional Semantics

dog

I saw a dog on a leash walking in the park.

Dogs are man’s best friend.

He walks his dog in the late afternoon

…

friend

leash

park

walking

walks

food

legs

runs

sleeps

sits

…

His dog is his best companion.

CS6501: Vision and Language

4 of 19

What we see

Distributional Semantics

dog

cat

person

food

walks

window

runs

mouse

invented

legs

sleeps

mirror

tail

…

This vocabulary can be extremely large

CS6501: Vision and Language

5 of 19

What we see

Toward more Compact Representations

dog

cat

person

food

walks

window

runs

mouse

invented

legs

sleeps

mirror

tail

…

This vocabulary can be extremely large

CS6501: Vision and Language

6 of 19

What we see

Toward more Compact Representations

dog =

=

legs, running, �walking

w1

+ w2

+ w3

tail, fur,

ears

mirror, window,�door

+ …

CS6501: Vision and Language

7 of 19

What we see

Toward more Compact Representations

dog =

w1

w2

w3

The basis vectors can be found using Principal Component Analysis (PCA)

CS6501: Vision and Language

8 of 19

What is word2vec?

word2vec is not a single algorithm
It is a software package for representing words as vectors, containing:

Two distinct models

CBoW
Skip-Gram (SG)

Various training methods

Negative Sampling (NS)
Hierarchical Softmax

A rich preprocessing pipeline

Dynamic Context Windows
Subsampling
Deleting Rare Words

Slide by Omer Levy

9 of 19

Embeddings capture relational meaning!

Slide by Dan Jurafsky

10 of 19

Skip-Grams with Negative Sampling (SGNS)

Marco saw a furry little wampimuk hiding in the tree.

10

“word2vec Explained…”

Goldberg & Levy, arXiv 2014

11 of 19

Skip-Grams with Negative Sampling (SGNS)

Marco saw a furry little wampimuk hiding in the tree.

11

“word2vec Explained…”

Goldberg & Levy, arXiv 2014

12 of 19

Skip-Grams with Negative Sampling (SGNS)

Marco saw a furry little wampimuk hiding in the tree.

words contexts

wampimuk furry

wampimuk little

wampimuk hiding

wampimuk in

… …

12

“word2vec Explained…”

Goldberg & Levy, arXiv 2014

13 of 19

Skip-Grams with Negative Sampling (SGNS)

13

“word2vec Explained…”

Goldberg & Levy, arXiv 2014

14 of 19

Word2Vec Objective

14

15 of 19

Word2Vec Objective

15

16 of 19

Skip-Grams with Negative Sampling (SGNS)

16

“word2vec Explained…”

Goldberg & Levy, arXiv 2014

17 of 19

Skip-Grams with Negative Sampling (SGNS)

17

“word2vec Explained…”

Goldberg & Levy, arXiv 2014

18 of 19

Skip-Grams with Negative Sampling (SGNS)

18

19 of 19

19

Questions?

CS6501: Vision and Language