Text representation as �Vector Semantics

Introduction

• Einat Minkov, senior lecturer at IS dep.
• 2003-08: PhD at Carnegie Mellon University, Language Technologies Institution (LTI).
• Joined Univ. of Haifa in 2010
• Univ. of Edinburgh 2018-19
• Google Research 2020, Nokia Research 2010, �MS Research 2006
• Also teach: Intro. to AI / Text mining (advanced)
• Research interests:
• Natural Language Processing (NLP)
• Information extraction (IE) / semantics
• Relational inference
• and more.

​

​

Information Retrieval (IR)

Goal: rank documents in a corpus (Web) by the match with the query.

How to quantify this match?

​

The vector space method:

Translate query/document into vectors in a vector space

one vector for each document

How to convert text into a vector?

​

Measure distance between vectors (documents)

How to measure vector distance?

​

A bag of words representation

I love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun… It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet.

A bag of words representation

The Vector-Space Model (IR)

• Assume t distinct terms remain after preprocessing; call them the vocabulary.
• These terms form a vector space.

Dimension = t = |vocabulary|

• Each term i in a document or query j �is given a real-valued weight, w_ij.
• Both documents and queries are expressed as �t-dimensional vectors:

d_j = (w_1j, w_2j, …, w_tj)

​

7

Graphic Representation

Example:

D₁ = 2T₁ + 3T₂ + 5T₃

D₂ = 3T₁ + 7T₂ + T₃

Q = 0T₁ + 0T₂ + 2T₃

8

• Is D₁ or D₂ more similar to Q?
• How to measure the degree of similarity? Distance? Angle? Projection?

Cosine Similarity Measure

• Cosine similarity measures the cosine of the angle between two vectors.
• Inner product normalized by the vector lengths.

9

D₁ = 2T₁ + 3T₂ + 5T₃ CosSim(D₁ , Q) = 10 / √(4+9+25)(0+0+4) = 0.81

D₂ = 3T₁ + 7T₂ + 1T₃ CosSim(D₂ , Q) = 2 / √(9+49+1)(0+0+4) = 0.13

Q = 0T₁ + 0T₂ + 2T₃

D₁ is 6 times better than D₂ using cosine similarity but only 5 times better using inner product.

CosSim(d_j, q) =

Example: Cosine similarity amongst 3 documents

• How similar are the novels �SaS: Sense and Sensibility, �PaP: Pride and Prejudice, and WH: Wuthering Heights?��(the first two – authored by Jane Austin, and the latter – by Emily Bronte)

Term frequencies (counts)

Example: Cosine similarity amongst 3 documents

• Log frequency weighting

cos(SaS,PaP) ≈

0.789 × 0.832 + 0.515 × 0.555 + 0.335 × 0.0 + 0.0 × 0.0

≈ 0.94

cos(SaS,WH) ≈ 0.79

cos(PaP,WH) ≈ 0.69

Question

• How would you use this model for text classification, or clustering?

The Problem

• synonymy: many ways to refer to the same object, e.g. car �and automobile -- leads to poor recall
• polysemy: most words have more than one distinct meaning, e.g., model, python, chip -- leads to poor precision

​

auto

engine

bonnet

tyres

lorry

boot

car

emissions

hood

make

model

trunk

make

hidden

Markov

model

emissions

normalize

Synonymy

Will have small cosine

but are related

Polysemy

Will have large cosine

but not truly related

What do words mean?

• First thought: look in a dictionary

​

• http://www.oed.com/

​

​

​

Words, Lemmas, Senses, Definitions

sense

definition

Lemma pepper

Sense 1: spice from pepper plant

Sense 2: the pepper plant itself

Sense 3: another similar plant (Jamaican pepper)

Sense 4: another plant with peppercorns (California pepper)

Sense 5: capsicum (i.e. chili, paprika, bell pepper, etc)

Name these items

Superordinate Basic Subordinate

​

chair office chair

piano chair rocking chair

furniture lamp torchiere

desk lamp

table end table coffee table

​

Distributional similarity �(Ludwig Wittgenstein):

"The meaning of a word is its use in the language"

A data-driven approach!

What does ongchoi mean?

Suppose you see these sentences:

• Ong choi is delicious sautéed with garlic.
• Ong choi is superb over rice
• Ong choi leaves with salty sauces
• And you've also seen these:
• …spinach sautéed with garlic over rice
• Chard stems and leaves are delicious
• Collard greens and other salty leafy greens
• Conclusion:
• Ongchoi is a leafy green like spinach, chard, or collard greens

​

Ong choi: Ipomoea aquatica �"Water Spinach"

Yamaguchi, Wikimedia Commons, public domain

Let's define words by their usages

• In particular, words are defined by their environments (the words around them)

​

• If A and B have almost identical environments, they are probably synonyms.

​

​

word-word matrix�(or "term-context matrix")

• Two words are similar in meaning if their context vectors are similar

​

23

24

Which pair of words is more similar?

cosine(apricot,information) =

​

cosine(digital,information) =

​

cosine(apricot,digital) =

​

Visualizing cosines �(angles)

Ask humans how similar two words are

SimLex-999 dataset (Hill et al., 2015)

Vector Semantics, part II:

�Dimensionality reduction & �Word embeddings

• Freq-based word vectors are
• long (length |V|= 20,000 to 50,000)
• sparse (most elements are zero)

Alternative: dense vectors

• vectors which are
• short (length 50-1000)
• dense (most elements are non-zero)

29

Singular Value Decomposition (SVD)

• �Example: apply SVD to extract dimensions of meaning (topics) from documents.
• consider documents d1-d4

Doc1 and doc2 similar 🡪 `computers’�Doc3 and doc4 similar 🡪 `automotive’

(the term ‘the’ is not really related to any topic)

• SVD will help up extract these dimensions, which we refer to as computers and automotive. it won‘t, however, give us nice human readable names for them.

SVD Example

SVD decomposes a matrix A into the product of three specially formed matrices (mysteriously named) U, S and V

S – describes the relative strength of features

U – relationship between terms and features

Vt – relationship between features and documents

Dimensionality Reduction

• Singular Value Decomposition

{A}={U}{S}{V}^T

• Dimension Reduction

{~A}~={~U}{~S}{~V}^T

(Pre-trained) dense word embeddings

• Word2vec (Mikolov et al.)
• https://code.google.com/archive/p/word2vec/

​

• Glove (Pennington, Socher, Manning)
• http://nlp.stanford.edu/projects/glove/

​

Word2Vec and Word embeddings

Word2Vec: Model I

Can you compute Softmax?

Can you compute Softmax?

Word2Vec: Model II

“ … who passes the … “

Skip-Gram Training Data

• Training sentence:
• ... lemon, a tablespoon of apricot jam a pinch ...
• c1 c2 target c3 c4

​

43

9/13/23

Asssume context words are those in +/- 2 word window

Basic idea behind skip-gram embeddings

from an input word w(t) in a document

construct hidden layer that “encodes” that word

So that the hidden layer will predict likely nearby words w(t-K), …, w(t+K)

final step of this prediction is a softmax over lots of outputs

Basic idea behind skip-gram embeddings

Training data:

positive examples are pairs of words w(t), w(t+j) that co-occur

Training data:

negative examples are samples of pairs of words w(t), w(t+j) that don’t co-occur

You want to train over a very large corpus (100M words+) and hundreds+ dimensions

Skip-Gram Training

• Training sentence:
• ... lemon, a tablespoon of apricot jam a pinch ...
• c1 c2 t c3 c4

46

9/13/23

• For each positive example, we'll create k negative examples.
• Using noise words
• Any random word that isn't t

​

Skip-Gram Training

• Training sentence:
• ... lemon, a tablespoon of apricot jam a pinch ...
• c1 c2 t c3 c4

47

9/13/23

k=2

Negative sampling

increase the probability for less frequent words and decrease the probability for more frequent words.

Learning the classifier

• Iterative process.
• We’ll start with 0 or random weights
• Then adjust the word weights to
• make the positive pairs more likely
• and the negative pairs less likely

​

Evaluating embeddings

• Compare to human scores on word similarity-type tasks:
• WordSim-353 (Finkelstein et al., 2002)
• SimLex-999 (Hill et al., 2015)
• Stanford Contextual Word Similarity (SCWS) dataset (Huang et al., 2012)
• TOEFL dataset: Levied is closest in meaning to: imposed, believed, requested, correlated

​

Ask humans how similar two words are

SimLex-999 dataset (Hill et al., 2015)

Words as vectors of context words

• Each word = a vector
• Similar words are "nearby in space"

​

​

Results from word2vec

https://www.tensorflow.org/versions/r0.7/tutorials/word2vec/index.html

Analogy: Embeddings capture relational meaning!

56

Word2Vec vs. �Distributional Semantic Models

• “ it generally makes no difference whatsoever whether you use word embeddings or distributional methods -- what matters is that you tune your hyperparameters and employ the appropriate pre-processing and post-processing steps.
• Recent papers from Jurafsky's group ^[5] [6] echo these findings and show that SVD -- not SGNS -- is often the preferred choice when you care about accurate word representations.”

​

Question

• How would you use word vectors (Counts or word embeddings) for document classification?

Conclusion