Large Language Models

Lecture 3

Word Representation: Word2vec

Krishnendu Ghosh

‘Meaning’ of a Word

To perform language modelling effectively, it is essential for the model to somehow capture the meaning of each word.

​

Definition: meaning (Webster dictionary)

• The idea that is represented by a word, phrase, etc.

• The idea that a person wants to express by using words, signs, etc.

• The idea that is expressed in a work of writing, art, etc.

Need for Word Representation

For language modeling:

• We need effective representation of words

• The representation must somehow encapsulate the word meaning

Words as Discrete Symbols

In traditional NLP, we regard words as discrete symbols:

​

hotel, conference, motel – a localist representation

​

Such symbols for words can be represented by one-hot vectors:

​

motel = [0 0 0 0 0 0 0 0 0 0 1 0 0 0 0]

hotel = [0 0 0 0 0 0 0 1 0 0 0 0 0 0 0]

​

Vector dimension = number of words in vocabulary (e.g., 500,000+)

Words as Discrete Symbols

Example: in web search, if a user searches for “Delhi motel”, we would also like to match documents containing “Delhi hotel”

But:

​

motel = [0 0 0 0 0 0 0 0 0 0 1 0 0 0 0]

hotel = [0 0 0 0 0 0 0 1 0 0 0 0 0 0 0]

​

• These two vectors are orthogonal

• There is no natural notion of similarity for one-hot vectors!

• Solution:

• Could try to rely on WordNet’s list of synonyms to get similarity?

Thesauri/Ontologies

WordNet 3.0

• A hierarchically organized lexical database

• Online thesaurus + aspects of a dictionary

• Some other languages available or under development

• (Arabic, Finnish, German, Portuguese...)

Thesauri/Ontologies

How is “sense” defined in WordNet?

• Using the synset (synonym set), the set of near-synonyms, instantiates a sense or concept, with a gloss.

• Example:

• chump as a noun with the gloss:

“a person who is gullible and easy to take advantage of”

• This sense of “chump” is shared by 9 words:

chump1, fool2, gull1, mark9, patsy1, fall guy1, sucker1, soft touch1, mug2

• Each of these senses have this same gloss

• (Not every sense; sense 2 of gull is the aquatic bird)

Thesauri/Ontologies

Example:

​

Senses of ‘bass’:

Thesauri/Ontologies

• A useful resource but missing nuance

• e.g., “proficient” is listed as a synonym for “good”: this is only correct in some contexts

• Also, WordNet lists offensive synonyms in some synonym sets without any coverage of the connotations or appropriateness of words

• Missing new meanings of words

• e.g., wicked, badass, nifty, wizard, genius, ninja, bombest

• Impossible to keep up-to-date!

• Subjective

• Requires human labor to create and adapt

Words by Their Context

Distributional semantics: A word’s meaning is given by the words that frequently appear close-by.

“You shall know a word by the company it keeps” (J. R. Firth 1957: 11)

​

• When a word w appears in a text, its context is the set of words that appear nearby (within a fixed-size window).

• We can have many contexts of w to build up a representation of w

• ...government debt problems turning into banking crises as happened in 2009...

• ...saying that Europe needs unified banking regulation to replace the hodgepodge...

• ...India has just given its banking system a shot in the arm...

• These context words will represent banking

Count-based Methods

Co-occurrences for Word Similarity

The Term-Context matrix (or, word-word matrix)

• Each cell: number of times the row (target) word and the column (context) word co-occur in some context in the corpus

• Generally, smaller contexts are used, like:

• Paragraph

• Window of 10 words

• Each word is a count vector in ℕ^v: a row below (V: size of vocabulary)

20 words (Brown corpus)

• equal amount of sugar, a sliced lemon, a tablespoonful of apricot preserve or jam, a pinch each of clove and nutmeg,
• on board for their enjoyment. Cautiously she sampled her first pineapple and another fruit whose taste she likened to that of
• of a recursive type well suited to programming on the digital computer. In finding the optimal R-stage policy from that of
• substantially affect commerce, for the purpose of gathering data and information necessary for the study authorized in the first section of this

Co-occurrences for Word Similarity

The Term-Context matrix (or, word-word matrix)

​

• Two words are similar in meaning if their context vectors are similar

Should We Use Raw Counts?

• Raw word frequency is not a great measure of association between words

• It’s very skewed

• “the” and “of” are very frequent, but maybe not the most discriminative

​

• We’d rather have a measure that asks whether a context word is particularly informative about the target word.

​

• For the term-document matrix:

• We generally use tf-idf instead of raw term counts.

Term Frequency (tf)

​

​

​

​

​

Instead of using raw count, we squash a bit:

​

​

Document Frequency (df)

df_t is the number of documents t occurs in.

(note this is NOT collection frequency: total count across all documents)

​

Example: "Romeo" is very distinctive for one Shakespeare play:

Inverse Document Frequency (idf)

What is a Document?

• Could be a play or a Wikipedia article

​

• But for the purposes of tf-idf, documents can be anything; we often call each paragraph a document!

Final tf-idf

Co-occurrence Matrix Approach

Drawbacks of Co-occurrence Matrix Approach

​

• Quadratic space needed

​

• Relative position and order of words not considered

Low Dimensional Vectors

Store only “important” information in fixed, low dimensional vector.

• Singular Value Decomposition (SVD) on co-occurrence matrix

• is the best rank k approximation to X , in terms of least squares

• Motel = [0.286, 0.792, -0.177, -0.107, 0.109, -0.542, 0.349, 0.271]

Low Dimensional Vectors

Drawbacks of SVD-based Approach

​

• Computational cost scales quadratically for n x m matrix: O(mn²) flops (when n<m)

​

• Hard to incorporate new words or documents

​

• Does not consider order of words

Prediction-based Methods

Word Embedding

• Dense vector

• Helps in learning less parameters

• May generalize better

• Can capture synonyms better

• car and automobile are synonyms; but have distinct dimensions

• A word with car as a neighbor and a word with automobile as a neighbor should be similar, but are not

Represent Word Meaning: Word2vec

Two basic neural network models:

• Continuous Bag of Word (CBOW): use a window of word to predict the middle word

• Skip-gram (SG): use a word to predict the surrounding words in window

Word2vec

• Instead of counting how often each word w occurs near “apricot”

• Train a classifier on a binary prediction task:

• Is w likely to show up near "apricot"?

• We don’t actually care about this task

• But we'll take the learned classifier weights as the word embeddings

• Big idea: Self-supervision

• A word c that occurs near apricot in the corpus cats as the gold "correct answer" for supervised

learning

• No need for human labels

• Bengio et al. (2003); Collobert et al. (2011)

Approach:

Predict if candidate ‘c’ is a "neighbor"

1. Treat the target word t and a neighboring context word c as positive examples.

​

2. Randomly sample other words in the lexicon to get negative examples

​

3. Use logistic regression to train a classifier to distinguish those two cases

​

4. Use the learned weights as the embeddings

Skip-Gram Training Data

Assume a +/- 2 word window, given training sentence:

Skip-Gram Classifier

(assuming a +/- 2 word window)

​

​

​

​

Goal: Train a classifier, that, given a candidate (word, context) pair

(apricot, jam)

(apricot, aardvark)

...

​

assigns each pair a probability:

P(+ | w, c)

P(− | w, c) = 1 − P(+ | w, c)

Similarity using Dot Product

• Remember: Two vectors are similar if they have a high dot product

​

• Cosine is just a normalized dot product

​

• Similarity(w , c) ∝ w∙ c

• We’ll need to normalize to get a probability

• Cosine isn't a probability either

Dot Products into Probabilities

Sim(w , c) ≈ w ∙ c

• To turn this into a probability

• We'll use the sigmoid function, as in logistic regression:

Skip-gram computes P(+|w, c)

​

​

• This is for one context word, but we have lots of context words.

• We'll assume independence and just multiply them:

Skip-gram Classifier: Summary

• A probabilistic classifier, given

• a test target word w

• its context window of L words c_1:L

​

• Estimates probability that w occurs in this window based on similarity of w (embeddings) to c_1:L (embeddings).

​

• To compute this, we just need embeddings for all the words.

The Embeddings We’ll Need:

A Set for w, A Set for c

Learning the Embeddings

For each positive example we'll grab k negative examples, sampling by frequency

Learning the Embeddings

Choosing Negative Examples

Word2vec: Learning Word Vectors

• Given the set of positive and negative training instances, and an initial set of embedding vectors

• The goal of learning is to adjust those word vectors such that we:

• Maximize the similarity of the target word, context word pairs (w , cpos) drawn from the positive data

• Minimize the similarity of the (w , cneg) pairs drawn from the negative data.

Loss Function

Maximize the similarity of the target with the actual context words, and minimize the similarity of the target with the k negative sampled non-neighbor words.

Learning the Classifier

• How to learn?

• Stochastic gradient descent!

​

• We’ll adjust the word weights to

• make the positive pairs more likely

• and the negative pairs less likely, over the entire training set.

Reminder: Gradient Descent

Derivatives of The Loss Function

Update Equation in SGD

Start with randomly initialized C and W matrices, then incrementally do updates

Two Sets of Embeddings

Skip-gram learns two sets of embeddings:

​

1. Target embeddings matrix W

2. Context embedding matrix C

​

It's common to just add them together, representing i-th word as the vector W[i] + C[i]

Summary: Learning Word2vec

• Start with V random d-dimensional vectors as initial embeddings

• Train a classifier based on embedding similarity

• Take a corpus and take pairs of words that co-occur as positive examples

• Take pairs of words that don't co-occur as negative examples

• Train the classifier to distinguish these by slowly adjusting all the embeddings to improve the classifier

performance

• Throw away the classifier code and keep the embeddings

Some Tricks

​

Sub-sampling Frequent Words

​

There are two “problems” with common words like “the”:

​

1. When looking at word pairs, (“fox”, “the”) doesn’t tell us

much about the meaning of “fox”. “the” appears in the

context of pretty much every word.

​

2. We will have many more samples of (“the”, ...) than we

need to learn a good vector for “the”.

• If we have a window size of 10, and we remove a specific instance of “the” from our text:

• As we train on the remaining words, “the” will not appear in any of their context windows.

• We’ll have 10 fewer training samples where “the” is the input word.

Some Interesting Results

Some Interesting Results

Word2vec: Problems

The cat sat on the mat

​

Word2vec can’t capture the information like:

​

• Is “The” a special context of the words “cat” and “mat”?

​

Or

​

• Is “The” just a stopword?

Word2vec: Problems

• Word2Vec can’t handle unknown words – words appearing in a test corpus but were unseen in the training corpus

fasttext embedding

Subword embedding

Each word is represented by itself plus a bag of constituent n-grams, with special boundary symbols ‘<’ and ‘>’ added to each word.

​

• For example, with n = 3 the word where would be represented by the sequence plus the character n-grams:

​

where, <wh, whe, her, ere, re>

​

• Skip-gram is learned for each constituent n-gram

• where is represented by the sum of all of the embeddings of its constituent n-grams.

• Unknown words can then be presented only by the sum of the constituent n-grams

Code

Source: https://lcs2-iitd.github.io/ELL881-AIL821-2401/lectures/