1 of 33

CSCI-SHU 376: Natural Language Processing

Hua Shen

2026-09-09

Fall 2026

Lecture 4: Word Embedding

​

Course Agenda: 2026 Fall-NLP-[CSCI-SHU-376]-Class Schedule

2 of 33

Today’s Plan

  • Vector Semantics
  • Word2vec
  • Glove, FastText
  • Evaluation

​

​

​

​

3 of 33

How do we represent words in NLP models?

  • N-gram models

​

​

​

  • Naïve Bayes

​

4 of 33

How do we represent words in NLP models?

  • Logistic Regression

​

​

​

​

5 of 33

Words

  • Synonyms: couch / sofa, car / automobile
  • Antonyms: dark / light, rise / fall, up/down
  • Some words are not synonyms but they share some element of meaning
    • Cat / dog, car / bicycle, cow / horse
  • Some words are not familiar but are related
    • Coffee / cup, house / door

6 of 33

Need for word meaning

  • Previously, each feature is a word identity (just string)
    • Requires exact same word to be in the training / test set e.g., terrible != horrible

​

​

  • If we can represent word meaning in vectors:
    • Similar words might have similar vectors
    • We can generalize to similar but unseen words

7 of 33

Distributional hypothesis

  • Words that occur in similar contexts tend to have similar meanings

​

​

​

​

  • When a word appears in a text, its context is the set of words appear nearby

8 of 33

Distributional hypothesis: Ongchoi

  • Ongchoi is delicious sauteed with garlic
  • Ongchoi is superb over rice
  • Ongchoi leaves with salty sauces

​

Q: what do you think "Ongchoi” means?

A. a savory snack B. a green vegetable

C. an alcoholic beverage D. a cooking sauce

​

​

​

​

9 of 33

Distributional hypothesis: Ongchoi

  • Ongchoi is delicious sauteed with garlic
  • Ongchoi is superb over rice
  • Ongchoi leaves with salty sauces

​

Q: what do you think "Ongchoi” means?

A. a savory snack B. a green vegetable

C. an alcoholic beverage D. a cooking sauce

​

  • You mean have seen: Spanich sauteed with garlic over rice

Chard stems and leaves are delicious

Collard greens and other salty leafty greens

​

​

10 of 33

Distributional hypothesis: Ongchoi

What should we do? See what other words occur in those contexts

11 of 33

Words and Vectors

  • Easy solution: use word-word co-occurrence to represent the meanings

  • Each word is represented by corresponding row vector

​

​

​

12 of 33

Measuring Similarity

13 of 33

Sparse vs Dense Vectors

  • Sparse vector (e.g., one-hot vector)
    • Long: vocab size length
    • Sparse: most entries are zeros

​

  • Dense vector
    • Short (usually 50-300 dimensions), and dense (real-valued vectors)
    • Basis for modern NLP systems

​

14 of 33

Why Dense Vectors?

 

15 of 33

Word Embeddings

 

16 of 33

Word Embeddings

  • Learned dense vectors from text (i.e. corpora) for representing words
  • Property: similar words have similar vectors (Important!)

17 of 33

Word Embeddings

  • Property: similar words have similar vectors (Important!)

18 of 33

Word Embeddings

  • Property: similar words have similar vectors (Important!)

19 of 33

Trained Word Embeddings

​

​

  • FastText: https://fasttext.cc

​

20 of 33

Today’s Plan

  • Vector Semantics
  • Word2vec
  • Glove, FastText
  • Evaluation

​

​

​

​

21 of 33

Word2Vec

  • (Mikolov at el 2013a): Efficient Estimation of word representations in vector space
  • (Mikolov at el 2013b): Distributed Representations of Words and Phrases and their Compositionality

​

​

22 of 33

Skip-gram

  • Assume we have a large corpus
  • Key idea: use each word to predict other words in its context
  • Context: a fixed window of size 2m (m=2 in the example)

​

​

​

23 of 33

Skip-gram

  • Sliding window: convert the corpus into tuples

(into, problems), (into, turning), (into, banking), (into, crises), (banking, turning), (banking into), (banking, crises), (banking, as) …

​

24 of 33

Skip-gram: Objective function

 

25 of 33

 

Dot-product of word u and context v

 

26 of 33

Negative Sampling

​

​

 

27 of 33

Today’s Plan

  • Vector Semantics
  • Word2vec
  • Glove, FastText
  • Evaluation

​

​

​

​

28 of 33

Continuous Bag of Words (CBOW)

​

​

29 of 33

GloVe: Global Vectors

​

​

 

30 of 33

FastText: Subword Embeddings

​

​

 

31 of 33

Today’s Plan

  • Vector Semantics
  • Word2vec
  • Glove, FastText
  • Evaluation

​

​

​

​

32 of 33

Intrinsic Evaluation: word similarity

  • Wordsim-353; 353 pairs of words with human judgement

​

​

33 of 33

Extrinsic Evaluation

  • Plug word embeddings into a downstream NLP task and see whether it improves performance (e.g., sentiment classification)
  • The most important metric

​

​

​