1 of 13

Large Language Models

Lecture 4

Word Representation: GloVe

Krishnendu Ghosh

2 of 13

Count-based vs Prediction-based

3 of 13

GloVe – Global Vectors

Crucial insight: Ratios of co-occurrence probabilities can encode word meaning

4 of 13

GloVe – Global Vectors

Crucial insight: Ratios of co-occurrence probabilities can encode word meaning

5 of 13

Co-occurrence Matrix

6 of 13

Learn Word Vectors

7 of 13

Learn Word Vectors

8 of 13

Learn Word Vectors

9 of 13

Learn Word Vectors

10 of 13

Weighting function

11 of 13

GloVe: Advantages

• Fast training

​

• Scalable to huge corpora

​

• Good performance even with small corpus and small vectors

12 of 13

Details About GloVe

Original paper: https://nlp.stanford.edu/pubs/glove.pdf

​

Blogs with easy explanations:

​

https://medium.com/sciforce/word-vectors-in-natural-language-processing-global-vectors-glove-51339db89639

​

https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/?fbclid=IwAR3-pws3-K-Snfk6aqbixdxS8zFf-uuPDJ_0ipb94kWeygrdCSEqE9HWmNs

​

https://towardsdatascience.com/light-on-math-ml-intuitive-guide-to-understanding-glove-embeddings-b13b4f19c010

13 of 13

Source: https://lcs2-iitd.github.io/ELL881-AIL821-2401/lectures/