2 of 22

Phrase embeddings
Subword-based Representation: FastText
Sentence and Document Representation: Doc2Vec
Hierarchical, network and knowledge graph representation

3 of 22

GloVe vectors

https://github.com/stanfordnlp/GloVe

4 of 22

How to Learn Phrase Embeddings?

5 of 22

Other kinds of static embeddings

fastText [Bojanowski et al, 2017]

Limitation of word2vec: a distinct vector representation for each word, but we learned about subwords and their benefits
Cannot handle out-of-vocabulary (OOV) tokens
An extension which takes into account subword information
https://fasttext.cc/ is also a library for learning of word embeddings and text classification created by Facebook's AI Research lab ⇒ Word representations · fastText

6 of 22

So, how does fastText work?

For a word at position t and a chosen context position c, using the binary logistic loss, we obtain the following negative log-likelihood:

Negative samples

However, this formulation ignores the internal structure of the word

fastText denotes each word as a bag of character n-grams

Image source: https://amitness.com/posts/fasttext-embeddings

7 of 22

So, how does fastText work?

For the word, all character n-grams between 3 to 6 characters are used

Image source: https://amitness.com/posts/fasttext-embeddings

We also include the word w itself in the set of its n-grams: <eating>

8 of 22

So, how does fastText work?

https://aclanthology.org/Q17-1010.pdf

How to calculate s(w,c)?

9 of 22

So, how does fastText work?

Image source: https://amitness.com/posts/fasttext-embeddings

But the unique n-grams can be huge!

Apply hashing to bound the memory requirements.

Instead of learning an embedding for each unique n-gram, we learn total B embeddings where B denotes the bucket size. The paper used a bucket of a size of 2 million.

10 of 22

How does fastText handle OOV words?

Image source: https://aclanthology.org/Q17-1010.pdf

Illustration of the similarity between character n-grams in out-of-vocabulary words. The OOV word is shown on the x axis. Red indicates positive cosine, while blue negative.

Simply average the vector representation of its n-grams

11 of 22

Beyond words: Sentence and documents

After word2Vec, the default way for representing larger units such as sentences was to use mean embeddings
Some works also attempted use of weighted average (tf-idf weights)
But can we also learn the representations of larger units directly?

Le and Mikolov, “Distributed Representations of Sentences and Documents”, 2014

word2Vec

doc2Vec

12 of 22

Hierarchical Representations

https://github.com/facebookresearch/poincare-embeddings

13 of 22

Node embeddings