Document Embeddings

A concise literature review

by Shay Palachy

www.shaypalachy.com

For a complementary literature review see https://towardsdatascience.com/document-embedding-techniques-fed3e7a6a25d

Agenda

Agenda

• Motivation
• Approaches
• Classic Techniques
• Unsupervised document embedding techniques
• Supervised document embedding techniques
• Trends and challenges
• How to choose which technique to use

Motivation

Motivation

Recall: Word embeddings are mappings of words into real-numbered vectors

• The building blocks for text representation for ML models (handle only numeric data)
• Lead to great improvement in almost every NLP task
• Modern methods yield rich vector space representations...

Modern word embedding spaces

• Axes/dimensions in the embedding space capture deep semantic or syntactic concepts and phenomena.
• Distances in the embedding space reflect real semantic and syntactic differences between words.
• Millions of words are represented using only dozens or hundreds of dimensions.

Motivation

• So far deriving a representation for a sequence of words from word embedding meant concating/averaging them
• Is this the best we can do?
• Can supervised and self-supervised techniques used for word embeddings be extended to learn more meaningful embeddings for larger units of texts?

Applications

• doc2vec: Text classification and sentiment analysis tasks [Le & Mikolov, 2014] and document similarity tasks [Dai et al, 2015].
• Skip-thought: Semantic relatedness, paraphrase detection, image-sentence ranking, classification, sentiment [Kiros et al, 2015] POS tags and dependency relations [Broere, 2017].
• Deep Semantic Similarity Model: Information retrieval and web search ranking, ad selection/relevance, contextual entity search and interestingness tasks, question answering, knowledge inference, image captioning, and machine translation tasks.

Approaches

Approaches

• Summarizing word vectors
• Topic modelling
• Encoder-decoder models
• Supervised representation learning
• Learning document embeddings from labeled data
• Task-specific supervised document embeddings
• Jointly learning sentence representations

(self-supervised learning)

unsupervised learning

Classic techniques

A recap

Bag-of-words

• Choose vocabulary size
• Take K most important words (how? frequency?)
• 1 where a vocabulary words appears, 0 otherwise

Bag-of-ngrams

• Gain back some of the word order information
• Encode short phrases as “words”

tf-idf weighting

• TF - Count the number of appearance in a document
• The TF term grows as the word appears more often, while the IDF term increases with the word’s rarity
• Adjusts frequency scores for the fact that some words appear more (or less) frequently

LDA (topic models)

• Bag-of-words = A simple probabilistic model of documents as distributions over words
• Add a latent (hidden) intermediate layer of K topics
• Topics are now characterized by distributions over words, while documents are distributions over topics

The probabilistic model shift from bag-of-words to LDA

LDA

To generate a set of M documents of lengths {Nᵢ}, assuming a predetermined number of K topics:

(Dirichlet dist.)

LDA

• While inferring the model, a vector space of dimension K was inferred
• It captures in some way the topics or themes in our corpus and the way they are shared between documents in it
• Actually an embedding space for these documents�
• Depending on the choice of K it can be of a significantly smaller dimension than the vocabulary

every document is now a mixture of topics!

Unsupervised document embedding techniques

Unsupervised document embedding techniques

• n-gram embeddings
• Averaging word embeddings
• Sent2Vec
• Paragraph vectors (doc2vec)
• Doc2VecC
• Skip-thought vectors
• FastSent
• Quick-thought vectors
• Word Mover’s Embedding (WME)

n-gram embeddings

• Identify many short phrases using a data-driven approach
• Treat them as individual tokens during the training of the word2vec model
• Less suitable for learning longer phrases
• Does not generalizes well to unseen phrases

[Mikolov et al, 2013b]

Averaging word embeddings

• Use a fixed (unlearnable) vector summarization operator (e.g. avg, sum, concat)
• Learn word embeddings in a preceding layer, using a learning target tailored for document embedding
• e.g. using a sentence to predict context sentences
• Main advantage: Word embeddings are optimized for averaging into document representations!

Averaging word embeddings

[Kenter et al, 2016] [Hill et al, 2016] [Sinoara et al, 2019]

Averaging word embeddings

• [Arora et al, 2016] showed it to be a very good baseline by adding two small variations:
• Use a smooth inverse frequency weighting scheme
• Remove the common discourse component (find w/ PCA) from word vectors, presumably related to syntax

Python implementation

Sent2Vec

• Combine the two above methods; word2vec’s CBOW w/:
• Word n-grams
• Optimize the word/n-grams embeddings for averaging into document vectors
• No frequent word subsampling (to get n-grams features)
• No dynamic context windows; context = whole sentence

C++-based Python implementation

[Pagliardini et al, 2017] [Gupta et al, 2019]

sent2vec = unsupervised fastText, w/:�1) context = entire sentence�2) class labels = vocabulary words

Paragraph Vectors (doc2vec)

• Add a memory vector, capturing paragraph topic/context
• Map paragraphs to unique vectors
• At prediction time, compute vectors for new paragraphs

Gensim, PyTorch

[Le & Mikolov, 2014]

Paragraph Vectors: Distributed Memory (PV-DM)

• Predict word from m previous words + paragraph�
• Concat paragraph and word vectors

(preserves order information)

paragraph matrix D

avg/concat

classifier

word embeddings

(like word2vec’s CBOW)

Paragraph Vectors: Distributed Bag of Words (PV-DBOW)

• Predict context words from paragraph

(like word2vec’s skip-gram)

Document vector through corruption (doc2vecC)

C implementation

[Chen, 2017]

represent documents as embedding avg of randomly sampled words

Corrupt = randomly drop many words

• Training speed-up
• Regularization

Skip-thought vectors

• Predict previous/next sentence from current one
• Learn vector representations w/ RNN encoder-decoder
• Learn word embedding for small vocabulary (20,000)
• Learn a mapping from word2vec to this vector space

Python, PyTorch, TensorFlow

[Kiros et al, 2015]

shared word embedding layer

different encoders for prev/next

Skip-thought vectors

Three improvements to skip-thought: [Tang et al, 2017]

• Only learn to decode the next sentence
• Add avg+max connections between encoder and decoder
• Allows non-linear non-parametric feature engineering
• Good word embedding initialization

([Gan et al, 2016] use a hierarchical CNN-LSTM-based encoder instead)

Quick-thought vectors

• Reformulate sentence context prediction as a supervised classification problem

Python implementation

[Logeswaran & Lee, 2018]

quick-thought

skip-thought

Quick-thought vectors

• Reformulate sentence context prediction as a supervised classification problem
• Learn two (RNN) encoders: f for input, g for candidates
• Candidate set built from (1) valid context sentences +�(2) other non-context sentences
• Training objective maximizes the probability of identifying the correct context sentences

Python implementation

[Logeswaran & Lee, 2018]

Word Mover’s Embedding (WME)

Given a rich unsupervised word embedding space

• Build doc distance metric w/ Word Mover’s Distance (WMD)
• Derive from it a positive-definite kernel w/ D2KE
• Derive doc embedding w/ random features approx of kernel

Python implementation

[Wu et al, 2018b]

Supervised document embedding techniques

Learning document embeddings from labeled data

• Explicitly learn phrase embeddings w/ a parallel corpus of phrases for statistical machine translation [Cho et al, 2014a]
• Minimize cosine similarity between pairs of paraphrases [Wieting et al, 2015]
• Train neural language models to map dictionary definitions to pre-trained word embeddings of the words defined by those definitions [Hill et al, 2015]

word embeddings

stacks of 3 bidirectional GRUs

both branches share parameter weights

Task-specific supervised document embeddings

• Train a neural network on some supervised NLP tasks
• Input word embeddings
• All hidden layers of the network can be considered to produce a vector embedding of an input document
• Produces task-specific document embeddings
• Less robust than unsupervised ones

Jointly learning sentence representations

• Learn sentence representations from multiple text classification tasks
• Combine them with pre-trained word-level and sentence-level encoders
• Get robust sentence representations that are useful for transfer learning

Trends and challenges

Trends and challenges

• Encoder-Decoder Optimization
• Architecture: NN/CNN/RNN
• Hyperparams: n-grams, projection functions, weighing
• Goal: Improve success metrics
• Goal: Train models over larger corpora / in shorter time

Trends and challenges

• Learning objective design
• Quick-thought
• Word Mover’s Distance
• Innovations might be applicable to the problem of word embedding

Trends and challenges

• Benchmarking
• Open-sourcing
• Reproducibility & application to real word problems
• Cross-task applicability
• More of a problem for supervised method
• Lack of large labeled corpora
• Only a problem for supervised methods

(no winners so far)

How to choose which technique to use

How to choose which technique to use

• Averaging word vectors is a strong baseline
• Focus on good word embeddings
• Compare different techniques
• Try some tricks (like in [Arora et al, 2016])
• Performance can be a key consideration
• Fast: Word vector avg, sent2vec and FastSent
• Slower: doc2vec

How to choose which technique to use

• Consider the validity of the learning objective to your task
• skip/quick-thought model strong relations between sentences based on their distance in a document
• Many open-source implementations means you can compare different solutions
• There are no clear task-specific leaders. :(

That’s it!

For a more thorough overview please read:

https://towardsdatascience.com/document-embedding-techniques-fed3e7a6a25d