Convolutional Neural Networks for NLP

Human Language Technologies

Giuseppe Attardi

Some slides from Christopher Manning

Dealing with Sequences: Idea

• Main CNN idea:
• What if we compute vectors for every possible word subsequence of a certain length?
• Example: “tentative deal reached to keep government open” computes vectors for:

tentative deal reached, deal reached to, reached to keep, to keep government, keep government open

• Regardless of whether phrase is grammatical
• Not very linguistically or cognitively plausible
• Then group them afterwards

Slide from Chris Manning

CNN

• Convolution is classically used to extract features from images
• Models position-invariant identification

​

​

• 2d example

Yellow color and red numbers show filter (=kernel) weights

Green shows input

Pink shows output �

​

Slide from Chris Manning

Convolutiona Neural Network

Convolutional Neural Network

• A convolutional layer in a NN is composed by a set of filters.
• A filter combines a "local" selection of input values into an output value.
• All filters are "sweeped" across all input.
• During training each filter specializes into recognizing some kind of relevant combination of features.
• CNNs work well on stationary features, i.e., those independent from position.

A 1D convolution for text

Apply a filter (or kernel) of size 3

+ bias�➔ non-linearity (RELU)

First filter, w1,w2,w3 = -1.0

➔

+

Filters

Filters have additional parameters that define:

• behavior at the start/end of documents (padding)
• size of the sweep step (stride)
• possible presence of holes in the filter window (dilation).

A filter using a filter size of 5 is applied to all the sequences of 5 words in a text.

3 filters using a size of 5 applied to a text of 10 words produce 18 output values. Why?

1D convolution for text with padding

Apply a filter (or kernel) of size 3

3D channel convolution with padding

Apply 3 filters (or kernel) of size 3

conv1d, padded, with max pooling over time

Apply 3 filters (or kernel) of size 3

conv1d, padded, average pooling over time

Apply 3 filters (or kernel) of size 3

conv1d, padded, average pooling, stride = 2

Apply 3 filters (or kernel) of size 3

Keras

from tensorflow.random import normal

batch_size = 16

word_embed_size = 4

seq_len = 7

input = normal((batch_size, seq_len, word_embed_size))

​

kernel_size = 3

conv1 = Conv1D(3, kernel_size) # can add: padding=1

hidden1 = conv1(input)

hidden2 = np.max(hidden1, dim=2) # max pool

PyTorch

batch_size = 16

word_embed_size = 4

seq_len = 7

input = torch.randn(batch_size, word_embed_size, seq_len)

​

conv1 = Conv1d(in_channels=word_embed_size, out_channels=3,

kernel_size=3) # can add: padding=1

hidden1 = conv1(input)

hidden2 = torch.max(hidden1, dim=2) # max pool

Single Layer CNN for Sentence Classification

• Yoon Kim (2014): Convolutional Neural Networks for Sentence Classification. EMNLP 2014. https://arxiv.org/pdf/1408.5882.pdf
• A variant of convolutional NNs of Collobert, Weston et al. (2011) Natural Language Processing (almost) from Scratch.
• Goal: Sentence classification:
• Mainly positive or negative sentiment of a sentence
• Other tasks like:
• Subjective or objective language sentence
• Question classification: about person, location, number, ...

​

Code

See notebook:

http://medialab.di.unipi.it:8000/hub/user-redirect/lab/tree/HLT/Lectures/CnnNLP.ipynb

Sentiment Analysis on Tweets

​

​

​

Evolution

• SemEval Shared Task Competition
• 2013, Task 2
• 2014, Task 9
• 2015, Task 10
• 2016
• 2017
• Evolution of technology:
• Top system in 2013: SVM with sentiment lexicons and many lexical features
• Top system in 2016: CNN with word embeddings
• In 2017: most systems used CNN or variants

​

Semeval 2013 - Examples

SemEval 2013, Task 2

Best Submission:

NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets, Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu, In Proceedings of the seventh international workshop on Semantic Evaluation Exercises (SemEval-2013), June 2013, Atlanta, USA.

​

Approach:

SVM with lots of handcrafted features:

• word ngrams, char ngrams, all-caps, POS, elongated words, emoticons, negation, etc.
• sentiment lexicon: counts of polarized words, sum of scores, max score, last score

SemEval 2015 – Task 10

Best Submission:

A. Severyn, A. Moschitti. 2015. UNITN: Training Deep Convolutional Neural Network for Twitter Sentiment Classification. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 464–469, Denver, Colorado, June 4-5, 2015.�https://www.aclweb.org/anthology/S15-2079

CNN for Sentiment Classification

1. Embeddings Layer, R^d (d = 300)
2. Convolutional Layer with Relu activation

Multiple filters of sliding windows of various sizes h

c_i = f(F ⊗ S_i:i+h−1 + b)

• max-pooling layer
• dropout layer
• linear layer with tanh activation
• softmax layer

S

Frobenius elementwise

matrix product

Distant Supervision

• A. Severyn and A. Moschitti, UNITN at SemEval 2015 Task 10.
• Word embeddings from plain text are completely clueless about their sentiment behavior
• Distant supervision approach using our convolutional neural network to further refine the embeddings
• Collected 10M tweets treating tweets containing positive emoticons, used as distantly supervised labels to train sentiment-aware embeddings

Results of UNITN on SemEval 2015

Phrase-level subtask A

Message-level subtask B

Sentiment Specific Word Embeddings

• Sentiment Specific Word Embeddings

​

​

​

​

​

• Uses an annotated corpus with polarities (e.g. tweets)
• SS Word Embeddings achieve SotA accuracy on tweet sentiment classification
• G. Attardi, D. Saertiano. UniPi at SemEval 2016 Task 4: Convolutional Neural Networks for Sentiment Classification.�https://www.aclweb.org/anthology/S/S16/S16-1033.pdf

​

U

the cat sits on

LM likelihood + Polarity

Learning SS Embeddings

Semeval 2015 Sentiment on Tweets

SwissCheese at SemEval 2016

Three-phase procedure:

• creation of word embeddings for initialization of the first layer. Word2vec on an unlabeled corpus of 200M tweets.
• distant supervised phase, where the network weights and word embeddings are trained to capture aspects related to sentiment. Emoticons used to infer the polarity of a balanced set of 90M tweets.
• supervised phase, where the network is trained on the provided supervised training data.

​

Ensemble of Classifiers

• combining the outputs of two 2-layer CNNs having similar architectures but differing in the choice of certain parameters (such as the number of convolutional filters).
• networks were also initialized using different word embeddings and used slightly different training data for the distant supervised phase.
• A total of 7 outputs were combined

Results

Breakdown over all test sets

Sentiment Classification from a single neuron

• A char-level LSTM with 4096 units has been trained on 82 millions reviews from Amazon.
• The model is trained only to predict the next character in the text
• After training one of the units had a very high correlation with sentiment, resulting in state-of-the-art accuracy when used as a classifier.
• The model can be used to generate text.
• By setting the value of the sentiment unit, one can control the sentiment of the resulting text.

Blog post - Radford et al. Learning to Generate Reviews and Discovering Sentiment. Arxiv 1704.01444

Follow up

Zhang and Wallace (2015) A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification

https://arxiv.org/pdf/1510.03820.pdf

​

Regularization

A pitfall when fine-tuning word vectors

• Setting: We are training a logistic regression classification model for movie review sentiment using single words.
• In the training data we have “TV” and “telly”
• In the testing data we have “television”
• The pre-trained word vectors have all three similar:

​

​

​

​

​

​

• Question: What happens when we update the word vectors?

​

TV

telly

television

A pitfall when fine-tuning word vectors

• Question: What happens when we update the word vectors?
• Answer:
• Those words that are in the training data move around
• “TV”and“telly”
• Words not in the training data stay where they were
• “television”

​

TV

telly

television

What to do

• Question: Should I use available “pre-trained” word vectors Answer:
• Almost always, yes!
• They are trained on a huge amount of data, and so they will know about words not in your training data and will know more about words that are in your training data
• Have 100s of millions of words of data? Okay to start random
• Question: Should I update (“fine tune”) my own word vectors?
• Answer:
• If you only have a small training data set, don’t train the word vectors
• If you have have a large dataset, it probably will work better to�train = update = fine-tune word vectors to the task

​