SDS4299

Natural Language Processing 2

Going Deeper with Neural Nets

Any Burning Questions for us?

Slido Code: #4299

Our Team

Recap

01

Last Time, In SDS1101

We covered:

Text Preprocessing

Naive Bayes Classifier

We got a good result from Naive Bayes..

… But we can do better!

NLP is Hard…

Machines can’t understand language, but we can!

It’s all thanks to this!

So, why don’t we try to mimic it?

Introduction to Neural Networks

Neural networks were inspired by Neurons

Neural networks were inspired by Neurons

Signal

Signal

Signal

Signal

Signal

Input layer

Hidden layer 1

Hidden layer 2

Output layer

𝑤₁

𝑤₂

Perceptron

​

Activation functions

What is an activation function?

• Takes in the weighted sum and decides if the neuron should “fire” (activate)

In this example, the neuron would activate and output 1 if 𝑤₀ + 𝑥 ₁ᐧ𝑤₁ + 𝑥₂ᐧ𝑤₂ ≥ 0

Non-linear activation functions

Sigmoid or Logistic Activation Function

Output is bound to [0,1] and can be used to predict probability

Non-linear activation functions

Tanh or Hyperbolic Tangent Activation Function

Similar to sigmoid, but ranges from -1 to 1

Non-linear activation functions

ReLU (Rectified Linear Unit) Activation Function

Quantifying Loss

For the rest of the workshop, we will assume a classification task with binary cross entropy as our cost function

Gradient Descent

An iterative solution to optimize the loss

We cannot jump to the value, so let’s walk there instead.

​

Let us say that the direction of steepest descent is represented by a vector v.

​

Then, we have to move some distance α in direction v

​

Once we finish the step, let us again check for direction of steepest descent!

​

We then stop at the minima - Our Goal!

Gradient Descent

Choosing the learning rate α

Gradient Descent

Weight update

Let us consolidate every individual weight in the network into a huge column vector: Ө

Then, Ө_updated = Ө - αv

If α is set right, we should be able to reach a minimum!

We should be able to tweak our weights to minimise the cost function now!

Backpropagation

We need the ability to calculate the direction of steepest descent!

But how?

Derivative -

Slope at a point for 2D

Gradient:

Direction of steepest ascent

Backpropagation

z₁

ŷ

w₁

w₂

L(W)

=

*

*

Repeat this for every weight in the network using gradients from later layers

How does a small change in one weight (w₁) affect the final loss L(W)

x

Backpropagation

Flow of information

Backpropagation algorithm:

1. Take the derivative (gradient) of the loss with respect to each parameter
2. Shift parameters in order to minimize loss

Backpropagation

We just need to run the backpropagation algo for each weight in the neural net, to get our gradient

We use this gradient to take a step, improving our net slightly

We finally stop when our cost function is minimized!

Dropout

At every iteration, randomly selects some neurons and removes it along with all of its incoming & outgoing connections

​

​

​

The probability that these neurons will be selected depends on the rate specified (also a hyperparameter)

Embeddings

02

What are Embeddings? �Why do we need them?

What?

Low dimensional continuous vector representations

Why?

‘Hello!’

1

[ 6

1

4

3

5

7

9 ]

Input

Output

Machine readable format

Can I use one-hot encoding to ‘embed’ my words?

One-hot Encoding

1 Step: Create a vector for each unique word

​

​

E.g

​

​

​

​

​

​

​

​

​

​

Is this machine readable?

​

​

​

​

​

​

​

​

​

​

Is this a good embedding?

​

​

​

​

​

​

​

​

​

​

YES!

​

​

​

​

​

​

​

​

​

​

NO :(

​

​

​

​

​

​

​

​

​

​

Why not?

1. As the number of words INCREASES, the number of dimension of our vector INCREASES, computational complexity INCREASES, time taken INCREASES

​

1000 words (in our corpus) = 1000 dimensional vector

​

• High dimensional sparse matrix don’t work well with many machine learning models

​

• No semantic similarity between any of the vectors

​

​

​

​

​

Semantic Similarity

Ideally, the vectors used should ‘capture the meaning’ of our words

Man-made equipment are close together

​

​

​

​

​

​

​

​

Animals are close together

​

​

​

​

​

​

​

​

Types of Embeddings available

Custom Embeddings

Pre-trained Embeddings

1. Embeddings built specifically for your dataset
2. Build it on your own training data using word2Vec, TF-IDF etc

​

(Note: one-hot encoding is �also in this category!)

1. Already trained on a large dataset
2. Use the pre-defined vectors corresponding to the words in our dataset

​

Examples:GloVe, Word2Vec

What is Word2Vec?

Single-layered Neural Network based embedding model created in 2013 by researchers at Google

CBOW

In Continuous Bag of Words,

1. The words before & after the ‘target’ word are given as inputs to the model�
2. Model predicts the probability of the ‘target’ word

​

​

​

Skip-gram

In Skip-gram,

• The ‘target’ word is given as inputs to the model�
• Model predicts the probability of the neighbouring words before & after it

​

​

​

How do we generate vectors?

In both models, we are interested in the weights of the hidden layer!

What is GloVe?

In addition to semantic similarity, it takes into account the frequency of each word in the corpus

Created in 2014 by Stanford

Let’s work it out in our Colab file:�

tinyurl.com/nlp2colab�

RNN

03

Recurrent Neural Networks

Recurrent Neural Networks (RNN)

• RNN is a special type of artificial neural network that is adapted to work for time series or data that involves sequences

​

• RNNs have the concept of “memory” that helps them store the states or information of previous inputs to generate the next output of the sequence

​

Recurrent Neural Network

• Standard neural networks go from input to output in one direction

​

• In contrast, RNNs has loops in them which allows information to persists over time.

​

• At time step t, the RNN takes in as input x_t, and computes y_t. In addition to the output, it computes an internal state h_t, and passes this information from this time step to the next

Unfolding an RNN

Recurrence relation

A recurrence relation is applied at every time step to process a sequence:

State at previous time step

The same function and set of parameters are used at every time step

Backpropagation Through Time (BPTT)

RNN

x_t-1

y_t-1

RNN

x_t

y_t

RNN

x_t+1

y_t+1

h_t-1

h_t

Let’s go back to the Colab and see how a simple RNN model is implemented!

LSTM

04

Long Short Term Memory

Problems with RNNs

Backpropagation Through Time (BPTT)

Gradient

Gradient

Gradient

Gradient Slowly Disappears!!

So what?

Neural Nets learn using the gradient

So, what happens if the gradient vanishes??

The Network Doesn’t Learn!

Specifically, the network becomes focused on the last few inputs

Remembers this

Forgets this

The network has an extremely short term memory!

How do we fix this?

We have to add some memory!

“Forget” the less useful stuff, and implement some memory!

RNNs

h₁

h₂

h₃

Information carried along by the hidden state

Hidden state is dominated by recency bias

Implementing Memory

Key Idea

h

c

While passing along the hidden state, pass along a “cell state” as well!

Carries local info

Carries important info from the past!

Anatomy of an LSTM Cell

Seems complicated!

Let’s break it down.

3 parts to it!

Forget

Input

Output

• Get rid of irrelevant info

• Update Cell State

• Get new hidden vector to pass along

Forget Gate

h_i-1

x_i

h_i-1+ x_i

Sigmoid

f_i

Vector encodes what previous info we need to forget!

Forget vector

Input Gate

h_i-1

x_i

h_i-1+ x_i

Sigmoid

i_i

Input vector

tanh

c’_i

Candidate vector

i_i *c’_i

Update applied to cell state

Applying the Update

f_i

c_i-1

f_i*c_i-1

i_i *c’_i

f_i*c_i-1+ i_i *c’_i

c_i

New cell state

Output Gate

c_i

h_i-1+ x_i

tanh

Sigmoid

Multiply

h_i

Putting it all together

X

X

+

X

Forget Gate

Input Gate

Output Gate

c_i-1

h_i-1

x_i

h_i-1+ x_i

h_i-1+ x_i

h_i-1+ x_i

f_i

i_i

c’_i

i_i *c’_i

c_i

h_i

c_i

Time for a practical demo!

Transformers

05

Pay Attention!

LSTMs are great, but…..

They take ages to train

Still have some issues in training longer sequences

They still use ALL the info of the sequence!

Wait what?

Consider the following:

I

am

a

human

Input

Eu

sou

um

humano

Output

Machine

Translation

Position matters!

For a sequence to sequence mapping, the order and position matters!

I

am

a

human

Input

Output

This matters ..

…more than this!

We need to account for each individual input along with the context!

Encoder - Decoder Models Explained

Key Idea: Use 2 LSTMs!

Encoder LSTM

Converts Input to Fixed Length representation

Decoder LSTM

Converts Fixed Length Representation to Output

Great for sequence to sequence tasks!

A 2-part model

Encoder

Decoder

Context

Input

Output

+ All hidden states!

Why transformers are so good

Local and Global!

Idea: Use context vector + hidden state at each timestep in the decoder!

Decoder hidden state

h₁

h₂

h₃

Hidden states at every time

Compute context

How Do We Know What’s Relevant?

We get inspired by search engines!

Query

Keys

Values

Mapped onto

Get Best Matches

Multiply

Score

From Encoder

From Decoder

Computing an Output

For a particular output:

Assign a score to each hidden state!

Softmax the scores

Scale the hidden states by the softmaxed scores

+

+

Add the hidden states!

Making a prediction!

+

+

Hidden State: Contains global info

Context Vector - contains local info

+

Output

Let’s Move to the Colab

Current State of Transformer Models

Transformers are the current state of the art models for NLP

• 2017- Transformers introduced
• 2018 - Improved encoding with BERT
• 2019 - XLNet and ERNIE, as well as GPT2
• 2020 - GPT3 released

Further research is still ongoing!

All of these models use the attention mechanism!

Conclusion

06

Summary of Results

Simple RNN

LSTM

Basic NN

78%

57%

87%

Keep in mind that all these networks are shallow!

Transformers

93%

Quick Summary

Recap on Deep Learning & NLP 1 Workshop��Embeddings��RNNs

​

LSTMs

What’s next?

Recap on Deep Learning & NLP 1 Workshop��Embeddings��RNNs

​

LSTMs

Own NLP Project?

Any Further Questions for us?

Slido Code: #4299