3 of 62

WHY RNNs?

As soon as we introduce a new architecture, questions arises in the minds of everyone, that is why do we need it? What does it do better that the other models weren’t capable of? So, now we are going to answer these questions!

4 of 62

RNNs do sequence modelling like never before!

In various applications where we require our model to give an output based on a given sequence of information, it could be anything……

5 of 62

SPEECH RECOGNITION:

Quite a common application these days(everyone with smartphone will know about this)

6 of 62

SENTIMENT CLASSIFICATION:

Another popular application of sequence model.

The output can be in the form of ratings or can depict the sentiment of sentence.

7 of 62

Or a sequence of images in case of video analysis

8 of 62

Let’s take the case for language modelling to delve deeper into the architecture of RNNs.

hidden states

input sequence (any length)

…

Core idea: Apply the same weights repeatedly

outputs (optional)

9 of 62

1. The Simple RNN Language Model

the students

opened

their

words / one-hot vectors

books

laptops

word embeddings

zoo

output distribution

hidden states

is the initial hidden state

10 of 62

Training an RNN Language Model

Get a big corpus of text which is a sequence of words
Feed into RNN-LM; compute output distribution

for every step t.

i.e., predict probability dist of every word, given words so far

Loss function on step t is cross-entropy between predicted probability distribution , and the true next word (one-hot for ):

Average this to get overall loss for entire training set:

11 of 62

Training an RNN Language Model

= negative log prob of “students”

Loss

Predicted prob dists

…

Corpus

the

students

opened

their

exams

…

12 of 62

Training an RNN Language Model

Loss

Predicted prob dists

…

= negative log prob of “opened”

Corpus

the

students

opened

their

exams

…

13 of 62

Training an RNN Language Model

Loss

Predicted prob dists

…

= negative log prob of “their”

Corpus

the

students

opened

their

exams

…

14 of 62

Training an RNN Language Model

Loss

Predicted prob dists

…

= negative log prob of “exams”

Corpus

the

students

opened

their

exams

…

15 of 62

Training an RNN Language Model

+ … =

…

Loss

Predicted prob dists

“Teacher forcing”

Corpus

the

students

opened

their

exams

…

16 of 62

Training a RNN Language Model

However: Computing loss and gradients across entire corpus expensive!

is too

In practice, consider as a sentence (or a document)

Recall: Stochastic Gradient Descent allows us to compute loss and gradients for small chunk of data, and update.

Compute loss for a sentence (actually, a batch of sentences), compute gradients and update weights. Repeat.

17 of 62

Training the parameters of RNNs: Backpropagation for RNNs

…

Question: What’s the derivative of

w.r.t. the repeated weight matrix ?

Answer:

“The gradient w.r.t. a repeated weight is the sum of the gradient

w.r.t. each time it appears”

Why?

18 of 62

Multivariable Chain Rule

Gradients sum at outward branches!

Source:

https://www.khanacademy.org/math/multivariable-calculus/multivariable-derivatives/differentiating-vector-valued-functions/a/multivariable-chain-rule-simple-version

19 of 62

Backpropagation for RNNs: Proof sketch

…

In our example:

equals

Apply the multivariable chain rule:

= 1

Source:

https://www.khanacademy.org/math/multivariable-calculus/multivariable-derivatives/differentiating-vector-valued-functions/a/multivariable-chain-rule-simple-version

20 of 62

Backpropagation for RNNs

…

Question: How do we calculate this?

Answer: Backpropagate over timesteps

i=t,…,0, summing gradients as you go.

This algorithm is called “backpropagation through time” [Werbos, P.G., 1988, Neural Networks 1, and others]

21 of 62

Generating text with a RNN Language Model

Just like a n-gram Language Model, you can use an RNN Language Model to generate text by repeated sampling. Sampled output becomes next step’s input.

my favorite

season

…

favorite

sample

season

sample

spring

sample

spring

22 of 62

Generating text with an RNN Language Model

Let’s have some fun!

You can train an RNN-LM on any kind of text, then generate text in that style.
RNN-LM trained on Harry Potter:

Source: https://medium.com/deep-writing/harry-potter-written-by-artificial-intelligence-8a9431803da6

23 of 62

Why should we care about Language Modeling?

Language Modeling is a benchmark task that helps us measure our progress on understanding language

Language Modeling is a subcomponent of many NLP tasks, especially those involving generating text or estimating the probability of text:

Predictive typing
Speech recognition
Handwriting recognition
Spelling/grammar correction
Authorship identification
Machine translation
Summarization
Dialogue
etc.

24 of 62

Recap

Language Model: A system that predicts the next word

Recurrent Neural Network: A family of neural networks that:

Take sequential input of any length
Apply the same weights on each step
Can optionally produce output on each step

Recurrent Neural Network ≠ Language Model

We’ve shown that RNNs are a great way to build a LM

But RNNs are useful for much more!

25 of 62

2. Other RNN uses: RNNs can be used for sequence tagging

e.g., part-of-speech tagging, named entity recognition

knocked

over

the

vase

the startled cat

VBN

26 of 62

RNNs can be used for sentence classification

e.g., sentiment classification

the

movie

lot

overall

enjoyed

positive

Sentence encoding

How to compute

sentence encoding?

27 of 62

RNNs can be used for sentence classification

the

movie

lot

overall

enjoyed

positive

Sentence encoding

^equals

How to compute sentence encoding?

Basic way:

Use final hidden state

e.g., sentiment classification

28 of 62

RNNs can be used for sentence classification

the

movie

lot

overall

enjoyed

positive

Sentence encoding

How to compute sentence encoding?

Usually better: Take element-wise max or mean of all hidden states

e.g., sentiment classification

29 of 62

RNNs can be used as a language encoder module

e.g., question answering, machine translation, many other tasks!

Context: Ludwig van Beethoven was a German composer and pianist. A crucial figure …

Beethoven ?

^Question:what nationality was

Here the RNN acts as an encoder for the Question (the hidden states represent the Question). The encoder is part of a larger neural system.

Answer: German

lots ^{of neural}

arch^itecture

^{lots o}f neural

^architecture

30 of 62

RNN-LMs can be used to generate text

e.g., speech recognition, machine translation, summarization

RNN-LM

what’s the

the weather

what’s

This is an example of a conditional language model.

We’ll see Machine Translation in much more detail next class.

Input (audio)

<START>

conditioning

31 of 62

3. Problems with Vanishing and Exploding Gradients

32 of 62

Vanishing gradient intuition

33 of 62

Vanishing gradient intuition

chain rule!

34 of 62

Vanishing gradient intuition

chain rule!

35 of 62

Vanishing gradient intuition

chain rule!

36 of 62

Vanishing gradient intuition

What happens if these are small?

Vanishing gradient problem: When these are small, the gradient signal gets smaller and smaller as it backpropagates further

37 of 62

Vanishing gradient proof sketch (linear case)

Recall:
What if were the identity function, ?

Source: “On the difficulty of training recurrent neural networks”, Pascanu et al, 2013. http://proceedings.mlr.press/v28/pascanu13.pdf

(and supplemental materials), at http://proceedings.mlr.press/v28/pascanu13-supp.pdf

If W_his “small”, then this term gets exponentially problematic as becomes large

(chain rule)

Consider the gradient of the loss on step , with respect to the hidden state on some previous step . Let

(chain rule)

(value of )

38 of 62

Vanishing gradient proof sketch (linear case)

What’s wrong with ?

Consider if the eigenvalues of are all less than 1:

We can write

as a basis:

What about nonlinear activations (i.e., what we use?)

Pretty much the same thing, except the proof requires

for some

dependent on dimensionality and

Source: “On the difficulty of training recurrent neural networks”, Pascanu et al, 2013. http://proceedings.mlr.press/v28/pascanu13.pdf

(and supplemental materials), at http://proceedings.mlr.press/v28/pascanu13-supp.pdf

(eigenvectors) using the eigenvectors of

Approaches 0 as

grows, so gradient vanishes

sufficient but

not necessary

39 of 62

Why is vanishing gradient a problem?

Gradient signal from far away is lost because it’s much smaller than gradient signal from close-by.

So, model weights are updated only with respect to near effects, not long-term effects.

40 of 62

Effect of vanishing gradient on RNN-LM

LM task: When she tried to print her tickets, she found that the printer was out of toner. She went to the stationery store to buy more toner. It was very overpriced. After installing the toner into the printer, she finally printed her

To learn from this training example, the RNN-LM needs to model the dependency between “tickets” on the 7^thstep and the target word “tickets” at the end.

But if gradient is small, the model can’t learn this dependency

So, the model is unable to predict similar long-distance dependencies at test time

41 of 62

Why is exploding gradient a problem?

If the gradient becomes too big, then the SGD update step becomes too big:

learning rate

gradient

This can cause bad updates: we take too large a step and reach a weird and bad parameter configuration (with large loss)

You think you’ve found a hill to climb, but suddenly you’re in Iowa

In the worst case, this will result in Inf or NaN in your network (then you have to restart training from an earlier checkpoint)

42 of 62

Gradient clipping: solution for exploding gradient

Gradient clipping: if the norm of the gradient is greater than some threshold, scale it down before applying SGD update

Intuition: take a step in the same direction, but a smaller step

In practice, remembering to clip gradients is important, but exploding gradients are an easy problem to solve

Source: “On the difficulty of training recurrent neural networks”, Pascanu et al, 2013. http://proceedings.mlr.press/v28/pascanu13.pdf

43 of 62

How to fix the vanishing gradient problem?

The main problem is that it’s too difficult for the RNN to learn to preserve information over many timesteps.

In a vanilla RNN, the hidden state is constantly being rewritten

How about a RNN with separate memory?

44 of 62

4. Long Short-Term Memory RNNs (LSTMs)

A type of RNN proposed by Hochreiter and Schmidhuber in 1997 as a solution to the vanishing gradients problem.

Everyone cites that paper but really a crucial part of the modern LSTM is from Gers et al. (2000) 💜

On step t, there is a hidden state and a cell state

Both are vectors length n
The cell stores long-term information
The LSTM can read, erase, and write information from the cell

The cell becomes conceptually rather like RAM in a computer

The selection of which information is erased/written/read is controlled by three corresponding gates

The gates are also vectors length n
On each timestep, each element of the gates can be open (1), closed (0), or somewhere in-between
The gates are dynamic: their value is computed based on the current context

“Long short-term memory”, Hochreiter and Schmidhuber, 1997. https://www.bioinf.jku.at/publications/older/2604.pdf “Learning to Forget: Continual Prediction with LSTM”, Gers, Schmidhuber, and Cummins, 2000. https://dl.acm.org/doi/10.1162/089976600300015015

45 of 62

Long Short-Term Memory (LSTM)

We have a sequence of inputs 𝑥^(𝑡), and we will compute a sequence of hidden states ℎ^(𝑡)and cell states

𝑐^(𝑡). On timestep t:

All these are vectors of same length n

Forget gate: controls what is kept vs forgotten, from previous cell state

Input gate: controls what parts of the new cell content are written to cell

Output gate: controls what parts of cell are output to hidden state

New cell content: this is the new content to be written to the cell

Cell state: erase (“forget”) some content from last cell state, and write (“input”) some new cell content

Hidden state: read (“output”) some content from the cell

Sigmoid function: all gate values are between 0 and 1

Gates are applied using element-wise (or Hadamard) product: ⊙

46 of 62

Long Short-Term Memory (LSTM)

You can think of the LSTM equations visually like this:

Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

47 of 62

Long Short-Term Memory (LSTM)

^ct-1

^ht-1

h_t

i_t

o_t

c_t

~_c

You can think of the LSTM equations visually like this:

Compute the forget gate

Forget some cell content

Compute the input gate

Compute the new cell content

Compute the output gate

Write some new cell content

Output some cell content to the hidden state

The + sign is the secret!

Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

48 of 62

How does LSTM solve vanishing gradients?

The LSTM architecture makes it easier for the RNN to preserve information over many timesteps

e.g., if the forget gate is set to 1 for a cell dimension and the input gate set to 0, then the information of that cell is preserved indefinitely.
In contrast, it’s harder for a vanilla RNN to learn a recurrent weight matrix W_hthat preserves info in the hidden state
In practice, you get about 100 timesteps rather than about 7

LSTM doesn’t guarantee that there is no vanishing/exploding gradient, but it does provide an easier way for the model to learn long-distance dependencies

49 of 62

LSTMs: real-world success

In 2013–2015, LSTMs started achieving state-of-the-art results

Successful tasks include handwriting recognition, speech recognition, machine translation, parsing, and image captioning, as well as language models
LSTMs became the dominant approach for most NLP tasks

Now (2021), other approaches (e.g., Transformers) have become dominant for many tasks

For example, in WMT (a Machine Translation conference + competition):
In WMT 2016, the summary report contains “RNN” 44 times
In WMT 2019: “RNN” 7 times, ”Transformer” 105 times

Source: "Findings of the 2016 Conference on Machine Translation (WMT16)", Bojar et al. 2016, http://www.statmt.org/wmt16/pdf/W16-2301.pdf Source: "Findings of the 2018 Conference on Machine Translation (WMT18)", Bojar et al. 2018, http://www.statmt.org/wmt18/pdf/WMT028.pdf Source: "Findings of the 2019Conference on Machine Translation (WMT19)", Barrault et al. 2019, http://www.statmt.org/wmt18/pdf/WMT028.pdf

50 of 62

Gated Recurrent Units (GRU)

Proposed by Cho et al. in 2014 as a simpler alternative to the LSTM.
On each timestep t we have input and hidden state (no cell state).

"Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation", Cho et al. 2014, https://arxiv.org/pdf/1406.1078v3.pdf

Update gate: controls what parts of hidden state are updated vs preserved

Reset gate: controls what parts of previous hidden state are used to compute new content

Hidden state: update gate simultaneously controls what is kept from previous hidden state, and what is updated to new hidden state content

New hidden state content: reset gate selects useful parts of prev hidden state. Use this and current input to compute new hidden content.

How does this solve vanishing gradient? Like LSTM, GRU makes it easier to retain info long-term (e.g. by setting update gate to 0)

51 of 62

Is vanishing/exploding gradient just a RNN problem?

For example:

Residual connections aka “ResNet”
Also known as skip-connections
The identity connection preserves information by default
This makes deep networks much easier to train

No! It can be a problem for all neural architectures (including feed-forward and convolutional), especially very deep ones.

Due to chain rule / choice of nonlinearity function, gradient can become vanishingly small as it backpropagates
Thus, lower layers are learned very slowly (hard to train)

Solution: lots of new deep feedforward/convolutional architectures that add more direct connections (thus allowing the gradient to flow)

"Deep Residual Learning for Image Recognition", He et al, 2015. https://arxiv.org/pdf/1512.03385.pdf

52 of 62

Is vanishing/exploding gradient just a RNN problem?

Highway connections aka “HighwayNet”
Similar to residual connections, but the identity connection vs the transformation layer is controlled by a dynamic gate
Inspired by LSTMs, but applied to deep feedforward/convolutional networks

Solution: lots of new deep feedforward/convolutional architectures that add more direct connections (thus allowing the gradient to flow)

Other methods:

Dense connections aka “DenseNet”
Directly connect each layer to all future layers!

”Densely Connected Convolutional Networks", Huang et al, 2017. https://arxiv.org/pdf/1608.06993.pdf

”Highway Networks", Srivastava et al, 2015. https://arxiv.org/pdf/1505.00387.pdf

53 of 62

Is vanishing/exploding gradient just a RNN problem?

No! It can be a problem for all neural architectures (including feed-forward and convolutional), especially very deep ones.

Due to chain rule / choice of nonlinearity function, gradient can become vanishingly small as it backpropagates
Thus, lower layers are learned very slowly (hard to train)

Solution: lots of new deep feedforward/convolutional architectures that add more direct connections (thus allowing the gradient to flow)

Conclusion: Though vanishing/exploding gradients are a general problem, RNNs are particularly unstable due to the repeated multiplication by the same weight matrix [Bengio et al, 1994]

”Learning Long-Term Dependencies with Gradient Descent is Difficult", Bengio et al. 1994, http://ai.dinfo.unifi.it/paolo//ps/tnn-94-gradient.pdf

54 of 62

5. Bidirectional and Multi-layer RNNs: motivation

terribly

exciting

the

movie was

positive

Sentence encoding

elemen^{t-wise mean/max}

^{element-wise mean}/max

We can regard this hidden state as a representation of the word “terribly” in the context of this sentence. We call this a contextual representation.

These contextual representations only contain information about the left context (e.g. “the movie was”).

What about right

context?

In this example, “exciting” is in the right context and this modifies the meaning of “terribly” (from negative to positive)

Task: Sentiment Classification

55 of 62

Bidirectional RNNs

was ^terribly

exciting

the

movie

Forward RNN

Backward RNN

Concatenated hidden states

This contextual representation of “terribly” has both left and right context!

56 of 62

Bidirectional RNNs

Forward RNN

Backward RNN Concatenated hidden states

This is a general notation to mean “compute one forward step of the RNN” – it could be a vanilla, LSTM or GRU computation.

We regard this as “the hidden state” of a bidirectional RNN. This is what we pass on to the next parts of the network.

Generally, these two RNNs have separate weights

On timestep t:

57 of 62

Bidirectional RNNs: simplified diagram

terribly

exciting

the

movie was

The two-way arrows indicate bidirectionality and the depicted hidden states are assumed to be the concatenated forwards+backwards states

58 of 62

Bidirectional RNNs

Note: bidirectional RNNs are only applicable if you have access to the entire input sequence

They are not applicable to Language Modeling, because in LM you only have left context available.

If you do have entire input sequence (e.g., any kind of encoding), bidirectionality is powerful (you should use it by default).

For example, BERT (Bidirectional Encoder Representations from Transformers) is a powerful pretrained contextual representation system built on bidirectionality.

You will learn more about transformers include BERT in a couple of weeks!

59 of 62

Multi-layer RNNs

RNNs are already “deep” on one dimension (they unroll over many timesteps)

We can also make them “deep” in another dimension by applying multiple RNNs – this is a multi-layer RNN.

This allows the network to compute more complex representations

The lower RNNs should compute lower-level features and the higher RNNs should compute higher-level features.

Multi-layer RNNs are also called stacked RNNs.

60 of 62

Multi-layer RNNs

was ^terribly

exciting

the

movie

RNN layer 1

RNN layer 2

RNN layer 3

The hidden states from RNN layer i

are the inputs to RNN layer i+1

61 of 62

Multi-layer RNNs in practice

High-performing RNNs are often multi-layer (but aren’t as deep as convolutional or feed-forward networks)

For example: In a 2017 paper, Britz et al find that for Neural Machine Translation, 2 to 4 layers is best for the encoder RNN, and 4 layers is best for the decoder RNN

Usually, skip-connections/dense-connections are needed to train deeper RNNs (e.g., 8 layers)

Transformer-based networks (e.g., BERT) are usually deeper, like 12 or 24 layers.

You will learn about Transformers later; they have a lot of skipping-like connections

“Massive Exploration of Neural Machine Translation Architecutres”, Britz et al, 2017. https://arxiv.org/pdf/1703.03906.pdf

62 of 62

In summary

Lots of new information today! What are some of the practical takeaways?

1. LSTMs are powerful

2. Clip your gradients

3. Use bidirectionality when possible

4. Multi-layer RNNs are more powerful, but you might need skip connections if it’s deep