2 of 33

1. The Simple RNN Language Model

the students

opened

their

words / one-hot vectors

books

laptops

word embeddings

zoo

output distribution

Note: this input sequence could be much longer now!

hidden states

is the initial hidden state

3 of 33

Training the parameters of RNNs: Backpropagation for RNNs

…

Question: What’s the derivative of

w.r.t. the repeated weight matrix ?

Answer:

“The gradient w.r.t. a repeated weight is the sum of the gradient w.r.t. each time it appears”

Why?

4 of 33

Backpropagation for RNNs

…

Question: How do we calculate this?

Answer: Backpropagate over timesteps i=t,…,0, summing gradients as you go.

This algorithm is called “backpropagation through time” [Werbos, P.G., 1988, Neural Networks 1, and others]

5 of 33

3. Problems with Vanishing and Exploding Gradients

We will backpropagate the overall loss.

6 of 33

Vanishing gradient intuition

We will backpropagate the overall loss.

7 of 33

Vanishing gradient intuition

chain rule!

8 of 33

Vanishing gradient intuition

chain rule!

9 of 33

Vanishing gradient intuition

chain rule!

10 of 33

Vanishing gradient intuition

What happens if these are small?

Vanishing gradient problem: When these are small, the gradient signal gets smaller and smaller as it backpropagates further

11 of 33

Why is vanishing gradient a problem?

Gradient signal from far away is lost because it’s much smaller than gradient signal from close-by.

So, model weights are updated only with respect to near effects, not long-term effects.

12 of 33

Effect of vanishing gradient on RNN-LM

LM task: When she tried to print her tickets, she found that the printer was out of toner. She went to the stationery store to buy more toner. It was very overpriced. After installing the toner into the printer, she finally printed her

To learn from this training example, the RNN-LM needs to model the dependency between “tickets” on the 7^thstep and the target word “tickets” at the end.

But if gradient is small, the model can’t learn this dependency

So, the model is unable to predict similar long-distance dependencies at test time

13 of 33

Another example of errors by RNN LMs due to vanishing gradient:

The writer of the books

The LM is asked to predict the next word between ‘is’ and ‘are’

Due to vanishing gradients, RNN-LMs are prone to make mistakes in identifying such subject-verb dependencies [Linzen et al 2016]

14 of 33

How to fix the vanishing gradient problem?

The main problem is that it’s too difficult for the RNN to learn to preserve information over many timesteps.

In a vanilla RNN, the hidden state is constantly being rewritten

How about a RNN with separate memory?

We want this separate memory to be more flexible, so that we can use it to preserve some specific information (instead of being rewritten at every stage).

15 of 33

4. Long Short-Term Memory RNNs (LSTMs)

A type of RNN proposed by Hochreiter and Schmidhuber in 1997 as a solution to the vanishing gradients

problem.

Everyone cites that paper but really a crucial part of the modern LSTM is from Gers et al. (2000)

On step t, there is a hidden state and a cell state

Both are vectors length n
The cell stores long-term information
The LSTM can read, erase, and write information from the cell

The cell becomes conceptually rather like RAM in a computer

The selection of which information is erased/written/read is controlled by three corresponding gates

The gates are also vectors length n

On each timestep, each element of the gates can be open (1), closed (0), or somewhere in-between
The gates are dynamic: their value is computed based on the current context

https://www.bioinf.jku.at/publications/older/2604.pdf “Learning to Forget: Continual Prediction with LSTM”, Gers, Schmidhuber, and Cummins, 2000. https://dl.acm.org/doi/10.1162/089976600300015015 “Long short-term memory”, Hochreiter and Schmidhuber, 1997.

Probabilistic gates: each element of a gate is open with a probability

16 of 33

Long Short-Term Memory (LSTM)

We have a sequence of inputs 𝑥^(𝑡), and we will compute a sequence of hidden states ℎ^(𝑡)and cell states

𝑐^(𝑡). On timestep t:

All these are vectors of same length n

Forget gate: controls what is kept vs forgotten, from previous cell state

Input gate: controls what parts of the new cell content are written to cell

Output gate: controls what parts of cell are output to hidden state

New cell content: this is the new content to be written to the cell

Cell state: erase (“forget”) some content from last cell state, and write (“input”) some new cell content

Hidden state: read (“output”) some content from the cell

Sigmoid function: all gate values are between 0 and 1

Gates are applied using element-wise

These 3 equations are exactly similar to the eqn. for hidden state in vanilla RNN

(or Hadamard) product: ⊙

Each gate has its own set of parameters W, U, b (which will be learned)

17 of 33

Long Short-Term Memory (LSTM)

We have a sequence of inputs 𝑥^(𝑡), and we will compute a sequence of hidden states ℎ^(𝑡)and cell states

𝑐^(𝑡). On timestep t:

All these are vectors of same length n

Forget gate: controls what is kept vs forgotten, from previous cell state

Input gate: controls what parts of the new cell content are written to cell

Output gate: controls what parts of cell are output to hidden state

New cell content: this is the new content to be written to the cell

Cell state: erase (“forget”) some content from last cell state, and write (“input”) some new cell content

Hidden state: read (“output”) some content from the cell

Sigmoid function: all gate values are between 0 and 1

Gates are applied using element-wise

For the new cell state, we want to remember some, but probably not all, of the previous cell state

(or Hadamard) product: ⊙

A candidate cell update (to be used to compute the new cell state)

To get the new cell state c(t), sum

product of old cell state with forget gate
product of the candidate cell update with the input gate

The new hidden state, obtained from new cell state

18 of 33

Long Short-Term Memory (LSTM)

You can think of the LSTM equations visually like this:

Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

19 of 33

Long Short-Term Memory (LSTM)

^ct-1

^ht-1

h_t

i_t

o_t

c_t

_c~

You can think of the LSTM equations visually like this:

Compute the forget gate

Forget some cell content

Compute the input gate

Compute the new cell content

Compute the output gate

Write some new cell content

Output some cell content to the hidden state

The + sign is the secret!

Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

20 of 33

How does LSTM solve vanishing gradients?

The LSTM architecture makes it easier for the RNN to preserve information over many timesteps

e.g., if the forget gate is set to 1 for a cell dimension and the input gate set to 0, then the information of that cell is preserved indefinitely.
In contrast, it’s harder for a vanilla RNN to learn a recurrent weight matrix W_hthat preserves info in the hidden state
In practice, you get about 100 timesteps rather than about 7

LSTM doesn’t guarantee that there is no vanishing/exploding gradient, but it does provide an easier way for the model to learn long-distance dependencies

21 of 33

LSTMs: real-world success

In 2013–2015, LSTMs started achieving state-of-the-art results

Successful tasks include handwriting recognition, speech recognition, machine translation, parsing, and image captioning, as well as language models
LSTMs became the dominant approach for most NLP tasks

Post (2018) and Now, other approaches (e.g., Transformers) have become dominant for many tasks

For example, in WMT (a Machine Translation conference + competition):
In WMT 2016, the summary report contains “RNN” 44 times
In WMT 2019: “RNN” 7 times, ”Transformer” 105 times

Source: "Findings of the 2016 Conference on Machine Translation (WMT16)", Bojar et al. 2016, http://www.statmt.org/wmt16/pdf/W16-2301.pdf Source: "Findings of the 2018 Conference on Machine Translation (WMT18)", Bojar et al. 2018, http://www.statmt.org/wmt18/pdf/WMT028.pdf Source: "Findings of the 2019Conference on Machine Translation (WMT19)", Barrault et al. 2019, http://www.statmt.org/wmt18/pdf/WMT028.pdf

22 of 33

Another RNN architecture to address vanishing gradients: GRU (Gated Recurrent Unit)

GRUs are simplified versions of LSTMs

LSTM - 3 gates - input gate, output gate, forget gate

GRU - 2 gates - update gate and reset gate. The update gate controls how much of the previous hidden state should be retained, and the reset gate determines how much of the past information to forget.

LSTM has two memories (hidden state and cell state) while GRU has just one memory (hidden state). GRU has fewer parameters, can train faster. GRU and LST are comparable performance-wise.

Your task: learn the details of GRU

23 of 33

We have now got a better representation of words than Word embeddings: Contextual or context-sensitive representations

The boat was tied on the river bank …

I deposited the account cheque in the bank …

Will the RNN / LSTM representation of the word “bank” be the same in the two sentences?

24 of 33

We have now got a better representation of words than Word embeddings: Contextual or context-sensitive representations

RNN

Word

embedding

an ant went to

The boat was tied

the river

on the

bank

river

bank …

I deposited the account cheque in the bank …

Will the RNN / LSTM representation of the word “bank” be the same in the two sentences?

25 of 33

5. Bidirectional and Multi-layer RNNs: motivation

terribly

exciting

the

movie was

positive

Sentence encoding

elemen^{t-wise mean/max}

^{element-wise mean}/max

We can regard this hidden state as a representation of the word “terribly” in the context of this sentence. We call this a contextual representation.

These contextual representations only contain information about the left context (e.g. “the movie was”).

What about right context?

In this example, “exciting” is in the right context and this modifies the meaning of “terribly” (from negative to positive)

Task: Sentiment Classification

26 of 33

Bidirectional RNNs

was ^terribly

exciting

the

movie

Forward RNN

A separate RNN, with its own parameters, learning a backward representation of the sentence

Backward RNN

Concatenated hidden states

This contextual representation of “terribly” has both left and right context!

27 of 33

Bidirectional RNNs

Forward RNN

Backward RNN Concatenated hidden states

This is a general notation to mean “compute one forward step of the RNN” – it could be a vanilla, LSTM or GRU computation.

We regard this as “the hidden state” of a bidirectional RNN. This is what we pass on to the next parts of the network.

Generally, these two RNNs have separate weights

On timestep t:

28 of 33

Bidirectional RNNs: simplified diagram

terribly

exciting

the

movie was

The two-way arrows indicate bidirectionality and the depicted hidden states are assumed to be the concatenated forwards+backwards states

29 of 33

Bidirectional RNNs

Note: bidirectional RNNs are only applicable if you have access to the entire input sequence

They are not applicable to Language Modeling, because in LM you only have left context available.

If you do have entire input sequence (e.g., any kind of encoding), bidirectionality is powerful (you should use it by default).

For example, BERT (Bidirectional Encoder Representations from Transformers) is a powerful pretrained contextual representation system built on bidirectionality.

You will learn more about transformers include BERT in a couple of weeks!

30 of 33

Multi-layer RNNs

RNNs are already “deep” on one dimension (they unroll over many timesteps)

We can also make them “deep” in another dimension by applying multiple RNNs – this is a multi-layer RNN.

This allows the network to compute more complex representations

The lower RNNs should compute lower-level features and the higher RNNs should

compute higher-level features.

Multi-layer RNNs are also called stacked RNNs.

Examples:

Lower-level features - word properties, such as POS or whether the word is part of a named entiry

Higher-level features - about the overall structure of a sentence / phrase, e.g., the meaning, associated sentiment, etc.

31 of 33

Multi-layer RNNs

was ^terribly

exciting

the

movie

RNN layer 1

RNN layer 2

RNN layer 3

The hidden states from RNN layer i are the inputs to RNN layer i+1

32 of 33

There is only one unit at each layer, i.e., only one set of parameter matrices at each layer.

The images are shown unfolded / unrolled over time.

Multilayer RNN

h”

•••

33 of 33

Multi-layer RNNs in practice

For example: In a 2017 paper, Britz et al find that for Neural Machine Translation, 2 to 4 layers is best for the encoder RNN, and 4 layers is best for the decoder RNN

Usually, skip-connections/dense-connections are needed to train deeper RNNs (e.g., 8 layers)

Transformer-based networks (e.g., BERT) are usually deeper, like 12 or 24 layers.

You will learn about Transformers later; they have a lot of skipping-like connections

High-performing RNNs are often multi-layer (but aren’t as deep as convolutional or

feed-forward networks)

“Massive Exploration of Neural Machine Translation Architecutres”, Britz et al, 2017. https://arxiv.org/pdf/1703.03906.pdf

Multi-layer or stacked RNNs allow a network

to compute more complex representations of text.