1 of 59

CS 162: Natural Language Processing

Saadia Gabriel

Lecture 7:

Language Modeling Part 4

2 of 59

Announcements

  • If you don’t have a group by the interest form deadline, submit the form anyway with “n/a” for members 2-4 and I’ll group people with similar interests together

3 of 59

Quiz #2

https://forms.gle/PCgrGS1jizC9LELQ6

4 of 59

Demo Recap

Part 1: Build a bigram model

5 of 59

Demo Recap

Part 1: Build a bigram model

(1) What’s the probability of “I am Sam”?

(2) What’s the probability of “Sam am I do”?

.111

0

6 of 59

Demo Recap

Part 2: Shakespeare model

7 of 59

Demo Recap

Part 3: Curse of zero probabilities

8 of 59

Demo Recap

Part 3: Curse of zero probabilities

9 of 59

Demo Recap

Part 4: Perplexity

10 of 59

Demo Recap

Full demo solution is shared here:

https://drive.google.com/file/d/1ZN6q1T5cI733XCAyySdImRjCv-TpQ0ly/view?usp=sharing

11 of 59

Q&A

No, because the inputs change! A decoder RNN takes the previous step’s output and the previous hidden state as the new inputs.

Sometimes they are still used, e.g. RNNs for long sequences. But transformers are standard in many cases.

12 of 59

Q&A

One of today’s topics: the transformer!

We wouldn’t normally see a feed-forward NN on its own, but they are still used today as layers within transformer models!

13 of 59

Last time:

We started discussing RNNs

Today we talk about adding attention mechanisms, these are central to transformers!

A specific kind of RNN design, the long-short term memory model or LSTM

Example of RNN classifier (courtesy of MDPI)

14 of 59

Today’s Goals (according to Gemini):

15 of 59

Today’s Goals:

  • Introduce attention mechanisms
  • Discuss: is attention all you need?
  • Introduce transformers

16 of 59

RNN Recap

17 of 59

The 3 Gates in a LSTM:

3. Output gate

2. Forget gate

1. Input gate

18 of 59

Encoder-Decoder Architectures

Example from Dive into Deep Learning

Encoder calculations

Use final hidden state from encoder to initialize decoder hidden states

Decoder calculations

19 of 59

Vanishing Gradients are still a problem…

20 of 59

Enter the Attention Mechanism

21 of 59

Slide courtesy of Mohit Iyyer

22 of 59

Seq2Seq with Attention

23 of 59

Basics of Attention

Slide courtesy of Mohit Iyyer

In practice, we scale attention weights to stabilize gradients during training

24 of 59

Basics of Attention

Slide courtesy of Mohit Iyyer

25 of 59

Basics of Attention

Slide courtesy of Mohit Iyyer

26 of 59

t=2

Basics of Attention

Slide courtesy of Mohit Iyyer

Keep in mind, these attention weights are specific to each decoder timestep.

Attention distribution from t=1

27 of 59

Attention Mechanism

Attention scores calculated as a function of the current decoder hidden state and encoder state for a specific input position.

e.g scaled dot product

28 of 59

Attention Summary

Slide courtesy of Mohit Iyyer

29 of 59

Advanced NLP topics:

Is attention explanation?

*** Discuss ***

30 of 59

Advanced NLP topics:

Is attention explanation?

31 of 59

XAI Background

Image from MathWorks

Attention weights can be interpreted as a local explanation method

Many types of explanation methods have been proposed, including text-based and feature-based.

32 of 59

  • Attention weights can correlate poorly with other explanation methods like gradient-based explanations (also it seems like removing highly attributed features using gradient-based methods can be more harmful)

​

  • Shuffling attention weights often doesn’t hurt prediction performance (explanations are therefore neither faithful nor unique)

​

Limitations of Attention

  • Attention is a type of local explanation method for a model’s behavior, but…

33 of 59

But…is attention all you need to model language?

34 of 59

Transformers

35 of 59

Transformers

36 of 59

Transformers

37 of 59

Structures of each 

encoder and decoder

What is the difference between these two?

38 of 59

The input to the first encoder is word embeddings

The subsequent encoders get the output of the previous encoder

2 linear transformations,

ReLU activation

New terminology!

39 of 59

Self-attention

Self-attention allows it to look at other positions in the input sequence for clues that can help lead to a better encoding for this word.

https://nlp.seas.harvard.edu/2018/04/03/attention.html

This is how we can learn locally contextual embeddings like shown in homework 1.

40 of 59

Self-attention

Step 1: create three vectors

They are the query, key and value vectors. 

​

41 of 59

Self-attention

Step 2: calculate the attention

  • Attention: how much focus to place on other parts of the input sentence
  • The first word will “query” each other word and based on their keys to decide its attention on the words.

42 of 59

Self-attention

43 of 59

Step 5: Multiply each value vector by the attention score

  • Focus on the words we want to attend on (high attention weights), and ignore irrelevant words (low attention weights)

Step 6: sum up the weighted value vectors

Self-attention

44 of 59

Self-attention

Packing embeddings into matrix

  • Each row in the X matrix corresponds to a word in the input sentence

45 of 59

46 of 59

Self-attention

47 of 59

Self-attention Summary

48 of 59

We got rid of recurrence.

How do we capture positional information?

*** Discuss ***

49 of 59

A few more details…

In RNNs, we captured positional information with recurrence. In the absence of recurrence, we using positional embeddings.

Positional Encoding�t is position, i is dimension index, d is the vector size

50 of 59

Byte Pair Encoding

Slide courtesy of Mohit Iyyer

51 of 59

Byte Pair Encoding

Slide courtesy of Mohit Iyyer

52 of 59

Byte Pair Encoding

53 of 59

Byte Pair Encoding

54 of 59

Byte Pair Encoding

55 of 59

Byte Pair Encoding

56 of 59

There are even more details, but we’ll discuss next time.

57 of 59

BertViz (Vig, 2019)

https://colab.research.google.com/drive/1hXIQ77A4TYS4y3UthWF-Ci7V7vVUoxmQ?usp=sharing

58 of 59

Take-aways

59 of 59

Next Time…

Pretrained Transformers