1 of 64

1

CS 589 Lecture 9

Monday 6:30-9:00

Kidde 228

All zoom links in Canvas

Most slides adapted from Stanford CS224

Seq2seq, Attention, Transformer

photo: https://www.scubedstudios.com/information-retrieval/

2 of 64

Review of language models

Language modeling is the task of predicting what word comes next

Given a sequence of words x1, x2, …, xt, compute the probability distribution for the next word xt+1

where xt+1 can be any word in the vocabulary

2

mary has a little

lamb

goat

bit

book

0.2

0.3

0.05

0.01

3 of 64

Training RNN method 1: Language modeling

Minimizes the cross entropy loss:

3

mary

a

little

...

[0.01,

0.01,

0.85]

...

[0.79,

0.01,

0.10]

...

[0.09,

0.82,

0.10]

...

[0.09,

0.62,

0.10]

...

0.85]

0.62,

[0.79,

0.82,

mary

has

a

little

4 of 64

Training RNN method 2: Text classification

Cross entropy loss (min):

4

output distribution

[0.2,

0.3]

0.5,

neg

pos

neutral

Accuracy (max):

mary

has

a

little

5 of 64

Today’s lecture

Seq2seq

Attention mechanism

Transformer

5

6 of 64

Machine translation

Translating a sentence from one language (source language) to another language (target language)

6

7 of 64

Training NN method 3: Machine Translation

Minimizes the cross entropy loss:

7

man

is

born

free

is

born

free

...

[0.01,

0.01,

0.85]

...

[0.79,

0.01,

0.10]

...

[0.09,

0.82,

0.10]

...

[0.09,

0.10]

...

0.85,

0.62]

[0.79,

0.82,

0.01]

fers

le

dans

(only for the output sentences)

8 of 64

Encoder-decoder framework

… dans les fers <START>

8

9 of 64

Encoder-decoder framework

… dans les fers <START> man

9

man

10 of 64

Encoder-decoder framework

… dans les fers <START> man is

10

man is

11 of 64

Encoder-decoder framework

… dans les fers <START> man is born ….

11

man is born free

hN' encodes the info of input French sentence

encoder RNN

decoder RNN

12 of 64

Training NN task 3: Neural machine translation

Translating a sentence from one language (source language) to another language (target language)

12

train:

predict:

similar training objective as language modeling

13 of 64

Encoder-decoder framework: Other applications

What does SIT stand for?

Stevens Institute of Technology

Open the file file.txt and read it line by line

with open('file.txt', 'r') as fin:

lines = fin.readlines()

Can you show me the ticket from NY to LA?

OK, what are the dates of your flight?

machine translation

factoid question answering

natural language to code

conversational agent

14 of 64

Encoder (Input Network)

… dans les fers

14

LSTM encoder:

Other encoders:

GRU

LSTM/GRU + attention

Elmo

BERT

…

encoder RNN

hN' encodes the info of input French sentence

15 of 64

Decoder (Output network)

<START> man is born ….

15

Decoder selects the word to generate in the target sentence

Decoder framework can be different from encoder framework

Training:

Train by objective on decoder

Prediction:

Predict the output sequence using the decoder

man is born free

decoder RNN

16 of 64

Decoding (Prediction)

<START> man is born ….

16

Greedy decoding

Disadvantage of greedy decoding: cannot undo decisions

Exhaustive search? exponential search space

man is born free

decoder RNN

17 of 64

Beam search decoding

A trade-off between greedy decoding + exhaustive search

In each step of the decoder, keep track of the k most probable partial translations (hypotheses)

17

<START>

he

I

…

18 of 64

Beam search decoding

A trade-off between greedy decoding + exhaustive search

In each step of the decoder, keep track of the k most probable partial translations (hypotheses)

18

<START>

he

I

hit

struck

was

got

19 of 64

Beam search decoding

A trade-off between greedy decoding + exhaustive search

In each step of the decoder, keep track of the k most probable partial translations (hypotheses)

19

<START>

he

I

hit

struck

was

got

20 of 64

Beam search decoding

A trade-off between greedy decoding + exhaustive search

In each step of the decoder, keep track of the k most probable partial translations (hypotheses)

20

<START>

he

I

hit

struck

was

got

21 of 64

Decoding: How do we know when to stop?

When do we stop the generation?

Append an <END> token to every target sentence in the training data
Train the model, so it knows when to predict <END>
During decoding, stop a hypothesis if <END> is predicted

We continue beam search util

We reach time step T, or
We have at least n completed hypotheses

21

22 of 64

Beam search decoding: Selecting output hypothesis

Each hypothesis has a log probability score

Longer hypothesis always have lower scores

Solution: normalize hypotheses by length

22

23 of 64

Machine translation timeline

Statistical machine translation: IBM model 1-5 [Brown et al. 1993]

2014: first seq2seq paper was published

2016: Google Translate switched from SMT to NMT

The SMT system, built by hundreds of engineers over many years, outperformed by the NMT systems trained by a handful of engineers in a few months

23

24 of 64

Evaluating machine translation

How to evaluate the correctness of a translated sentence?

Compare the similarity between the prediction and the ground truth
Capturing the similarity beyond unigram matching

BLEU (Bilingual Evaluation Understudy)

BLEU compares the machine-written translation to one or several human-written translation(s), and computes a similarity score based on:

n-gram precision (usually for 1, 2, 3 and 4-grams)
a penalty for too-short system translations

BLEU is useful but imperfect (non-overlapping ngrams)

24

25 of 64

BLEU score

Step 1: computing the percentage of the i-gram tuples in the hypothesis that also occur in the references

Step 2: averaging over the percentages:

Penalizing short translations:

25

#times t_i occurs in the hypothesis h

#times t_i occurs in the j-th reference for hypothesis h

n: length of hypothesis

L: length of reference

total #tokens in hypothesis h

26 of 64

Today’s lecture

Seq2seq

Attention mechanism

Transformer

26

27 of 64

Attention mechanism

Training the neural network to attend to the most relevant input to the next word to predict

27

?

28 of 64

Attention mechanism

Attention mechanism is designed for each word to focus on one or a just a few words in the source sentences

28

… est dans les <START>

encoder

attention

dot product [Luong et al. 2015];

RNNSearch [Bahdanau et al. 2014]

29 of 64

Attention mechanism: Dot-product attention

Step 1: Getting the attention weight

Using the hidden vector to compute the attention weight

29

attention

weight

encoder

[Luong et al. 2015]

30 of 64

Attention mechanism: Dot-product attention

Step 2: Computing the memory vector ct using the integration of the hidden vector

30

weight

[Luong et al. 2015]

31 of 64

Attention mechanism: Dot-product attention

Step 3: Using the combination of memory vector ct and hidden vector for prediction

[Luong et al. 2015]

Step 1-2

32 of 64

Attention mechanism: Dot-product attention vs RNNSearch

32

For t = 1,…, T:

RNN search attention

For t = 1,…, T:

dot product attention

33 of 64

Attention mechanism: RNNSearch

33

attention

weight

encoder

GRU

[Bahdanau et al. 2014]

34 of 64

Attention mechanism: RNNSearch

34

weight

[Bahdanau et al. 2014]

35 of 64

Attention mechanism: RNNSearch

Step 1-2

GRU

[Bahdanau et al. 2014]

36 of 64

Attention mechanism: RNNSearch

Step 1-2

output

[Bahdanau et al. 2014]

37 of 64

Performance of attention [Luong et al. 2015]

37

38 of 64

Today’s lecture

Seq2seq

Attention mechanism

Transformer

38

39 of 64

Transformer [2017]

Non-recurrent sequence to sequence encoder-decoder model

Encoder and decoder architectures

Multi-head self attention [Lin et al. 2017]

Position encoding [Gehring et al. 2017]

40 of 64

Encoder/decoder in transformer

Encoder: 6 identical layers

First layer: multi-head self attention
Second layer: fully connected feed forward network, followed by layer normalization

Decoder: 6 identical layers

First & second: multi-head self attention
Third layer: fully connected feed forward

41 of 64

Self attention

What do we mean by self attention?

In dot-product attention, target attends to source only
Source shall never attend to target (why?)
In self attention, target can attend to source, source itself can also attend to source, target can also attend to target, so called internal attention

source: https://pretteyandnerdy.wordpress.com/2019/04/26/attention-is-all-you-need/

[Lin et al. 2017]

42 of 64

Self attention

… dans les fers <START> man

42

RNN-style recursive attention:

[Lin et al. 2017]

man

43 of 64

Self attention

source-source attention

source-target attention

target-target attention

+

43

Transformer-style attention (computed at the same time):

tous les hommes naissent égaux

all men are born equal

44 of 64

Self attention

S to S

T to S

T to T

44

The input and output embeddings are copied 3 times as query, key and value

45 of 64

Attention vs. Self Attention

Attention is often applied to RNN, whereas self attention does not require RNN (i.e., attention is all you need)
Attention happens from decoder -> encoder, self attention also looks from encoder to encoder state, and from decoder state to decoder state
Self attention time cost increase quadratically with the sequence length

46 of 64

Position embedding

So far we have not considered the order between words
Word orders are important for encoding sentence semantics
How to solve the problem? Position embedding

For example, with 4-dimensional embedding, dmodel = 6:

47 of 64

KV Cache

48 of 64

Summary

Seq2seq framework for machine translation

Attention mechanism: RNNSearch and dot-product attention

Transformer: self attention, multi-headed attention, position embedding

48

49 of 64

Transformer code explained

Seq2seq framework for machine translation

Attention mechanism: RNNSearch and dot-product attention

Transformer: self attention, multi-headed attention, position embedding

49

50 of 64

Pre-trained word embeddings

In RNN, we randomly initialize the weight, use the back propagation to update this weight

The random initialization may not give us a good starting point

We can choose a better starting point with embedding vectors pertained in a large unlabelled corpus to help the initialization [Mikolov et al. 13]

mary

has

a

little

51 of 64

From pretrained embedding to pretrained encoder

Pretrained embeddings: fixed vectors

Pretrained encoder representation:

Neural network layers which takes an input sentence and encodes it into a layer output to be fed into another network, the encoder representation can be fine tuned
A pretrained LSTM on auto encoder & next word prediction can be used as a starting point to improve the performance of downstream supervised LSTM tasks [Dai & Le 2015]
ELMo [Peters et al. 2018]

52 of 64

ELMo [Peters et al. 2018]

Train an L-layered bidirectional LSTM model, i.e., predicting the next word & the previous word
Collapse all the 2L+1 representations into a single layer

Concatenate the ELMo weights into task specific models

Concatenate with input
Concatenate with output

53 of 64

Competition of pretrained language models

source: https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/

54 of 64

BERT: Bidirectional Encoder Representations

Pre-training BERT

Task #1: Masked LM

54

Mask out k% input words
Predict the masked words given all its context (“cloze” style)
Enables representation to fuse the left and right context

55 of 64

BERT: Bidirectional Encoder Representations

Pre-training BERT

Task #1: Masked LM

55

Problem: pre-training and fine-tuning mismatch

[MASK] will never appear in downstream task training

Solution: not always use [MASK]

80% use [MASK]
10% use a random token
10% use the original token

56 of 64

BERT: Bidirectional Encoder Representations

Pre-training BERT

Task #1: Masked LM
Task #2: Next Sentence Prediction (NSP)

56

Predict whether B is the actual sentence that proceeds A (True / False)
To learn relationship between sentences

57 of 64

BERT: Bidirectional Encoder Representations

Pre-training BERT

Input: unlabeled sentence pair
Training

Masked LM
NSP (label position [CLS])

57

58 of 64

BERT: Bidirectional Encoder Representations

Fine-tuning BERT

58

59 of 64

BERT experimental results

59

60 of 64

Effects of pretraining tasks

Masked LM (compared to left-to-right LM) is very important on some tasks, Next Sentence Prediction is important on other tasks

Left-to-right model does very poorly on word-level task (SQuAD), although this is mitigated by BiLSTM

60

61 of 64

Effects of dimensionality and training time

Masked LM takes slightly longer to converge because we only predict 15% instead of 100%

But absolute results are much better almost immediately

61

62 of 64

Effects of dimensionality and training time

Big models help a lot

Going from 110M -> 340M params helps even on datasets with 3,600 labeled examples

Improvements have not asymptoted

62

63 of 64

BERT: Open source release

One reason for BERT’s success was the open source release

Minimum dependency
Abstracted so people could including a single file to use model
Thorough README
Idiomatic code
Well-documented code
Good support (for the first few months)

63

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")

64 of 64

Homework 4

Using Google colab to train a text classification model

Supports keras, tensor flow and pytorch
Free Tesla K80 GPU

Homework 4: StackOverflow programming language prediction using hugging face’s transformer library

64