ME 5990: Introduction to Machine Learning

�

Attention Model Discussion

Slide Based on:

Lena Voita: https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html

​

​

Outline

• Seq2Seq encoder-decoder
• Attention

Universal Grammar

• Noam Chomsky
• The basic postulate of UG is that there are innate constraints on what the grammar of a possible human language could be.

​

​

Noam Chomsky in 2017�from Wikipedia

Fun fact: legend says Prof Chomsky is the most cited living individual for publications between 1980-1992�(Totaled 3874 citations)

Review: Autoencoder

• Encoder: transform the input (human-interpretable) information into a “code”. The code is a highly-abstracted that contains the input information.
• Decoder: regenerate the human-interpretable information from the code. A good encoder-decoder model will let the regenerated information be similar to the input

​

https://blog.roboflow.com/what-is-an-autoencoder-computer-vision/

​

Review: recurrent network

• For simple RNN, there are two transitions:

​

Review: recurrent network

• A recurrent network is efficient for sequential information

Seq-2-seq

• The encoding and decoding
• For instance, we want to train translation: input ABC -> output WXYZ

Target: W X Y Z <EOS>

Loss

Encoder

Decoder

Seq-2-seq

• We can use a similar idea to encode sequential information using recurrent models

Seq-2-seq

• RNN with two languages

Seq-2-seq

• Training the encoder and decoder:

​

Seq-2-seq

• RNN:

​

Gated Recurrent Unit

• The hidden layer is not necessarily to be “fully connected”, it can be a compound of various functions as a “cell”
• The cell is referred as gated recurrent unit
• It can be:

​

Seq-2-seq

• Implementation

https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

• GRU: gated recurrent unit,
• Embedding: a way to encode vocabulary based on input words exist

​

Seq-2-seq

• Decoder

Attention

• Problem of seq-2-seq:
• We assume at the end of each sentence, the hidden vector carries a meaning of the sentence, regardless of the language

​

​

Attention

• The concept

Attention

• The attention output and input

Attention

Attention

Attention

Attention

Attention

Attention

Attention

Attention

Attention

• Recurrent GRU cell can be FC, but also can be LSTM
• C state: long-term memory
• H state: short-term memory

​

C State

H State

Attention

• We can use LSTM for attention NN:
• https://youtu.be/PSs6nxngL6k?si=d0KTyz0mwOXPS5VM

​

​

Summary

• All recurrent model’s long term memory fades away
• Attention give “weights” to the hidden states, so the hidden states appears to “pay attention to” several sequence of outputs
• Apply to sequences where the outputs order may be different from inputs order

​

​