Attention�Self-Attention�Transformer
Few Reminders
Project proposal template [You can add other points if you have]
RNN/LSTM
RNN based seq2seq translation
Sutskever et al, “Sequence to sequence learning with neural networks”, NeurIPS 2014
We are learning Language
Context vector
Initial decoder state
RNN based seq2seq translation
We are learning Language [START]
estamos
RNN based seq2seq translation
estamos Aprendiendo
We are learning Language [START] estamos
RNN based seq2seq translation
We are learning Language [START]
estamos Aprendiendo español [END]
estamos Aprendiendo español
RNN based seq2seq translation
We are learning Language [START]
estamos Aprendiendo español [END]
estamos Aprendiendo español
-How about long-context and dependency on a fixed-size vector?
-Dependency of previous state and Un-Parallel .
Attention
“The animal didn't cross the street because it was too tired”
Decoder to Encoder Attention
We are learning Language
Context vector
Initial decoder state
Bahdanau et al, “Neural machine translation by jointly learning to align and translate”, ICLR 2015
Decoder to Encoder Attention
We are learning Language
Context vector
Initial decoder state
Normalize attention (sum attention = 0)
Bahdanau et al, “Neural machine translation by jointly learning to align and translate”, ICLR 2015
Decoder to Encoder Attention
We are learning Language
Context vector
Initial decoder state
Normalize attention (sum attention = 0)
Bahdanau et al, “Neural machine translation by jointly learning to align and translate”, ICLR 2015
Decoder to Encoder Attention
We are learning Language
Context vector
Initial decoder state
Normalize attention (sum attention = 0)
Bahdanau et al, “Neural machine translation by jointly learning to align and translate”, ICLR 2015
Decoder to Encoder Attention
We are learning Language
Context vector
Initial decoder state
Normalize attention (sum attention = 0)
Bahdanau et al, “Neural machine translation by jointly learning to align and translate”, ICLR 2015
estamos Aprendiendo español [END]
Attention weight
Example: English to French translation
Input: “The agreement on the European Economic Area was signed in August 1992.”
Output: “L’accord sur la zone économique européenne a été signé en août 1992.”
Bahdanau et al, “Neural machine translation by jointly learning to align and translate”, ICLR 2015
Attention weight
Example: English to French translation
Input: “The agreement on the European Economic Area was signed in August 1992.”
Output: “L’accord sur la zone économique européenne a été signé en août 1992.”
Bahdanau et al, “Neural machine translation by jointly learning to align and translate”, ICLR 2015
Decoder to Encoder Attention
We are learning Language
Context vector
Initial decoder state
Normalize attention (sum attention = 0)
Bahdanau et al, “Neural machine translation by jointly learning to align and translate”, ICLR 2015
estamos Aprendiendo español [END]
-How about long-context and dependency on a fixed-size vector?
-Dependency of previous state and Un-Parallel .
Decoder to Encoder Attention
We are learning Language
Context vector
Initial decoder state
Normalize attention (sum attention = 0)
Bahdanau et al, “Neural machine translation by jointly learning to align and translate”, ICLR 2015
estamos Aprendiendo español [END]
-How about long-context and dependency on a fixed-size vector? Fixed
-Dependency of previous state and Un-Parallel .
Self-Attention
Embedding
a3
a2
a1
a4
Self-Attention
Embedding
a3
a2
a1
a4
X4
Keys | Values |
X1 | v1 |
x2 | v2 |
x3 | v3 |
x4 | v4 |
v4
v4
Query
Self-Attention as a look-up analogy
X4
Keys | Values |
X1 | v1 |
x2 | v2 |
x3 | v3 |
x4 | v4 |
v4
X4
Keys | Values |
X1 | v1 |
x2 | v2 |
x3 | v3 |
x4 | v4 |
Query
Query
Weighted sum of values
Value = Attention
Hypothetical example
Attention for word “learning”
Embedding
k
K
K
k
V
V
V
V
Q
Self-Attention
Q, K, and V are three learnable weight matrices
Hypothetical example
Embedding
k
K
K
k
V
V
V
V
Q
Self-Attention
Q, K, and V are three learnable weight matrices
Attention for word “learning”
What is happening in matrix space?
https://jalammar.github.io/illustrated-transformer/
What is happening in matrix space?
https://jalammar.github.io/illustrated-transformer/
Can we use self-attention only for downstream task? ( prediction ..)
Can we use self-attention only for downstream task? ( prediction ..)
Positional Encoding
Position: Binary
Positional Encoding
Position: Binary
Positional Encoding
Hypothetical example with positional encoding
Embedding
k
K
K
k
V
V
V
V
Q
Self-Attention
Q, K, and V are three learnable weight matrices
Attention for word “learning”
+ + + +
Positional vector
Adding FF layer with self-attention
https://medium.com/@umbertofontana/nlp-part-7-self-attention-and-transformers-c770dea1283a
Multi-Head Attention
🡪
Multi-Head Attention
https://jalammar.github.io/illustrated-transformer/
Masking in self-attention ( Machine translation )
Masking in self-attention ( Machine translation )
Embedding
k
K
K
k
V
V
V
V
Q
Self-Attention
+ + + +
Positional vector
estamos Aprendiendo español [END]
Decoder
Transformer
Attention is all you need
Image source: Walmart.com
Transformer
Neuron
Transformer
Transformer
Thinking
Machine
0.31 | 0.14 | 0.93 |
0.14 | 0.88 | 0.98 |
Learning
Language
0.85 | 0.20 | 0.14 |
0.46 | 0.61 | 0.49 |
Batch Normalization
Transformer
Thinking
Machine
0.31 | 0.14 | 0.93 |
0.14 | 0.88 | 0.98 |
Learning
Language
0.85 | 0.20 | 0.14 |
0.46 | 0.61 | 0.49 |
Layer Normalization
Transformer Feed-Forward Layers Are Key-Value Memories
https://arxiv.org/pdf/2012.14913
Transformer Feed-Forward Layers Are Key-Value Memories
https://arxiv.org/pdf/2012.14913
Transformer Decoder
https://towardsdatascience.com/transformers-explained-visually-part-3-multi-head-attention-deep-dive-1c1ff1024853
Transformer training
https://www.youtube.com/watch?v=XowwKOAWYoQ&t=1589s