1 of 48

Transformer

Prof. Seungchul Lee

1

2 of 48

From RNN to Attention

2

3 of 48

From Sequential Modeling to Context Modeling

Is sequential modeling enough for language?

How can a model connect distant but related words?

3

4 of 48

What We Already Know

State (or latent state) summarizes information about the entire past, enabling the system to maintain memory of the past without explicitly storing all previous outputs

4

5 of 48

Language Is Not Just a Time Series

Sentence is not just a simple sequence, but a network of relationships between words

Sentence is read from left to right
Meaning is not always determined only by neighboring words
Therefore, language models must learn not only order, but also relationships.

Question

Does “he” refer to the researcher or the assistant?
Why is “satisfied” important?

�

5

The researcher asked the assistant to run the experiment again because he was not satisfied with the results.

6 of 48

Compress the Past into a State

RNNs and LSTMs read a sentence sequentially and accumulate past information into a single hidden state
However, a bottleneck occurs because all information must be compressed into one vector

Earlier words gradually fade in longer sentences

�

6

I

go

to

school

LSTM

FC

나는

FC

학교에

FC

간다

<s>

나는

LSTM

학교에

LSTM

간다

LSTM

<eos>

FC

LSTM

Embedding

7 of 48

Compress the Past into a State

RNNs and LSTMs read a sentence sequentially and accumulate past information into a single hidden state
However, a bottleneck occurs because all information must be compressed into one vector

Earlier words gradually fade in longer sentences

�

7

LSTM

I

LSTM

go

LSTM

to

LSTM

school

LSTM

Context�Vector

FC

나는

FC

학교에

FC

간다

<s>

나는

LSTM

학교에

LSTM

간다

LSTM

<eos>

FC

Bottleneck

Embedding

8 of 48

Compress the Past into a State

RNNs and LSTMs read a sentence sequentially and accumulate past information into a single hidden state
Important earlier words are hard to access directly�

8

LSTM

I

LSTM

go

LSTM

to

LSTM

school

LSTM

Context�Vector

FC

나는

FC

학교에

FC

간다

<s>

나는

LSTM

학교에

LSTM

간다

LSTM

<eos>

FC

Bottleneck

Embedding

I

go

to

school

나는

학교에

간다

English

Korean

9 of 48

Improving Sequential Learning

Sequential processing

Slow, no parallel processing

Vanishing gradient problem

Not able to incorporate long term dependencies

Better way?

Identify and attend to what is important

9

Attention!

10 of 48

Attention

10

11 of 48

Two Different Sentences

11

“I swam across the river to get to the other bank.”

“I walked across the road to get cash from the bank.”

12 of 48

Key Concepts in Attention (1)

Some words are more important than others in interpreting ‘bank’ correctly.

12

13 of 48

Key Concepts in Attention (2)

Particular locations that should receive more attention depend on the input sequence itself

Once the network is trained, the weights for their associated inputs are generally fixed
By contrast, attention uses weighting factors whose values depend on the specific input data

13

“I swam across the river to get to the other bank.”

“I walked across the road to get cash from the bank.”

14 of 48

Attention as Weighted Aggregation

14

15 of 48

Analogy: Attention and Class Activation Maps (CAM)

Class activation map is

15

Class Activation Map

16 of 48

Attention Mechanism (?)

To compute attention, each input vector is transformed into three parts

Query (Q): What this timestep is looking for
Key (K): What each other timestep offers (the index)
Value (V): The information each timestep holds (the content)

16

Query

(question)

Key

(index)

Value

(content)

He

opened

the

door

and

smiled

He

opened

the

door

and

smiled

17 of 48

The Dictionary Analogy

In Python, a dictionary maps keys to values and supports exact lookup:

This is a hard lookup

the query must match a key exactly, and exactly one value is returned

17

18 of 48

The Dictionary Analogy

In Python, a dictionary maps keys to values and supports exact lookup:

This is a hard lookup

the query must match a key exactly, and exactly one value is returned

18

key

19 of 48

The Dictionary Analogy

In Python, a dictionary maps keys to values and supports exact lookup:

This is a hard lookup

the query must match a key exactly, and exactly one value is returned

19

key

value

20 of 48

The Dictionary Analogy

In Python, a dictionary maps keys to values and supports exact lookup:

This is a hard lookup

the query must match a key exactly, and exactly one value is returned

20

key

value

query

21 of 48

The Dictionary Analogy

Attention generalizes this to a soft lookup
Instead of requiring an exact match, every key is compared against the query by a similarity score, and the result is a weighted sum over all values:

No entry is completely excluded
Every value contributes in proportion to how well its key matches the query
The model does not commit to a single answer; it returns a blend, weighted by relevance

21

22 of 48

The Dictionary Analogy

Suppose the query is “fast and strongly typed language”
No key in the dictionary matches this string exactly
But intuitively, “c++” and “java” are more relevant than “python” or “Haskell”
A soft lookup would return a weighted combination of all values, where the weights reflect how well each key matches the query

22

23 of 48

Word Tokenization

Token

The minimum input unit an LLM processes
The model reads, predicts, and generates in units of tokens

Embedding

The process of converting each token into a high-dimensional real-valued vector (or the vector itself)

23

power

of

industrial

AI

power of industrial AI

0.1	1.2		0.3

0.8	0.2		0.1

0.3	1.8		1.3

0.5	0.7		0.2

24 of 48

Word Embedding

Embedding

The process of converting each token into a high-dimensional real-valued vector

24

https://arize.com/blog-course/embeddings-meaning-examples-and-how-to-compute/

The

cat

sat

Material type

Original text

Embedding

0.1	1.2	0.4	0.3

0.3	0.6	0.2	0.8

0.9	0.7	0.5	0.2

“Steel”

“Aluminum”

“Rubber”

0.6	0.9	0.0	0.1

0.5	0.4	0.3	0.9

0.2	0.7	0.8	0.7

25 of 48

Word Embedding

In this vector space

words with similar meanings cluster closer together, while unrelated words sit farther apart

25

https://arize.com/blog-course/embeddings-meaning-examples-and-how-to-compute/

26 of 48

Encoding Positions

The attention mechanism is that it treats its inputs as a set
It does not inherently account for the order or relative position of elements in a sequence

Question

How to explicitly inject position information into the input representations?

26

“the dog bit the man”

“the man bit the dog”

1

2

3

5

4

1

2

3

5

4

27 of 48

Encoding Positions (Intuitive Insights)

Binary Position Encoding

27

28 of 48

Encoding Positions (Intuitive Insights)

Binary Position Encoding

Continuous (Sinusoidal) Positioning Encoding

28

29 of 48

Positional Embedding

Position vector is added to input embedding at each step

Nearly orthogonal

Allows model to jointly process ‘what’ happened and ‘when’

29

The

cat

sat

0.1	1.2	0.4	0.3
0.3	0.6	0.2	0.8
0.9	0.7	0.5	0.2

Input embedding

Encoded position

Final input

+

=

Element-wise addition

0.0	1.0	1.0
0.8	0.5	0.9
0.9	-0.4	0.9

0.1	2.2	0.4	1.3
1.1	1.1	0.2	1.7
1.8	0.3	0.5	1.1

30 of 48

How to Compute Similarity Between Two Vectors

30

31 of 48

How Attention is Computed (w/ Single Query)

Given Query, Key, Values

Compute attention scores

Attention is a weighted attention score sum over all values

31

Softmax

Y

Query

Key

Value

Compute attention scores

Extract features with high attention

32 of 48

How Attention is Computed

Given Query, Key, Values

Compute attention scores

Attention is a weighted attention score sum over all values

32

Softmax

Query

Key

Value

Compute attention scores

Extract features with high attention

Y

33 of 48

From Encoded Inputs to Attention

Project embedded token input into query, key, and value spaces

Learned projection matrices

33

Encoded Input

0.1	2.2	0.4	1.3
1.1	1.1	0.2	1.7
1.8	0.3	0.5	1.1

Query

Key

Value

Learnable parts

34 of 48

Learning Self-Attention with Neural Networks

A self-attention head that can plug into a larger network
Foundational building block of the Transformer architecture

34

Mat mul

Scaling

Softmax

Mat mul

35 of 48

Transformer

35

36 of 48

Transformer

A subset of deep neural network architecture developed by Google

Based solely on attention mechanisms

No recurrent operations

36

37 of 48

Multi-Head Attention

The attention mechanism so far is called a single head
Transformers use multiple heads in parallel

37

38 of 48

Multi-Head Attention

To capture multiple patterns of attention that are relevant at the same time
Similar to how CNNs use multiple channels

38

Mat mul

Scaling

Softmax

Mat mul

Self-attention

Concatenate

Linear

39 of 48

Transformer Layers

Residual connections that bypass the multi-head structure
Layer normalization to improve training efficiency

39

Multi-head

self-attention

Add & norm

MLP

Add & norm

40 of 48

Why Are Transformers Effective for Sequential Learning?

Capable of modeling long-range dependencies

40

Local receptive field

Sequential processing

RNN

CNN

Transformer

Global attention

41 of 48

Encoder and Decoder Transformers

Encoder-decoder models

Cross attention

Query (Q) from decoder layer, Key (K) & Value (V) from encoder output

e.g. BART

Encoder only models

Produce embeddings from input data
Analyzing text, not generating text
e.g. BERT

Decoder only models

Autoregressively generates text with prompts
Careful with attention: Causal attention
e.g. ChatGPT

41

42 of 48

Encoder-decoder Transformers

42

Cross-attention transformer layer

Linear+Softmax

…

Self-attention transformer layer

…

Encoder

Decoder

43 of 48

Note: Encoder-decoder Structure for Natural Language

Sequence-to-sequence (Seq2Seq) Transformer

Example: language translation

43

44 of 48

Encoder Transformers

Given input sequences, produce fixed-length vectors or class labels
A randomly chosen subset of the data, e.g. 15%, are replaced with <mask>
The model is trained to predict the missing ones at the output nodes.

44

“I <mask> across the river to get to the <mask> bank.”

Transformer layer

<class>

Linear+

Softmax

Linear+

Softmax

Linear+

Softmax

…

45 of 48

Decoder Transformers

45

-inf	-inf	-inf	-inf
	-inf	-inf	-inf
		-inf	-inf
			-inf

I

swam

across

the

river

I

swam

across

the

<start>

Inputs

Outputs

Masked transformer layer

<start>

Linear+

Softmax

Linear+

Softmax

Linear+

Softmax

…

“I <mask> across the river to get to the other bank.”

“I swam <mask> the river to get to the other bank.”

“I swam across <mask> river to get to the other bank.”

…

46 of 48

Transformer Summary

Core concept of the transformer model is self-attention mechanism

Which allows the model to focus on relevant parts of the input sequence
Attention scores using query (Q), key (K), and value (V) vectors
Multi-head attention to focus on different parts of the input simultaneously

Transformers can be trained on large datasets to learn general-purpose representations.

Encoder-decoder model
Encoder only model
Decoder only model

46

47 of 48

Vision Transformers (ViT)

Need to decide how to convert an input image into tokens
the simplest choice is to use patches as a token

47

48 of 48

Lab: Transformer Implementation in Python