1 of 87

1

EC 500 A1: �Vision Transformers

2 of 87

Homework assignment & midterm

Homework – 3

Released (10/14)
Due on 10/26 at midnight

Midterm

10/28 in class

3 of 87

Final project first glance (subject to change)

Pre-defined tasks:

CVPR 2026 competition: https://cvpr.thecvf.com/Conferences/2026/Workshops

Keywords: competition & challenge
Example: https://sites.google.com/view/fgvc13/competitions

Improvement over an existing paper
Benchmarking existing algorithms
Instructor-provided tasks

Self-defined “research” tasks:

Need approval
Need justification (not your lab’s work)

4 of 87

Final project first glance (subject to change)

Team forming:

2 – 3 students: same expectations
MUST be a team project

Steps:

Team forming this week
Project sketch: 1 page at most: what you plan to do, who your teammates are [10/26]
Project proposal: 2 – 3 pages [11/2]
Project milestone report: : 2 – 3 pages [11/23]
Project report & code release: academic paper format (e.g., NeurIPS); LaTeX is required [12/10]
Project presentation: 7 – 10 minutes [12/16]

5 of 87

5

Questions?

6 of 87

Convolutional neural networks (CNN)

6

Vectorization + FC layers

7 of 87

Data structure (in training)

Image classification problem

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

8 of 87

Today

CNN revisit and extension
Transformer

Concept
Token
Attention
Vision transformer

9 of 87

Rethink about neural networks for images

9

Dog

Bird

Neural Network

(encoder)

Feature vector

……

Has four legs

Has wing

……

1

0

……

0

1

10 of 87

What is the final fully connected (FC) layer doing?

[Koh et al., Concept Bottleneck Models, 2020]

11 of 87

Illustration

red

blue

Long beak

Long leg

Cardinal

Flamingo

Blue Jay

Image

CNN

Yes: 1; No: -1

12 of 87

A simple CNN classifier (assuming gray images)

12

13 of 87

A simple CNN classifier (assuming gray images)

13

14 of 87

A simple CNN classifier (assuming gray images)

14

15 of 87

A simple CNN classifier (assuming gray images)

15

16 of 87

A simple CNN classifier (assuming gray images)

16

17 of 87

A simple CNN classifier (assuming gray images)

17

18 of 87

Rethink about neural networks for images

18

Dog

Bird

Neural Network

(encoder)

Feature vector

……

Has four legs

Has wing

……

1

0

……

0

1

19 of 87

A simple CNN classifier (assuming gray images)

19

20 of 87

Rethink about neural networks for images

20

Dog

Bird

Neural Network

(encoder)

……

1

0

……

0

1

Feature vector

……

21 of 87

21

Questions?

22 of 87

Popular CNN architectures

Encoder, decoder

22

23 of 87

Popular CNN architectures

Encoder + decoder for segmentation

23

24 of 87

Popular CNN architectures

U-Net: Encoder + decoder + skip links

24

25 of 87

25

Questions?

26 of 87

Class Activation Map (CAM)

26

[Zhou et al., Learning Deep Features for Discriminative Localization, CVPR 2016]

……

27 of 87

CAM for fine-grained tasks

CAM [Zhou et al., 2016] is not localized enough …

27

28 of 87

CAM for fine-grained tasks

28

Why Blue Grosbeak?

Why Grandala?

Target: Blue Grosbeak

Grad-CAM

29 of 87

Why cannot Grad-CAM see the fine difference?

29

0.4

0.3

logits

linear

classifier

…

feature

maps

encoder

blocks

30 of 87

Why cannot Grad-CAM see the fine difference?

30

0.4

0.3

logits

…

feature

maps

Finer-CAM

linear

classifier

encoder

blocks

31 of 87

Finer-CAM: Spotting the difference [CVPR 25]

31

Why Blue Grosbeak?

Why Grandala?

What’s the difference from Grandala?

Target: Blue Grosbeak

Grad-CAM

Finer-CAM

Target: Blue Grosbeak

Reference: Grandala

Finer-CAM: Spotting the Difference Reveals Finer Details for Visual Explanation, CVPR 2025

32 of 87

Finer-CAM: Spotting the difference [CVPR 25]

None

2^nd prediction

3^rd prediction

4^th prediction

Finer-CAM: Spotting the Difference Reveals Finer Details for Visual Explanation, CVPR 2025

33 of 87

Today

CNN revisit and extension
Transformer

Concept
Token
Attention
Vision transformer

34 of 87

How do I know it is a zebra?

35 of 87

Convolutions

35

Linear receptive field

36 of 87

Convolutions

36

Linear receptive field

Exponential receptive field

(with pooling + down-sampling)

37 of 87

CNN vs. Vision transformer: different aggregations

37

CNN

Convolution

Vision transformer

Transformer

neuron

38 of 87

Vision Transformer

38

Image (pixels)

neuron

39 of 87

Vision Transformer

39

(2) Vectorize each of them

+ encode each with a shared MLP

+ “spatial” encoding

(1) Split an image into patches

[Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021]

40 of 87

Revisit: data structure of images

Ordered array (matrix) of pixel values:

A set of pixels and their locations:

Make geometry explicit!

40

Regular spatial grid

41 of 87

Vision transformer

41

(2) Vectorize each of them

+ encode each with a shared MLP

+ “spatial” encoding

(1) Split an image into patches

1-layer of Transformer Encoder

[Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021]

42 of 87

1-layer of transformer encoder

42

K

Q

V

key, query, value

“learnable” matrices

Relatedness of patch-5 to others (after softmax)

Weighted value vectors

Single-head case

43 of 87

Pixels or patches relationship

Transformers

Convolutional neural nets (CNNs)

44 of 87

CNN vs. Vision Transformer

44

CNN

Convolution

Vision transformer

Transformer

45 of 87

45

Questions?

46 of 87

CNN vs. Vision Transformer

46

CNN

Convolution

Vision transformer

Transformer

47 of 87

Key difference

Replace “convolutional” layer by “attention” layers

48 of 87

Limitation of CNN

Assumption: locality (filters with small kernel sizes)

Challenge: getting global/holistic information of an image is hard.

Using size 3 kernel

How about Fully connected layers?

49 of 87

Attention

“Selectively” combine/compare neurons (features at each spatial location)!

50 of 87

Vision Transformer

50

(2) Vectorize each of them

+ encode each with a shared MLP

+ “spatial” encoding

(1) Split an image into patches

1-layer of Transformer Encoder

[Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021]

51 of 87

1-layer of transformer encoder

51

K

Q

V

key, query, value

“learnable” matrices

Relatedness of patch-5 to others (after softmax)

Weighted value vectors

Single-head case

52 of 87

Today

CNN revisit and extension
Transformer

Concept
Token
Attention
Vision transformer

53 of 87

Data type in transformers: tokens

Each token “encapsulates” group of information

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

54 of 87

Comparison: CNN vs. transformers

CNN

Transformer

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

55 of 87

Operations on tokens: overall

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

56 of 87

Operations on tokens: linear combination

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

57 of 87

Operations on tokens: parallel nonlinearity

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

58 of 87

Today

Transformer

Recap
Token
Attention
Vision transformer

Semantic segmentation

59 of 87

The attention layer

The combination weights are not fixed

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

60 of 87

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

61 of 87

How to obtain A?

An “explicit” query guided attention

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

62 of 87

How to obtain A?

An “explicit” query guided attention

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Each token will generate a key vector k

63 of 87

How to obtain A?

An “explicit” query guided attention

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Softmax:

64 of 87

What if there is no explicit question? Self-attention

Each token generates its own query (question), key, and value, by using the “shared learnable” W matrices

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

65 of 87

Self-attention expanded

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

66 of 87

Self-attention expanded

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Attention (contribution) matrix:

How does token j contribute to token?

67 of 87

Illustration

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

68 of 87

Illustration

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

69 of 87

Illustration

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

70 of 87

Illustration

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

71 of 87

Illustration

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

72 of 87

72

Questions?

73 of 87

MLP vs. (multi-layer) transformer

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

74 of 87

Properties: permutation invariant

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

75 of 87

Variants: masked attention

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

I go to school

76 of 87

Today

CNN revisit and extension
Transformer

Concept
Token
Attention
Vision transformer

77 of 87

Vision Transformer

77

(2) Vectorize each of them

+ encode each with a shared MLP

+ “spatial” encoding

(1) Split an image into patches

1-layer of Transformer Encoder

[Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021]

78 of 87

Vision Transformer

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Skip connection

Multi-head

Self-attention

Check textbook!

79 of 87

Position encoding

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

(x, y)

80 of 87

Position encoding

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

(x, y)

81 of 87

Position encoding

[https://erdem.pl/2021/05/understanding-positional-encoding-in-transformers]

82 of 87

82

Questions?

83 of 87

How to do classification?

83

(2) Vectorize each of them

+ encode each with a shared MLP

+ “spatial” encoding

(1) Split an image into patches

1-layer of Transformer Encoder

[Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021]

84 of 87

Adding a classification token

[Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021]

84

Often called a [CLS] token which is learnable

85 of 87

1-layer of transformer encoder

K

Q

V

key, query, value

“learnable” matrices

Single-head case

85

86 of 87

1-layer of transformer encoder

K

Q

V

key, query, value

“learnable” matrices

Multi-head case

86

K

Q

V

87 of 87

Short summary

A general architecture of CNN or visual transformers involves

Multiple layers of computations + nonlinearity + (pooling + striding)
These result in a (final) feature map
The map goes through FC layers (MLP)
Usually, we keep the network till the feature map

For feature extraction
For down-stream tasks
For image-to-image search

87