1 of 98

1

EC 500 A1: �From Image Processing to �Convolutional Neural Networks (CNN)

2 of 98

Recap: Convolution and Fourier Transform

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

3 of 98

Approximated Laplacian filter

4 of 98

Image pyramid : Gaussian pyramid

Create a sequence of down-sampled images

Down-sample

5 of 98

Residual image

Down-sample

Up-

sample

-

=

6 of 98

Gaussian pyramid

6

7 of 98

Laplacian pyramid

7

8 of 98

Image reconstruction

Up-sample

+

=

Up-sample

+

=

Up-sample

……

9 of 98

Today

Filter bank (chapter 22)
Convolutional neural network (CNN) (chapter 24)

9

10 of 98

Image processing by frequency filtering

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

11 of 98

Example

12 of 98

Can we have filters for specific frequencies?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Fourier domain

13 of 98

Recap: Binomial filters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

14 of 98

Gaussian filters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Frequency [u]

u

15 of 98

Can we have filters for specific frequencies?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Fourier domain

16 of 98

Rethinking frequency modulation (FM)

Frequency modulation (FM)

Music-1

Music-2

Music-3

time

frequency

17 of 98

Can we have filters for specific frequencies?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Fourier domain

Spatial domain

18 of 98

Gabor filters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

19 of 98

Gabor filters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Modulation: move Gaussian to different frequencies

20 of 98

Examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

21 of 98

Examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

22 of 98

What is this? How do you tell?

22

23 of 98

23

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

24 of 98

Today

Filter bank (chapter 22)
Convolutional neural network (CNN) (chapter 24)

24

25 of 98

How to perform classification?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

26 of 98

How to perform classification?

26

Dog

Bird

Neural Network

(encoder)

Feature vector

……

Has four legs

Has wing

……

1

0

……

0

1

27 of 98

Deep neural networks (DNN)

28 of 98

Convolution

A special computation between layers

A current node is not directly affected by “all nodes in the previous layer”
The network “weights” on the edges can be “re-used”

28

29 of 98

Convolution

29

0	0	0	0	1
0	0	0	1	1
0	0	1	1	1
0	1	1	1	1
1	1	1	1	1

0	0	1
0	1	1
1	1	1

Feature map (nodes) at layer t

Feature map at layer t+1

“Filter” weights

(3-by-3)

Inner product

Element-wise multiplication and sum

1

30 of 98

Convolution

30

0	0	0	0	1
0	0	0	1	1
0	0	1	1	1
0	1	1	1	1
1	1	1	1	1

0	0	1
0	1	1
1	1	1

“Filter” weights

(3-by-3)

Inner product

6

Feature map (nodes) at layer t

Feature map at layer t+1

31 of 98

Convolution

31

0	0	0	0	1
0	0	0	1	1
0	0	1	1	1
0	1	1	1	1
1	1	1	1	1

0	0	1
0	1	1
1	1	1

“Filter” weights

(3-by-3)

Inner product

1

Zero-padding: Set the missing values to be 0

Feature map (nodes) at layer t

Feature map at layer t+1

32 of 98

Convolution example

32

0	0	0
1	1	1
0	0	0

1	1	1
0	0	0
1	1	1

33 of 98

Convolution

33

“Filter” weights

(3-by-3)

“Filter” weights

(3-by-3-by-“2”)

0	0	0	0	1
0	0	0	1	1
0	0	1	1	1
0	1	1	1	1
1	1	1	1	1

0	0	1
0	1	1
1	1	1

Inner product

Feature map (nodes) at layer t

Feature map at layer t+1

34 of 98

Convolution

34

“Filter” weights

(3-by-3-by-“2”)

0	0	0	0	1
0	0	0	1	1
0	0	1	1	1
0	1	1	1	1
1	1	1	1	1

0	0	1
0	1	1
1	1	1

Inner product

1	1	1
0	0	0
1	1	1

Feature map (nodes) at layer t

Feature map at layer t+1

One filter for one output “channel” to capture a different “pattern” (e.g., edges, circles, eyes, etc.)

35 of 98

Convolution: properties

Process nearby pixels together
Translation invariant: “local patterns” can show up at different pixel locations
Can process arbitrary-size images

35

Top-left, Top right: has ears

Middle: has eyes

36 of 98

36

Questions?

37 of 98

Convolutional neural networks (CNN)

37

Shared weights

Vectorization + FC layers

Max pooling + down-sampling

Remove redundancy
Translation-tolerant
Enlarge receptive field

38 of 98

Recognition example

How many categories?

How many local patterns?

How many local pattern combinations?

39 of 98

Recognition example

Image

Recognition example

Image

Convolution

Feature map

Convolution

……

49 of 98

Recognition example

Convolution

Feature map

50 of 98

Recognition example

Convolution

Feature map

51 of 98

Recognition example

Convolution

Feature map

Convolution

……

52 of 98

Recognition example

Convolution

Feature map

Convolution

……

53 of 98

Recognition example

Convolution

Feature map

Convolution

……

54 of 98

54

Questions?

55 of 98

Convolutional neural networks (CNN)

55

Shared weights

Vectorization + FC layers

Max pooling + down-sampling

Remove redundancy
Translation-tolerance
Enlarge receptive field

What does a large response at each layer/channel mean?

61 of 98

61

Questions?

62 of 98

Convolutional neural networks (CNN)

62

Shared weights

Vectorization + FC layers

Max pooling + down-sampling

Remove redundancy
Translation-tolerance
Enlarge receptive field

63 of 98

Representative CNN networks

AlexNet

[Krizhevsky et al., 2012]

VGGnet

[Simonyan et al., 2015]

63

A block: parameters/compute
Edge: activation/tensors

64 of 98

Representative CNN networks

GoogleNet [Szegedy et al., 2014]
Inception

65 of 98

Representative CNN networks

ResNet

[He et al, 2016]

DenseNet

[Huang et al, 2017]

65

Advantages:

Optimization
Collect more information

A block: parameters/compute
Edge: activation/tensors

66 of 98

Representative CNN networks

A general architecture involves

Multiple layers of convolutions + ReLU (nonlinearity) + pooling + striding
These result in a (final) feature map

Positions on the map correspond to the image

The map goes through FC layers (MLP)

66

67 of 98

Representative CNN networks

A general architecture involves

Multiple layers of convolutions + ReLU (nonlinearity) + pooling + striding
These result in a (final) feature map

Positions on the map correspond to the image

The map goes through FC layers (MLP)
Usually, we keep the network till the feature map

For feature extraction
For down-stream tasks
For image-to-image search

67

68 of 98

68

Questions?

69 of 98

Convolutional neural networks (CNN)

69

Shared weights

Vectorization + FC layers

Max pooling + down-sampling

Remove redundancy
Translation-tolerance
Enlarge receptive field

70 of 98

Stride (down-sampled) convolution

70

71 of 98

Stride (down-sampled) convolution

71

72 of 98

Dilated convolution: without down-sampling

72

73 of 98

73

Questions?

74 of 98

Convolution with [?] filters

74

75 of 98

75

Questions?

76 of 98

Convolutional neural networks (CNN)

76

Vectorization + FC layers

77 of 98

A simple CNN classifier (assuming gray images)

77

78 of 98

A simple CNN classifier (assuming gray images)

78

79 of 98

A simple CNN classifier (assuming gray images)

79

80 of 98

A simple CNN classifier (assuming gray images)

80

81 of 98

A simple CNN classifier (assuming gray images)

81

82 of 98

A simple CNN classifier (assuming gray images)

82

83 of 98

A simple CNN classifier (assuming gray images)

83

84 of 98

84

Questions?

85 of 98

What if the image size changes?

85

Vectorization + FC layers

86 of 98

Revisit the bird image

86

How to apply?

87 of 98

Convolutional neural networks (CNN) for detection

87

88 of 98

Convolutional neural networks (CNN) for segmentation

88

89 of 98

Spatial output

89

90 of 98

A simple CNN classifier

90

91 of 98

Spatial output

91

92 of 98

92

Questions?

93 of 98

Popular CNN architectures

Encoder, decoder

93

94 of 98

Popular CNN architectures

Encoder + decoder for segmentation

94

95 of 98

Popular CNN architectures

U-Net: Encoder + decoder + skip links

95

96 of 98

96

Questions?

97 of 98

Besides translation, CNNs need to tackle object scale

97

98 of 98

Feature pyramid networks

98

[Feature Pyramid Networks for Object Detection, CVPR 2017]