1 of 95

Convolutional Neural Network

Lecture 16

Introduction to convolutional architecture for image analysis

EECS 189/289, Fall 2026 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

2 of 95

Roadmap

Network Design Considerations for Images
Element of the Convolutional Neural Network
CNN Design
What Do CNNs Learn?

1854077

3 of 95

Network Design Considerations for Images

Network Design Considerations for Images
Element of the Convolutional Neural Network
CNN Design
What Do CNNs Learn?

1854077

4 of 95

Story So Far

NNs are universal function approximators.
NNs can be trained through variations of gradient descent.
Can recognize patterns in data.

1854077

5 of 95

A Problem

Will a NN that recognizes the left image as a cat also recognize the one on the right one as a cat?

1854077

6 of 95

A Problem

Will a NN that recognizes the left image as a cat also recognize the one on the right one as a cat?

A network that recognizes "cat" feature regardless of the location and scale of the target object.

1854077

7 of 95

A Problem

Network must be shift and scale invariant.

1854077

8 of 95

Demo: An MLP on Moved Digits

A fully connected network (784, 256, 10) trained on centered MNIST digits.

Centered test digits: 96.0% correct.

The same digits moved 4 pixels right: 29.7%.

1854077

9 of 95

Scanning Is Referred to as Convolution

Scan for the presence of the cat in the image.
We need to apply a feature extractor in each region of the image.

1854077

10 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

11 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

12 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

13 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

14 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

15 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

16 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

17 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

18 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

19 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

1854077

20 of 95

Scanning Is Referred to as Convolution

We need to apply a feature extractor in each region of the image.

max

Maximum of all the outputs (Equivalent of Boolean OR)

The strength of having the feature anywhere in the image

Same shared weights

1854077

21 of 95

Scanning Is Referred to as Convolution

The entire operation can be viewed as one giant network.

With many subnetworks, one per window.

Restriction: All subnetworks are identical.
The network is shift-invariant!

max

1854077

22 of 95

Regular Network vs. Scanning Network

In a FC-NN, each neuron is connected by a unique weight to every neuron in the previous layer.

Entries in the weight matrix are unique.

In a scanning NN, each neuron is connected to a subset of neurons in the previous layer

The weights matrix is sparse.
The weights matrix is block structured with identical blocks.

max

1854077

23 of 95

Element of the Convolutional Neural Network

Network Design Considerations for Images
Element of the Convolutional Neural Network
CNN Design
What Do CNNs Learn?

1854077

24 of 95

Back to Scanning Network

Patches are referred to as "receptive fields".
The weights form a small two-dimensional grid known as a filter or a kernel.
Assume we unroll the 2D structures into a vector: output from each receptive field is calculated as:

max

1854077

25 of 95

How to Identify a Feature in a Receptive Field?

1854077

26 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

At each position: multiply the window by the kernel entry by entry and add the 9 products.

The same 9 weights are used at every position.

1854077

27 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

Stride = 1

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

0

At each position: multiply the window by the kernel entry by entry and add the 9 products.

The same 9 weights are used at every position.

1854077

28 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

Stride = 1

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

0

1

At each position: multiply the window by the kernel entry by entry and add the 9 products.

The same 9 weights are used at every position.

1854077

29 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

Stride = 1

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

0

1

3

At each position: multiply the window by the kernel entry by entry and add the 9 products.

The same 9 weights are used at every position.

1854077

30 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

Stride = 1

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

0

1

3

2

At each position: multiply the window by the kernel entry by entry and add the 9 products.

The same 9 weights are used at every position.

1854077

31 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

Stride = 1

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

0

1

3

2

1

At each position: multiply the window by the kernel entry by entry and add the 9 products.

The same 9 weights are used at every position.

1854077

32 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

Stride = 1

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

0

1

3

2

1

At each position: multiply the window by the kernel entry by entry and add the 9 products.

The same 9 weights are used at every position.

1854077

33 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

At each position: multiply the window by the kernel entry by entry and add the 9 products.

The same 9 weights are used at every position.

1854077

34 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

1854077

35 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

1854077

36 of 95

If we convolve a kernel of size 5x5 with an image of size 8x8 with the stride of 1, what are the output dimensions?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

1854077

37 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

-1	2	-1
-1	2	-1
-1	2	-1

Vertical line detector

1854077

38 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

-1	2	-1
-1	2	-1
-1	2	-1

Vertical line detector

0

1854077

39 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

-1	2	-1
-1	2	-1
-1	2	-1

Vertical line detector

0

1

1854077

40 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

-1	2	-1
-1	2	-1
-1	2	-1

Vertical line detector

0

1

0

1854077

41 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

-1	2	-1
-1	2	-1
-1	2	-1

Vertical line detector

0

1

0

2

1854077

42 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

-1	2	-1
-1	2	-1
-1	2	-1

Vertical line detector

0

1

0

2

-2

1854077

43 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

-1	2	-1
-1	2	-1
-1	2	-1

Vertical line detector

0

1

0

2

-2

1854077

44 of 95

How to Implement Scanning of Features with Weight-Sharing?

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Horizontal line detector

-1

0

Stride = 1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

0

1

0

2

-2

1

0

3

-3

-2

1

0

3

-3

-1

2

1

-2

-1

2

-1

-2

1

2

1

-2

0

-1	2	-1
-1	2	-1
-1	2	-1

Vertical line detector

Feature Map

1854077

45 of 95

If we convolve 4 kernels of size 5x5 with an image of size 8x8 with the stride of 1, what are the output dimensions?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

1854077

46 of 95

Convolution Example

Each output is bright where the digit contains the kernel's pattern.

Top: horizontal strokes.

Middle: vertical strokes.

Bottom: diagonals.

1854077

47 of 95

1854077

48 of 95

The dimension of features decreases but the depth usually increases.

1854077

49 of 95

What If We Want to Keep the Same Image Dimensions?

We can use padding.

0	0	0	0
0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0
0	0	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

0

1854077

50 of 95

What If We Want to Keep the Same Image Dimensions?

We can use padding.

0	0	0	0
0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0
0	0	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

0

Feature Map

1854077

51 of 95

Avoiding Redundancy

We do not need to move one pixel at a time.

0	0	0	0
0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0
0	0	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

0

Stride = 2

1854077

52 of 95

Avoiding Redundancy

We do not need to move one pixel at a time.

0	0	0	0
0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0
0	0	0	0

-1	-1	-1
2	2	2
-1	-1	-1

Feature extractor

Or kernel

Horizontal line detector

0

-1

Stride = 2

……

1854077

53 of 95

Characteristics of Image Analysis

Patterns are usually smaller than high-resolution images.

Patterns can appear anywhere in the image.

Addressed with convolution

1854077

54 of 95

Characteristics of Image Analysis

Patterns are usually smaller than high-resolution images.

Patterns can appear anywhere in the image.

Patterns might appear at different scales.

1854077

55 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

-1

0

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

0

1

0

2

-2

1

0

-3

-2

1

3

-3

-1

1

-2

-1

2

-1

-2

1

2

1

-2

0

-1	2	-1
-1	2	-1
-1	2	-1

3

0

2

-1

2

-1

Down-sampling features

This is called pooling.

1854077

56 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

0

1

0

2

-2

1

0

-3

-2

1

3

-3

-1

1

-2

-1

2

-1

-2

1

2

1

-2

0

-1	2	-1
-1	2	-1
-1	2	-1

3

0

2

-1

2

-1

This is called pooling.

Down-sampling features

1854077

57 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

0

1

0

2

-2

1

0

-3

-2

1

3

-3

-1

1

-2

-1

2

-1

-2

1

2

1

-2

0

-1	2	-1
-1	2	-1
-1	2	-1

3

0

2

-1

2

-1

Down-sampling features

1854077

58 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

0

1

0

2

-2

1

0

-3

-2

1

3

-3

-1

1

-2

-1

2

-1

-2

1

2

1

-2

0

-1	2	-1
-1	2	-1
-1	2	-1

3

0

2

-1

2

-1

Down-sampling features

1854077

59 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

3

2

1

0

1

0

-1

1

-1

1

2

1

0

-1

0

1

0

2

-2

1

0

-3

-2

1

3

-3

-1

1

-2

-1

2

-1

-2

1

2

1

-2

0

-1	2	-1
-1	2	-1
-1	2	-1

3

0

2

-1

2

-1

Down-sampling features

1854077

60 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

3

2

1

2

1

-1

0

1

0

2

-2

1

0

-3

-2

1

3

-3

-1

1

-2

-1

2

-1

-2

1

2

1

-2

0

-1	2	-1
-1	2	-1
-1	2	-1

3

0

2

-1

2

-1

Down-sampling features

1854077

61 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

3

2

1

2

1

-1

0

1

0

2

-2

1

0

-3

-2

1

3

-3

-1

1

-2

-1

2

-1

-2

1

2

1

-2

0

-1	2	-1
-1	2	-1
-1	2	-1

3

0

2

-1

2

-1

Down-sampling features

1854077

62 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

3

2

1

2

1

-2

1

3

-1

2

1

0

-1	2	-1
-1	2	-1
-1	2	-1

3

2

Down-sampling features

1854077

63 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

3

2

1

2

1

3

2

1

0

-1	2	-1
-1	2	-1
-1	2	-1

3

2

Down-sampling features

1854077

64 of 95

How Can We Detect Features at Different Scale?

-1	-1	-1
2	2	2
-1	-1	-1

3

2

1

2

1

3

2

1

0

-1	2	-1
-1	2	-1
-1	2	-1

3

2

Down-sampling features

Each 2×2 block keeps its largest value. The 6×6 map becomes 3×3.

1854077

65 of 95

CNN Design

Network Design Considerations for Images
Element of the Convolutional Neural Network
CNN Design
What Do CNNs Learn?

1854077

66 of 95

Convolution

Pooling

Good features for image understanding tasks

Flatten

Cat?

This network can also be deep.

1854077

67 of 95

Convolution

Flatten

Pooling

Convolution

Pooling

Output

Each convolution and pooling block gives more feature maps, each one smaller.

Flatten the last maps into one vector and classify it with a fully connected network.

1854077

68 of 95

Convolution

Flatten

Pooling

Convolution

Pooling

Output

def __init__(self, num_classes=10):

super().__init__()

self.conv1 = nn.Conv2d(1, 8, kernel_size=3, padding=1)

nn.Conv2d(

in_channels, # # of input feature maps (e.g., 3 for RGB)

out_channels, # # of filters you learn → # of output feature maps

kernel_size, # filter size (int or (kh, kw))

stride=1, # step size of the filter (int or (sh, sw))

padding=0, # add zeros around the image (int, tuple, or 'same')

bias=True, # learn an additive bias per output channel

padding_mode='zeros' # 'zeros' (default), 'reflect', 'replicate'

)

1854077

69 of 95

How many parameters are trained in the first convolution layer?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

1854077

70 of 95

Convolution

Flatten

Pooling

Convolution

Pooling

Output

def __init__(self, num_classes=10):

super().__init__()

self.conv1 = nn.Conv2d(1, 8, kernel_size=3, padding=1)

self.pool = nn.MaxPool2d(2, 2)

1854077

71 of 95

Convolution

Flatten

Pooling

Convolution

Pooling

Output

def __init__(self, num_classes=10):

super().__init__()

self.conv1 = nn.Conv2d(1, 8, kernel_size=3, padding=1)

self.pool = nn.MaxPool2d(2, 2)

self.conv2 = nn.Conv2d(8, 64, kernel_size=3, padding=1)

1854077

72 of 95

How many parameters are trained in the second convolution layer?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

1854077

73 of 95

Convolution

Flatten

Pooling

Convolution

Pooling

Output

def __init__(self, num_classes=10):

super().__init__()

self.conv1 = nn.Conv2d(1, 8, kernel_size=3, padding=1)

self.pool = nn.MaxPool2d(2, 2)

self.conv2 = nn.Conv2d(8, 64, kernel_size=3, padding=1)

self.fc1 = nn.Linear(64 * 7 * 7, 256)

self.fc2 = nn.Linear(256, num_classes)

self.dropout = nn.Dropout(0.25)

1854077

74 of 95

Convolution

Flatten

Pooling

Convolution

Pooling

Output

def forward(self, x):

x = self.pool(F.relu(self.conv1(x))) # 8x14x14

x = self.pool(F.relu(self.conv2(x))) # 64x7x7

x = x.view(x.size(0), -1) # flatten: 64*7*7

x = self.dropout(F.relu(self.fc1(x)))

return self.fc2(x)

1854077

75 of 95

Convolution

Flatten

Pooling

Convolution

Pooling

Output

1854077

76 of 95

Convolution

Flatten

Pooling

Convolution

Pooling

Output

class SmallCNN(nn.Module):

def __init__(self):

super().__init__()

self.features = nn.Sequential(

nn.Conv2d(1, 8, 3, padding=1),

nn.ReLU(),

nn.MaxPool2d(2),

nn.Conv2d(8, 64, 3, padding=1),

nn.ReLU(),

nn.MaxPool2d(2), )

self.classifier = nn.Sequential(

nn.Flatten(),

nn.Linear(64*7*7, 256),

nn.ReLU(),

nn.Dropout(0.25),

nn.Linear(256, 10))

def forward(self, x):

x = self.features(x)

return self.classifier(x)

1854077

77 of 95

Demo: Moved Digits, Three Networks

Test accuracy on digits moved 4 pixels:

MLP: 30%

SmallCNN: 65%

CNN with global max pooling: 98%

1854077

78 of 95

Removing the Fully Connected Head

Convolution and pooling keep the layout: a moved digit gives moved feature maps.
The Linear layer after Flatten gives every position its own weights, the MLP's problem again.
Instead: one 7×7 map per class, then the largest value in each map.

nn.Sequential(

conv, ReLU, pool, # as in SmallCNN

conv, ReLU, pool,

conv, ReLU,

nn.Conv2d(64, 10, 1), # 10 maps, 7x7

nn.AdaptiveMaxPool2d(1), # max per map

nn.Flatten()) # 10 class scores

24,170 parameters (SmallCNN: 810,394).

At 4 pixels moved: 98% correct (SmallCNN: 65%).

1854077

79 of 95

What Do CNNs Learn?

Network Design Considerations for Images
Element of the Convolutional Neural Network
CNN Design
What Do CNNs Learn?

1854077

80 of 95

What Do the Filters Learn?

A deep network trained on faces and on cars (Lee et al., 2009).

Bottom: layer 1, edges.

Middle: layer 2, parts such as eyes, noses, and wheels.

Top: layer 3, whole faces and cars.

1854077

81 of 95

SmallCNN's First-Layer Kernels

Top: the 8 learned 3×3 kernels (red positive, blue negative). Bottom: each kernel's response to a 9.
Some are edge detectors at different orientations; others pass most of the ink through, like a blur.

1854077

82 of 95

Gabor Filters

Hubel and Wiesel study introduced two types of cells in brain:

Simple cells: Strong response to visual inputs with a simple edge oriented at a particular angle and located at a particular position.

More studies showed that the responses from simple cells can be modeled as Gabor filters:

Complex cells: Respond to more complex stimuli derived by combining and processing the output of simple cells.

1854077

83 of 95

The ImageNet Task (2009)

14 millionimages in more than 20,000 hierarchical categories with hand-annotation

Indicate what objects are pictured
In at least one million of the images, bounding boxes are also provided

Basis for the ImageNet Large Scale Visual Recognition Challenge (ILSVRC)

1854077

84 of 95

AlexNet (Link)

Alex Krizhevsky (NeurIPS 2012)'s AlexNet architecture is shown here.

Winner of ILSVRC 2012 by a wide margin and started the deep learning wave in vision.

The paper has 185,000 citations.
ReLU activation is used in every hidden layer.
~60 million parameters, trained on two GPUs.

Credit: https://medium.com/data-science/advanced-topics-in-deep-convolutional-neural-networks-71ef1190522d

1854077

85 of 95

AlexNet's Tricks to Improve Generalization

Slides adapted from Geofrey Hinton, University of Toronto

1854077

86 of 95

AlexNet's First Convolution Layer

96 convolutional kernels of size 11×11×3 learned by the first convolutional layer on the 224×224×3 input images.

The top 48 kernels were learned on GPU 1 while the bottom 48 kernels were learned on GPU 2.

GPU 1 kernels: mostly gray oriented edges and stripes, like Gabor filters. GPU 2 kernels: mostly color blobs.

1854077

87 of 95

A Layer-2 Kernel Is Not a Picture

Layer 2 has 64 kernels of shape 8×3×3: one 3×3 slice per layer-1 feature map.
Each slice weights a feature map, not pixels. Drawn as an image, it shows little because each slice weights a feature map, not pixels.

1854077

88 of 95

Which Patches Excite a Layer-2 Unit?

A layer-2 unit sees an 8×8 patch of the input: its receptive field.
Each row: one channel's 8 strongest patches among the 10,000 test digits.
Channels respond to bars, diagonals, corners, and vertical strokes.

1854077

89 of 95

How About Higher Layers?

Which images make a specific neuron activate?

Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR, 2014

1854077

90 of 95

How About Higher Layers?

Which images make a specific neuron activate?

Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR, 2014

Top regions for six POOL5 units: Maximally activating images for some POOL5 (5th pool layer) neurons of an AlexNet. The activation values and the receptive field of the particular neuron are shown in white. Some neurons are responsive to upper bodies, text, or specular highlights.

1854077

91 of 95

Which Pixels Matter?

Cover an 8×8 square, measure the drop in P(9), and repeat at every position.
Covering the top of the loop turns the 9 into a 4 (P = 0.80).

1854077

92 of 95

Saliency and Guided Backpropagation

Saliency: the gradient of the class score with respect to each pixel, from one backward pass.
It is noisy: negative gradients flow back through every ReLU.
Guided backprop passes only positive gradients through each ReLU.

1854077

93 of 95

Saliency and Guided Backpropagation

Saliency: the gradient of the class score with respect to each pixel, from one backward pass.
It is noisy: negative gradients flow back through every ReLU.
Guided backprop passes only positive gradients through each ReLU.

1854077

94 of 95

Demo: AlexNet on a Photo

AlexNet with its ImageNet weights, on one photo.

Top-5 predictions, random crops and flips, layer-1 kernels, feature maps, occlusion, saliency, and guided backprop.

1854077

95 of 95

Convolutional Neural Network

Lecture 16

Reading: Chapter 10 in Bishop Deep Learning Textbook

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0

0	0	0	0
0	1	1	1
1	1	1	1
1	1	1	1
0	1	1	1
0	1	1	0
1	1	1	0
1	1	0	0