Lecture 14: Convolutional networks
1
Announcements
2
Image classification
3
“Fish”
Classifier
image x
label y
Source: Isola, Torralba, Freeman
Image classification
4
“Fish”
image x
label y
What should these be?
Source: Isola, Torralba, Freeman
Idea #1: Fully-connected network
5
Fully-connected (a.k.a. linear) layer
But is really big!
Say, 256 x 256 x 3 = 197k
6
Can we use convolution in a neural network?
filter
Source: Isola, Torralba, Freeman
7
Sparsely connected network
Source: Isola, Torralba, Freeman
Each unit is connected to a subset of the units in the previous layer.
8
Conv layer
Each output unit is computed from an image patch.
Convolutional neural network
Source: Isola, Torralba, Freeman
9
Weight sharing
Conv layer
Source: Isola, Torralba, Freeman
We “share” weights for each patch.
If a feature is useful in one position, it should be useful in others, too.
10
Toeplitz matrix
Source: Isola, Torralba, Freeman
Convolution is a linear function
11
Conv layer
Multiple channels
Source: Isola, Torralba, Freeman
12
Conv layer
Multiple channels
Source: Isola, Torralba, Freeman
E.g.:
13
Conv layer
Multiple channels
…
…
…
Source: Isola, Torralba, Freeman
14
Multiple channels: Example
…
…
Filter Bank with 3x3 filters
128
128
3
96
128
128
How many parameters does each filter have?
(a) 9 (b) 27 (c) 96 (d) 2592
Source: Isola, Torralba, Freeman
15
Multiple channels: Example
…
…
Filter Bank with 3x3 filters
128
128
3
96
128
128
How many parameters total does this layer have?
(a) 9 (b) 27 (c) 96 (d) 2592
Image classification
16
“Fish”
Classifier
image x
label y
Conv
ReLU
Conv
ReLU
Conv
ReLU
Source: Isola, Torralba, Freeman
Problems with this idea:
17
Recall: pyramid representations
Gaussian Pyramid
Laplacian Pyramid
Can we use a similar idea in CNNs?
Source: Isola, Torralba, Freeman
18
Pooling
Filter
Pool
Max pooling
Source: Isola, Torralba, Freeman
19
Max pooling
Pooling
Filter
Pool
Source: Isola, Torralba, Freeman
Mean pooling
20
Pooling — Why?
Pooling across spatial locations achieves stability w.r.t. small translations:
Source: Isola, Torralba, Freeman
21
Pooling — Why?
Pooling across spatial locations achieves stability w.r.t. small translations:
same large response regardless of exact position of edge
Source: Isola, Torralba, Freeman
22
Pooling — Why?
Pooling across spatial locations achieves stability w.r.t. small translations:
Source: Isola, Torralba, Freeman
however, if the image is translated a lot…
23
Downsampling
Filter
Pool
and downsample
Source: Isola, Torralba, Freeman
24
Downsampling
Filter
Downsample
Source: Isola, Torralba, Freeman
25
Conv layer
Strided operations
Stride 2
Strided operations combine a given operation (convolution or pooling) and downsampling into a single operation.
Strided convolution is an alternative to pooling layers: just do a strided convolution!
Source: Isola, Torralba, Freeman
26
“clown fish”
Filter
ReLU
Downsample
…
Computation in a neural net
Source: Isola, Torralba, Freeman
Classify
27
“clown fish”
Filter
ReLU
Downsample
…
Classify using
final linear layer
Computation in a neural net
Source: Isola, Torralba, Freeman
28
Receptive fields
Source: Isola, Torralba, Freeman
29
Receptive fields
3x1 Filter
Pool and downsample by 2
RF = RF + floor(3/2)*2
kernel size
Pool and downsample by 2
Source: Isola, Torralba, Freeman
RF = RF*2
RF = RF*2
Why have coefficients only for nearby pixels?
30
[Simoncelli: Statistical Modelling of Photographic Images, 2005]
Source: Isola, Torralba, Freeman
Implementation details
31
Option 1: Just make the giant convolution matrix and use the fully backprop equations for linear layers!
Early deep learning frameworks did this.
Option 2: Use a more optimized implementation. There are optimized GPU implementations. For very large filters, these use Fourier Transform.
Max pooling: backwards pass similar to ReLU.
Convolutional layers
Backwards pass: These take the form of convolutions.
Network designs
32
33
Source: Isola, Torralba, Freeman
ImageNet classification (top 5)
34
2012: AlexNet
5 conv. layers
Error: 16.4%
[Krizhevsky et al: ImageNet Classification with Deep Convolutional Neural Networks, NeurIPS 2012]
Source: Isola, Torralba, Freeman
ImageNet classification (top 5)
35
Alexnet — [Krizhevsky et al. NIPS 2012]
[227x227x3] INPUT
[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2 [13x13x256] MAX POOL2: 3x3 filters at stride 2
[13x13x256] NORM2: Normalization layer
[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1
[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1
[4096] FC6: 4096 neurons
[4096] FC7: 4096 neurons
[1000] FC8: 1000 neurons (class scores)
Source: Isola, Torralba, Freeman
36
What filters are learned?
Source: Isola, Torralba, Freeman
37
A
B
C
D
What filters are learned?
Source: Marc’Aurelio Ranzato & Isola, Torralba, Freeman
Filters in first layer
38
fx
fy
11x11 convolution kernel
(3 color channels)
Source: Isola, Torralba, Freeman
Filters in first layer
39
fx
fy
Source: Isola, Torralba, Freeman
Filters in first layer
40
fx
fy
Source: Isola, Torralba, Freeman
Filters in first layer
41
fx
fy
Source: Isola, Torralba, Freeman
Filters in first layer
42
fx
fy
Source: Isola, Torralba, Freeman
Filters in first layer
43
fx
fy
Source: Isola, Torralba, Freeman
Filters in first layer
44
fx
fy
Source: Isola, Torralba, Freeman
Filters in first layer
45
96 Units in conv1
Source: Isola, Torralba, Freeman
46
Source: Andrea Vedaldi
A good fit to the experimental results
47
Source: Andrea Vedaldi
Oriented filters
History: Neocognitron
48
Figure source: Wikipedia
K. Fukushima, 1980s
Source: S. Lazebnik
History: LeNet
49
Source: S. Lazebnik
[LeCun, Bottou, Bengio, Haffner. “Gradient-based learning applied to document recognition”, 1998]
ImageNet Challenge
50
[Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berge, Fei-Fei,
“ImageNet Large Scale Visual Recognition Challenge”, 2015]
More recent networks
51
And advances that make them work:�
52
Source: Isola, Torralba, Freeman
53
2014: VGG
16 conv. layers
Error: 7.3%
[Simonyan & Zisserman: Very Deep Convolutional Networks�for Large-Scale Image Recognition, ICLR 2015]
Source: Isola, Torralba, Freeman
VGG-Net [Simonyan & Zisserman, 2015]
Main developments
54
2014: VGG
16 conv. layers
Error: 7.3%
| | |
| | |
| | |
Source: Isola, Torralba, Freeman
Other tricks for designing convolutional nets
Chaining convolutions
56
| | |
| | |
| | |
| | |
| | |
| | |
3x3
Nonlinearity in between.
25 coefficients, but only
18 degrees of freedom
3x3
=
5x5
| | |
|
|
|
=
| | |
| | |
| | |
9 coefficients, but only
6 degrees of freedom.
Less common.�
Source: Isola, Torralba, Freeman
(nonlinearity in between)
57
| | |
| | |
| | |
|
3x3
1x1
1x1 convolutions
Why do this?
58
Grouped Convolutions
…
…
Output:
N channels
…
…
…
…
Input:
N/2 channels
Concatenate
…
Split channels into N groups, and process separately with N convolution layers.
59
Source: Isola, Torralba, Freeman
60
[He et al: Deep Residual Learning for Image Recognition, CVPR 2016]
2016: ResNet
>100 conv. layers
Error: 3.6%
Source: Isola, Torralba, Freeman
ResNet [He et al, 2016]
Main developments
61
Source: Isola, Torralba, Freeman
Residual Blocks
62
Idea: Make it easy to represent for the network to implement the identity.
Problem: Hard to train very deep nets (50+ layers). This is an optimization issue, not overfitting: shallow models often get higher training accuracy than deep ones!
Residual connection
Normal convolution + relu:
In general, do multiple convolutions (with nonlinearities) before summing:
Residual Blocks
63
Why do they work?
Source: Isola, Torralba, Freeman
64
Normalization layers
65
Normalization layers
ReLU
Norm
Source: Isola, Torralba, Freeman
66
Normalization layers
ReLU
Norm
Source: Isola, Torralba, Freeman
67
Normalization layers
ReLU
Norm
Source: Isola, Torralba, Freeman
68
Normalization layers
ReLU
Norm
Source: Isola, Torralba, Freeman
69
[Figure from Wu & He, 2018]
Instance normalization
[Ulyanov et al., 2015]
Filter value in a CNN layer
Average and standard deviation each filter response, estimated over instance.
Normalize a single hidden unit’s activations to be mean 0, standard deviation 1.
70
Batch normalization
Normalize a single hidden unit’s activations to be mean 0, standard deviation 1.
[Ioffe & Szegedy, 2015]
Filter value in a CNN layer
Average and standard deviation each filter response, estimated over whole batch.
At test time, remember the mean and standard deviation seen during training.
[Figure from Wu & He, 2018]
A few practical issues
71
Dealing with rectangular images
72
Rectangular images
Resize, then take square
crop from center
Training with data augmentation
73
Original image
Augmented examples
Flipping
Training with data augmentation
74
Original image
Augmented examples
Flipping
Cropping
Scaling
Color jittering
Beyond image labeling
75
76
Object recognition: what objects are in the image?
“Birds”
Source: Torralba, Freeman, Isola
77
Semantic segmentation
(Colors represent categories)
General technique: predict something at every pixel!
Source: Torralba, Freeman, Isola
78
Idea #1: Independently classify windows
79
“Sky”
“Sky”
“Bird”
“Bird”
What’s the object class of the center pixel?
Training data
…
“Bird”
“Bird”
“Sky”
Source: Torralba, Freeman, Isola
80
Idea #2: Fully convolutional networks
Fully convolutional network
81
Filter
ReLU
Downsample
…
Filter with K channels
Classification problem with K classes
82
Idea #3: Dilated convolutions
83
a | 0 | b | 0 | c |
0 | 0 | 0 | 0 | 0 |
d | 0 | e | 0 | f |
0 | 0 | 0 | 0 | 0 |
g | 0 | h | 0 | i |
a | b | c |
d | e | f |
g | h | i |
3x3
5x5
Dilated convolutions
CNN without dilated convolutions
84
Filter
ReLU
Downsample
…
Classify
CNN with dilated convolutions
85
Filter
ReLU
Downsample
…
Even with dilated convolutions, still not full resolution
Classify
Dilated conv.
86
Idea #4: Encoder-decoder models
Encoder-decoder architectures
87
Encoder
Decoder
Convolutions
Deconvolutions
Source: Torralba, Freeman, Isola
Upsampling
88
1 | 2 |
3 | 4 |
1 | 1 | 2 | 2 |
1 | 1 | 2 | 2 |
3 | 3 | 4 | 4 |
3 | 3 | 4 | 4 |
Convolution
Transposed convolution
89
1 | 1 |
2 | 3 |
1 | 2 |
3 | 4 |
1 | 2 | 0 |
3 | 4 | 0 |
0 | 0 | 0 |
0 | 1 | 2 |
0 | 3 | 4 |
0 | 0 | 0 |
0 | 0 | 0 |
2 | 4 | 0 |
6 | 8 | 0 |
0 | 0 | 0 |
0 | 3 | 6 |
0 | 9 | 12 |
+
1 | 3 | 2 |
5 | 14 | 10 |
6 | 17 | 12 |
I
F
Transposed convolution
90
1 | 1 |
2 | 3 |
1 | 2 |
3 | 4 |
1 | 2 | 0 |
3 | 4 | 0 |
0 | 0 | 0 |
0 | 1 | 2 |
0 | 3 | 4 |
0 | 0 | 0 |
0 | 0 | 0 |
2 | 4 | 0 |
6 | 8 | 0 |
0 | 0 | 0 |
0 | 3 | 6 |
0 | 9 | 12 |
+
1 | 3 | 2 |
5 | 14 | 10 |
6 | 17 | 12 |
I
F
Transposed convolution
91
1 | 1 |
2 | 3 |
1 | 2 |
3 | 4 |
[Odena et al. Distill article]
Can lead to “checkerboard” artifacts.
I
F
Encoder-decoder architectures
92
Encoder
Decoder
Convolutions
Deconvolutions
Skip connections
Source: Torralba, Freeman, Isola
Skip connections feed output of earlier layer to later one
Enable learning signal in deeper network architectures - reuse features
Encoder-decoder architectures
93
Transposed convolution
“Vanilla” encoder-decoder architecture
Early layers and late layers have
same shape. Concatenate channel-wise!
U-Net
Figures from [Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks”, 2017]
U-net
concat
…
U-net
concat
concat
…
Encoder-decoder architectures
96
Source: Torralba, Freeman, Isola
Other uses for U-nets
97
+
=
Goal: recover the original image
Recall: denoising problem
Denoising
98
Loss:
+
random noise
…
99
Segnet
FCN
Input
Source: Torralba, Freeman, Isola
100
PS6: CNNs
Next class: image generation with GANs
101