Module 4
Convolutional Neural Networks
Convolutional Neural Networks
History
Sparse connectivity with shared weights.
Basic Structure of a Convolutional Neural Network
Filter for Convolution Operation
– The filter’s spatial dimensions must be no layer’s spatial dimensions.
larger than
Convolution Operation
Fq + 1) × (Bq − Fq + 1) spatial positions.
– Corresponds to number of positions for top left corner in input volume, so that filter fully fits inside layer volume.
Convolution Operation: Pictorial Illustration of Dimensions
32
28
5
5
32
3
FILTER
28
5
3
INPUT
OUTPUT
DEPTH OF INPUT AND FILTER MUST MATCH
DEPTH DEFINED BY NUMBER OF DIFFERENT FILTERS (5)
HIGH ACTIVATION
1 | 1 | 1 |
0 | 0 | 0 |
-1 | -1 | -1 |
HORIZONTAL EDGE DETECTING FILTER
IMAGE
ZERO ACTIVATION
(a) Input and output dimensions
(b) Sliding the filter
Convolution Operation: Numerical Example with Depth 1
6 | 3 | 4 | 4 | 5 | 0 | 3 |
4 | 7 | 4 | 0 | 4 | 0 | 4 |
7 | 0 | 2 | 3 | 4 | 5 | 2 |
3 | 7 | 5 | 0 | 3 | 0 | 7 |
5 | 8 | 1 | 2 | 5 | 4 | 2 |
8 | 0 | 1 | 0 | 6 | 0 | 0 |
6 | 4 | 1 | 3 | 0 | 4 | 5 |
CONVOLVE
1 | 0 | 1 |
1 | 0 | 0 |
0 | 0 | 2 |
16
18 | 20 | 21 | 14 | 16 |
25 | 7 | 16 | 3 | 26 |
14 | 14 | 21 | 16 | 13 |
15 | 15 | 21 | 2 | 15 |
16 | 16 | 7 | 16 | 23 |
OUTPUT
INPUT
FILTER
26
16
Understanding Convolution
Effects of Convolution
– The activations in the first, second, and third hidden lay-ers capture pixel regions of size 3 × 3, 5 × 5, and 7 × 7, respectively, in the original input image.
Need for Padding
– This type of reduction in size is not desirable in general, because it tends to lose some information along the bor-ders of the image (or of the feature map, in the case of hidden layers).
Application of Padding with 2 Zeros
6 | 3 | 4 | 4 | 5 | 0 | 3 |
4 | 7 | 4 | 0 | 4 | 0 | 4 |
7 | 0 | 2 | 3 | 4 | 5 | 2 |
3 | 7 | 5 | 0 | 3 | 0 | 7 |
5 | 8 | 1 | 2 | 5 | 4 | 2 |
8 | 0 | 1 | 0 | 6 | 0 | 0 |
6 | 4 | 1 | 3 | 0 | 4 | 5 |
PAD
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 6 | 3 | 4 | 4 | 5 | 0 | 3 | 0 | 0 |
0 | 0 | 4 | 7 | 4 | 0 | 4 | 0 | 4 | 0 | 0 |
0 | 0 | 7 | 0 | 2 | 3 | 4 | 5 | 2 | 0 | 0 |
0 | 0 | 3 | 7 | 5 | 0 | 3 | 0 | 7 | 0 | 0 |
0 | 0 | 5 | 8 | 1 | 2 | 5 | 4 | 2 | 0 | 0 |
0 | 0 | 8 | 0 | 1 | 0 | 6 | 0 | 0 | 0 | 0 |
0 | 0 | 6 | 4 | 1 | 3 | 0 | 4 | 5 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Types of Padding
spatial footprint with (Fq − 1).
2 ∗ Amount Padded + Size Reduction = (Fq − 1)
(1)
Strided Convolution
– Exact divisibility is required.
convolutions are sometimes used in lieu of max-
Max Pooling
activation maps, the maximum of these values is returned.
Pq = Sq).
Pooling Example
6 | 3 | 4 | 4 | 5 | 0 | 3 |
4 | 7 | 4 | 0 | 4 | 0 | 4 |
7 | 0 | 2 | 3 | 4 | 5 | 2 |
3 | 7 | 5 | 0 | 3 | 0 | 7 |
5 | 8 | 1 | 2 | 5 | 4 | 2 |
8 | 0 | 1 | 0 | 6 | 0 | 0 |
6 | 4 | 1 | 3 | 0 | 4 | 5 |
3X3 POOLING STRIDE=1
7 | 7 | 5 | 5 | 5 |
7 | 7 | 5 | 5 | 7 |
8 | 8 | 5 | 5 | 7 |
8 | 8 | 6 | 6 | 7 |
8 | 8 | 6 | 6 | 6 |
| | | | |
| | | | |
| | | | |
| | | | |
8 | | | | |
INPUT
| | | | |
| | | | 7 |
| | | | |
| | | | |
8 | | | | |
OUTPUT
3X3 POOLING STRIDE=1
3X3 POOLING STRIDE=1
7 | 5 | 5 |
8 | 5 | 7 |
8 | 6 | 6 |
OUTPUT
ReLU
Fully Connected Layers: Stuck at the End
Interleaving Between Layers
CRCRP CRCRCRP
Example: LeNet-5: Full Notation
CONVOLUTION OPERATIONS
C1
32
28
C5
F6
O
28
14
INPUT: GRAYSCALE FEATURE MAP | | 5 | | 2 | 14 | S2 | 10 | C3 | | 5 | S4 | |
OF PIXELS 32 | 5 | | 2 | | | 5 5 | | 10 | 2 2 | | | 5 |
16
16
84
10
6
120
6
SUBSAMPLING OPERATIONS
Feature Engineering
1 | 1 | 1 |
0 | 0 | 0 |
-1 | -1 | -1 |
HORIZONTAL
EDGES DETECTED
FILTER
IMAGE
1 | 0 | -1 |
1 | 0 | -1 |
1 | 0 | -1 |
NEXT LAYER FILTER (VISUALIZATION UNINTERPRETABLE)
RECTANGLE DETECTED
FILTER
VERTICAL EDGES DETECTED
Hierarchical Feature Engineering
Backpropagation in Convolutional Neural Networks
– Passes gradient to a previous layer only if the original input value was positive.
Backpropagating through Convolutions
Backpropagation with an Inverted Filter [Single Channel]
a | b | c |
d | e | f |
g | h | i |
i | h | g |
f | e | d |
c | b | a |
FILTER DURING
CONVOLUTION
FILTER DURING
BACKPROPAGATION
Convolution as a Matrix Multiplication
Convolution as a Matrix Multiplication
1 | 3 | 4 |
9 | 6 | 7 |
5 | 0 | 2 |
CONVERT TO 4X9 SPARSE MATRIX C
FILTER
a | b |
c | d |
a b 0 c d 0 0 0 0
0 a b 0 c d 0 0 0
0 0 0 a b 0 c d 0
0 0 0 0 a b 0 c d
FLATTEN TO
9-DIMENSIONAL VECTOR
f= [1, 3, 4, 9, 6, 7, 5, 0, 2]T
INPUT
MULT IPLY
C * f
[a+3b+9c+6d, 3a+4b+6c+7d, 9a+6b+5c, 6a+7b+2d]T
RESHAPE TO
SPATIAL OUTPUT
a+3b+9c+6d | 3a+4b+6c+7d |
9a+6b+5c | 6a+7b+2d |
OUTPUT
Gradient-Based Visualization
∂xi
– Compute maximum absolute magnitude of gradient over RGB colors and create grayscale image.
Gradient-Based Visualization
Getting Cleaner Visualizations
backpropagated gradient is positive.
Guided Backpropagation
Illustration of “Deconvnet” and Guided Backpropagation
1 | -1 | 2 |
-2 | 3 | 4 |
-2 | 1 | 3 |
ACTIVATION (LAYER i)
FORWARD PASS
ReLU
1 | 0 | 2 |
0 | 3 | 4 |
0 | 1 | 3 |
ACTIVATION
(LAYER i+1)
“GRADIENTS”
(LAYER i+1)
“GRADIENTS” (LAYER i)
-3 | 2 | -1 |
-1 | 2 | 2 |
1 | 2 | -4 |
BACKWARD PASS
-3 | 0 | -1 |
0 | 2 | 2 |
0 | 2 | -4 |
0 | 2 | 0 |
0 | 2 | 2 |
1 | 2 | 0 |
0 | 0 | 0 |
0 | 2 | 2 |
0 | 2 | 0 |
TRADITIONAL BACKPROPAGATION
“DECONVNET” (APPLY ReLU)
GUIDED BACKPROPAGATION
Derivatives of Features with Respect to Input Pixels
deconv
guided backpropagation
corresponding image crops
deconv
guided backpropagation
corresponding image crops
Creating a fantasy image that matches a label
Maximizex J(x) = (o − λ||x||2)
Examples
cup
dalmatian
goose
Generalization to Autoencoders
AlexNet
– Brought attention to the area of deep learning
AlexNet Architecture
INPUT
224
C1
55
FC6 FC7
FC8
C2
11
C3
C4
C5
5
27
13
13
13
11
5
3
3
3
3
3
3
224
13
13
13
27
55
384
384
256
MP
1000
256
MP
4096 4096
96 MP
3
Other Comments on AlexNet
ZfNet
| AlexNet | ZFNet |
Volume: Operations: | 224 × 224 × 3 Conv 11 × 11 (stride 4) | 224 × 224 × 3 Conv 7 × 7 (stride 2), MP |
Volume: Operations: | 55 × 55 × 96 Conv 5 × 5, MP | 55 × 55 × 96 Conv 5 × 5 (stride 2), MP |
Volume: Operations: | 27 × 27 × 256 Conv 3 × 3, MP | 13 × 13 × 256 Conv 3 × 3 |
Volume: Operations: | 13 × 13 × 384 Conv 3 × 3 | 13 × 13 × 512 Conv 3 × 3 |
Volume: Operations: | 13 × 13 × 384 Conv 3 × 3 | 13 × 13 × 1024 Conv 3 × 3 |
Volume: Operations: | 13 × 13 × 256 MP, Fully connect | 13 × 13 × 512 MP, Fully connect |
FC6: Operations: | 4096 Fully connect | 4096 Fully connect |
FC7: Operations: | 4096 Fully connect | 4096 Fully connect |
FC8: Operations: | 1000 Softmax | 1000 Softmax |
VGG
Principle of Reduced Filter Size
VGG Configurations
Name: | A | A-LRN | B | C | D | E |
# Layers | 11 | 11 | 13 | 16 | 16 | 19 |
| C3D64 | C3D64 | C3D64 | C3D64 | C3D64 | C3D64 |
| | LRN | C3D64 | C3D64 | C3D64 | C3D64 |
| M | M | M | M | M | M |
| C3D128 | C3D128 | C3D128 | C3D128 | C3D128 | C3D128 |
| | | C3D128 | C3D128 | C3D128 | C3D128 |
| M | M | M | M | M | M |
| C3D256 | C3D256 | C3D256 | C3D256 | C3D256 | C3D256 |
| C3D256 | C3D256 | C3D256 | C3D256 | C3D256 | C3D256 |
| | | | C1D256 | C3D256 | C3D256 |
| | | | | | C3D256 |
| M | M | M | M | M | M |
| C3D512 | C3D512 | C3D512 | C3D512 | C3D512 | C3D512 |
| C3D512 | C3D512 | C3D512 | C3D512 | C3D512 | C3D512 |
| | | | C1D512 | C3D512 | C3D512 |
| | | | | | C3D512 |
| M | M | M | M | M | M |
| C3D512 | C3D512 | C3D512 | C3D512 | C3D512 | C3D512 |
| C3D512 | C3D512 | C3D512 | C3D512 | C3D512 | C3D512 |
| | | | C1D512 | C3D512 | C3D512 |
| | | | | | C3D512 |
| M | M | M | M | M | M |
| FC4096 | FC4096 | FC4096 | FC4096 | FC4096 | FC4096 |
| FC4096 | FC4096 | FC4096 | FC4096 | FC4096 | FC4096 |
| FC1000 | FC1000 | FC1000 | FC1000 | FC1000 | FC1000 |
| S | S | S | S | S | S |
VGG Design Choices and Performance
– Volume remained roughly constant.
GoogLeNet
Inception Module: Motivation
Basic Inception Module
FILTER CONCATENATION
1 X 1
CONVOLUTIONS
PREVIOUS LAYER
3 X 3
CONVOLUTIONS
5 X 5
CONVOLUTIONS
3 X 3 MAX-POOLING
Computationally Efficient Inception Module
FILTER CONCATENATION
3 X 3 CONVOLUTIONS
5 X 5 CONVOLUTIONS
1 X 1 CONVOLUTIONS
1 X 1
CONVOLUTIONS
1 X 1
CONVOLUTIONS
1 X 1
CONVOLUTIONS
3 X 3
MAX-POOLING
PREVIOUS LAYER
Design Principles of Output Layer
– Helps in reducing parameter footprint
GoogLeNet Architecture
Other Details on GoogLeNet
ResNet Motivation
– Poor convergence
ResNet
– Where have we heard this before?
Skip Connections in ResNet
7X7 CONV, 64, /2
POOL, /2
3X3 CONV, 64
x
WEIGHT LAYER
F(x)
ReLU
WEIGHT LAYER
F(x)+x
+
3X3 CONV, 64
3X3 CONV, 64
3X3 CONV, 64
3X3 CONV, 64
IDENTITY
x
3X3 CONV, 64
3X3 CONV, 128, 1/2
3X3 CONV, 128
3X3 CONV, 128
ReLU
3X3 CONV, 128
(a) Residual module
(b) Partial architecture
Details of ResNet
Importance of Depth
Name | Year | Number of Layers | Top-5 Error |
- | Before 2012 | ≤ 5 | > 25% |
AlexNet | 2012 | 8 | 15.4% |
ZfNet/Clarifai | 2013 | 8/> 8 | 14.8% / 11.1% |
VGG | 2014 | 19 | 7.3% |
GoogLeNet | 2014 | 22 | 6.7% |
ResNet | 2015 | 152 | 3.6% |
Pretrained Models
– A nearest neighbor classifier on raw pixels vs features from fully connected layers will show huge differences.
Object Localization
Object Localization
SOFTMAX
FULLY CONNECTED
CLASSIFICATION HEAD
CONVOLUTION LAYERS (WEIGHTS FIXED FOR BOTH CLASSIFICATION AND REGRESSION)
FULLY CONNECTED
CLASS PROBABILITIES
TRAIN FOR CLASSIFICATION
FULLY CONNECTED
LINEAR LAYER
FULLY CONNECTED
REGRESSION HEAD
BOUNDING BOX (FOUR NUMBERS)
TRAIN FOR REGRESSION
Other Applications