Lecture 5 Smaller Network: CNN

• We know it is good to learn a small model.
• From this fully connected model, do we really need all the edges?
• Can some of these be shared?

Consider learning an image:

• Some patterns are much smaller than the whole image

“beak” detector

Can represent a small region with fewer parameters

Same pattern appears in different places:�They can be compressed!�What about training a lot of such “small” detectors�and each detector must “move around”.

“upper-left beak” detector

“middle beak” detector

They can be compressed

to the same parameters.

A convolutional layer

A filter

A CNN is a neural network with some convolutional layers

(and some other layers). A convolutional layer has a number

of filters that does convolutional operation.

Beak detector

Convolution

6 x 6 image

Filter 1

Filter 2

……

These are the network parameters to be learned.

Each filter detects a small pattern (3 x 3).

Convolution

6 x 6 image

Filter 1

3

-1

stride=1

Dot

product

Convolution

6 x 6 image

Filter 1

3

-3

If stride=2

Convolution

6 x 6 image

Filter 1

3

-1

-3

-1

-3

1

0

-3

-3

-3

0

1

3

-2

-2

-1

stride=1

Convolution

6 x 6 image

3

-1

-3

-1

-3

1

0

-3

-3

-3

0

1

3

-2

-2

-1

Filter 2

-1

-1

-1

-1

-1

-1

-2

1

-1

-1

-2

1

-1

0

-4

3

Repeat this for each filter

stride=1

Two 4 x 4 images

Forming 2 x 4 x 4 matrix

Feature

Map

Color image: RGB 3 channels

Filter 1

Filter 2

image

convolution

……

……

Convolution v.s. Fully Connected

Fully-connected

6 x 6 image

Filter 1

1

2

3

…

8

9

…

13

14

15

…

Only connect to 9 inputs, not fully connected

4:

10:

16

1

0

0

0

0

1

0

0

0

0

1

1

3

fewer parameters!

Filter 1

1:

2:

3:

…

7:

8:

9:

…

13:

14:

15:

…

4:

10:

16:

1

0

0

0

0

1

0

0

0

0

1

1

3

-1

Shared weights

6 x 6 image

Fewer parameters

Even fewer parameters

The whole CNN

cat dog ……

Convolution

Max Pooling

Convolution

Max Pooling

Flattened

Can repeat many times

Max Pooling

3

-1

-3

-1

-3

1

0

-3

-3

-3

0

1

3

-2

-2

-1

Filter 2

-1

-1

-1

-1

-1

-1

-2

1

-1

-1

-2

1

-1

0

-4

3

Filter 1

Why Pooling

• Subsampling pixels will not change the object

Subsampling

bird

bird

We can subsample the pixels to make image smaller

fewer parameters to characterize the image

A CNN compresses a fully connected network in two ways:

• Reducing number of connections
• Shared weights on the edges
• Max pooling further reduces the complexity

Max Pooling

6 x 6 image

3

0

1

3

-1

1

3

0

2 x 2 image

Each filter

is a channel

New image

but smaller

Conv

Max

Pooling

The whole CNN

Convolution

Max Pooling

Convolution

Max Pooling

Can repeat many times

A new image

The number of channels is the number of filters

Smaller than the original image

The whole CNN

cat dog ……

Convolution

Max Pooling

Convolution

Max Pooling

Flattened

A new image

A new image

Flattening

Flattened

3

0

1

3

-1

1

0

3

Only modified the network structure and input format (vector -> 3-D tensor)

CNN in Keras

Convolution

Max Pooling

Convolution

Max Pooling

input

There are 25 3x3 filters.

……

Input_shape = ( 28 , 28 , 1)

1: black/white, 3: RGB

28 x 28 pixels

3

-1

-3

1

3

Only modified the network structure and input format (vector -> 3-D array)

CNN in Keras

Convolution

Max Pooling

Convolution

Max Pooling

Input

1 x 28 x 28

25 x 26 x 26

25 x 13 x 13

50 x 11 x 11

50 x 5 x 5

How many parameters for each filter?

How many parameters

for each filter?

9

225=

25x9

Only modified the network structure and input format (vector -> 3-D array)

CNN in Keras

Convolution

Max Pooling

Convolution

Max Pooling

Input

1 x 28 x 28

25 x 26 x 26

25 x 13 x 13

50 x 11 x 11

50 x 5 x 5

Flattened

1250

Output

AlphaGo

Neural

Network

19 x 19 matrix

Black: 1

white: -1

none: 0

Fully-connected feedforward network can be used

But CNN performs much better

AlphaGo’s policy network

Note: AlphaGo does not use Max Pooling.

The following is quotation from their Nature article:

CNN in speech recognition

Time

Frequency

Spectrogram

CNN

Image

The filters move in the frequency direction.

CNN in text classification

Source of image: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.703.6858&rep=rep1&type=pdf