1 of 101

Lecture 14: Convolutional networks

1

2 of 101

Announcements

  • PS5 due tonight, PS6 out tonight
  • Section this week: pytorch tutorial

​

2

3 of 101

Image classification

3

“Fish”

Classifier

image x

label y

Source: Isola, Torralba, Freeman

4 of 101

Image classification

4

“Fish”

image x

label y

What should these be?

Source: Isola, Torralba, Freeman

5 of 101

Idea #1: Fully-connected network

5

Fully-connected (a.k.a. linear) layer

  • If a feature is useful in one location, it should be useful in others, too.

But is really big!

Say, 256 x 256 x 3 = 197k

6 of 101

6

Can we use convolution in a neural network?

filter

Source: Isola, Torralba, Freeman

7 of 101

7

Sparsely connected network

Source: Isola, Torralba, Freeman

Each unit is connected to a subset of the units in the previous layer.

8 of 101

8

Conv layer

Each output unit is computed from an image patch.

Convolutional neural network

Source: Isola, Torralba, Freeman

9 of 101

9

Weight sharing

Conv layer

Source: Isola, Torralba, Freeman

We “share” weights for each patch.

If a feature is useful in one position, it should be useful in others, too.

10 of 101

10

Toeplitz matrix

  • Constrained linear layer
  • Fewer parameters: easier to learn, less overfitting
  • Usually use zero padding

Source: Isola, Torralba, Freeman

Convolution is a linear function

11 of 101

11

Conv layer

Multiple channels

Source: Isola, Torralba, Freeman

12 of 101

12

Conv layer

Multiple channels

Source: Isola, Torralba, Freeman

E.g.:

13 of 101

13

Conv layer

Multiple channels

…

…

…

Source: Isola, Torralba, Freeman

14 of 101

14

Multiple channels: Example

…

…

Filter Bank with 3x3 filters

128

128

3

96

128

128

How many parameters does each filter have?

(a) 9 (b) 27 (c) 96 (d) 2592

Source: Isola, Torralba, Freeman

15 of 101

15

Multiple channels: Example

…

…

Filter Bank with 3x3 filters

128

128

3

96

128

128

How many parameters total does this layer have?

(a) 9 (b) 27 (c) 96 (d) 2592

16 of 101

Image classification

16

“Fish”

Classifier

image x

label y

Conv

ReLU

Conv

ReLU

Conv

ReLU

Source: Isola, Torralba, Freeman

Problems with this idea:

  1. No “global” processing.
  2. How do you get the final label?

17 of 101

17

Recall: pyramid representations

Gaussian Pyramid

Laplacian Pyramid

Can we use a similar idea in CNNs?

Source: Isola, Torralba, Freeman

18 of 101

18

Pooling

Filter

Pool

Max pooling

Source: Isola, Torralba, Freeman

19 of 101

19

Max pooling

Pooling

Filter

Pool

Source: Isola, Torralba, Freeman

Mean pooling

20 of 101

20

Pooling — Why?

Pooling across spatial locations achieves stability w.r.t. small translations:

Source: Isola, Torralba, Freeman

21 of 101

21

Pooling — Why?

Pooling across spatial locations achieves stability w.r.t. small translations:

same large response regardless of exact position of edge

Source: Isola, Torralba, Freeman

22 of 101

22

Pooling — Why?

Pooling across spatial locations achieves stability w.r.t. small translations:

Source: Isola, Torralba, Freeman

however, if the image is translated a lot…

23 of 101

23

Downsampling

Filter

Pool

and downsample

Source: Isola, Torralba, Freeman

24 of 101

24

Downsampling

Filter

Downsample

Source: Isola, Torralba, Freeman

25 of 101

25

Conv layer

Strided operations

Stride 2

Strided operations combine a given operation (convolution or pooling) and downsampling into a single operation.

​

Strided convolution is an alternative to pooling layers: just do a strided convolution!

Source: Isola, Torralba, Freeman

26 of 101

26

“clown fish”

Filter

ReLU

Downsample

…

Computation in a neural net

Source: Isola, Torralba, Freeman

Classify

27 of 101

27

“clown fish”

Filter

ReLU

Downsample

…

Classify using

final linear layer

Computation in a neural net

Source: Isola, Torralba, Freeman

28 of 101

28

Receptive fields

Source: Isola, Torralba, Freeman

29 of 101

29

Receptive fields

3x1 Filter

Pool and downsample by 2

RF = RF + floor(3/2)*2

kernel size

Pool and downsample by 2

Source: Isola, Torralba, Freeman

RF = RF*2

RF = RF*2

30 of 101

Why have coefficients only for nearby pixels?

30

[Simoncelli: Statistical Modelling of Photographic Images, 2005]

Source: Isola, Torralba, Freeman

31 of 101

Implementation details

31

Option 1: Just make the giant convolution matrix and use the fully backprop equations for linear layers!

Early deep learning frameworks did this.

Option 2: Use a more optimized implementation. There are optimized GPU implementations. For very large filters, these use Fourier Transform.

Max pooling: backwards pass similar to ReLU.

Convolutional layers

Backwards pass: These take the form of convolutions.

32 of 101

Network designs

32

33 of 101

33

Source: Isola, Torralba, Freeman

ImageNet classification (top 5)

34 of 101

34

2012: AlexNet

5 conv. layers

Error: 16.4%

[Krizhevsky et al: ImageNet Classification with Deep Convolutional Neural Networks, NeurIPS 2012]

Source: Isola, Torralba, Freeman

ImageNet classification (top 5)

35 of 101

35

Alexnet — [Krizhevsky et al. NIPS 2012]

[227x227x3] INPUT

[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2 [13x13x256] MAX POOL2: 3x3 filters at stride 2

[13x13x256] NORM2: Normalization layer

[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1

[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1

[4096] FC6: 4096 neurons

[4096] FC7: 4096 neurons

[1000] FC8: 1000 neurons (class scores)

Source: Isola, Torralba, Freeman

36 of 101

36

What filters are learned?

Source: Isola, Torralba, Freeman

37 of 101

37

A

B

C

D

What filters are learned?

Source: Marc’Aurelio Ranzato & Isola, Torralba, Freeman

38 of 101

Filters in first layer

38

fx

fy

11x11 convolution kernel

(3 color channels)

Source: Isola, Torralba, Freeman

39 of 101

Filters in first layer

39

fx

fy

Source: Isola, Torralba, Freeman

40 of 101

Filters in first layer

40

fx

fy

Source: Isola, Torralba, Freeman

41 of 101

Filters in first layer

41

fx

fy

Source: Isola, Torralba, Freeman

42 of 101

Filters in first layer

42

fx

fy

Source: Isola, Torralba, Freeman

43 of 101

Filters in first layer

43

fx

fy

Source: Isola, Torralba, Freeman

44 of 101

Filters in first layer

44

fx

fy

Source: Isola, Torralba, Freeman

45 of 101

Filters in first layer

45

96 Units in conv1

Source: Isola, Torralba, Freeman

46 of 101

46

Source: Andrea Vedaldi

47 of 101

A good fit to the experimental results

47

Source: Andrea Vedaldi

Oriented filters

48 of 101

History: Neocognitron

48

Figure source: Wikipedia

K. Fukushima, 1980s

Source: S. Lazebnik

49 of 101

History: LeNet

  • Neocognitron + backpropagation
  • Average pooling
  • Sigmoid or tanh nonlinearity
  • Fully connected layers at the end
  • Trained on MNIST digit dataset with 60K training examples

49

Source: S. Lazebnik

[LeCun, Bottou, Bengio, Haffner. “Gradient-based learning applied to document recognition”, 1998]

50 of 101

ImageNet Challenge

50

  • ~14 million labeled images, 20k classes

​

  • Images gathered from Internet

​

  • Human labels via Amazon MTurk

​

  • ImageNet Large-Scale Visual Recognition Challenge (ILSVRC): �1.2 million training images, 1000 classes

[Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berge, Fei-Fei,

“ImageNet Large Scale Visual Recognition Challenge”, 2015]

51 of 101

More recent networks

51

And advances that make them work:�

  • Chaining small filters
  • Residual layers
  • Normalization

52 of 101

52

Source: Isola, Torralba, Freeman

53 of 101

53

2014: VGG

16 conv. layers

Error: 7.3%

[Simonyan & Zisserman: Very Deep Convolutional Networks�for Large-Scale Image Recognition, ICLR 2015]

Source: Isola, Torralba, Freeman

54 of 101

VGG-Net [Simonyan & Zisserman, 2015]

Main developments

  • Small convolutional kernels: only 3x3����
  • Increased depth (5 -> 16/19 layers)

54

2014: VGG

16 conv. layers

Error: 7.3%

​

​

​

​

​

​

​

​

​

Source: Isola, Torralba, Freeman

55 of 101

Other tricks for designing convolutional nets

56 of 101

Chaining convolutions

56

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

3x3

Nonlinearity in between.

25 coefficients, but only

18 degrees of freedom

3x3

=

5x5

​

​

​

​

​

​

=

​

​

​

​

​

​

​

​

​

9 coefficients, but only

6 degrees of freedom.

Less common.�

Source: Isola, Torralba, Freeman

57 of 101

(nonlinearity in between)

57

​

​

​

​

​

​

​

​

​

​

3x3

1x1

1x1 convolutions

Why do this?

58 of 101

58

Grouped Convolutions

…

…

Output:

N channels

…

…

…

…

Input:

N/2 channels

Concatenate

…

Split channels into N groups, and process separately with N convolution layers.

59 of 101

59

Source: Isola, Torralba, Freeman

60 of 101

60

[He et al: Deep Residual Learning for Image Recognition, CVPR 2016]

2016: ResNet

>100 conv. layers

Error: 3.6%

Source: Isola, Torralba, Freeman

61 of 101

ResNet [He et al, 2016]

Main developments

  • Increased depth possible through residual blocks

61

Source: Isola, Torralba, Freeman

62 of 101

Residual Blocks

62

Idea: Make it easy to represent for the network to implement the identity.

Problem: Hard to train very deep nets (50+ layers). This is an optimization issue, not overfitting: shallow models often get higher training accuracy than deep ones!

Residual connection

Normal convolution + relu:

In general, do multiple convolutions (with nonlinearities) before summing:

63 of 101

Residual Blocks

63

Why do they work?

​

  • Gradients can propagate faster�(via the identity mapping)�
  • Within each block, only small�residuals have to be learned

Source: Isola, Torralba, Freeman

64 of 101

64

  • Standardize activations by subtracting mean and dividing by standard deviation (averaged over all spatial locations).

Normalization layers

  • Obtains invariance to mean and variance.
  • This provides a constant “interface” for later layers of the networks. Ensures that the previous layer will have unit variance and zero mean.
  • Can allow you to train with larger learning rate and significantly speed up training!

65 of 101

65

Normalization layers

ReLU

Norm

Source: Isola, Torralba, Freeman

66 of 101

66

Normalization layers

ReLU

Norm

Source: Isola, Torralba, Freeman

67 of 101

67

Normalization layers

ReLU

Norm

Source: Isola, Torralba, Freeman

68 of 101

68

Normalization layers

ReLU

Norm

Source: Isola, Torralba, Freeman

69 of 101

69

[Figure from Wu & He, 2018]

Instance normalization

[Ulyanov et al., 2015]

Filter value in a CNN layer

Average and standard deviation each filter response, estimated over instance.

Normalize a single hidden unit’s activations to be mean 0, standard deviation 1.

70 of 101

70

Batch normalization

Normalize a single hidden unit’s activations to be mean 0, standard deviation 1.

[Ioffe & Szegedy, 2015]

Filter value in a CNN layer

Average and standard deviation each filter response, estimated over whole batch.

At test time, remember the mean and standard deviation seen during training.

[Figure from Wu & He, 2018]

71 of 101

A few practical issues

71

72 of 101

Dealing with rectangular images

72

Rectangular images

Resize, then take square

crop from center

73 of 101

Training with data augmentation

73

Original image

Augmented examples

Flipping

  • Less susceptible to overfitting.
  • Improves performance by simulating examples.

74 of 101

Training with data augmentation

74

Original image

Augmented examples

Flipping

Cropping

Scaling

Color jittering

75 of 101

Beyond image labeling

75

76 of 101

76

Object recognition: what objects are in the image?

“Birds”

Source: Torralba, Freeman, Isola

77 of 101

77

Semantic segmentation

(Colors represent categories)

General technique: predict something at every pixel!

Source: Torralba, Freeman, Isola

78 of 101

78

Idea #1: Independently classify windows

79 of 101

79

“Sky”

“Sky”

“Bird”

“Bird”

What’s the object class of the center pixel?

Training data

…

“Bird”

“Bird”

“Sky”

Source: Torralba, Freeman, Isola

80 of 101

80

Idea #2: Fully convolutional networks

81 of 101

Fully convolutional network

81

Filter

ReLU

Downsample

…

Filter with K channels

Classification problem with K classes

82 of 101

82

Idea #3: Dilated convolutions

83 of 101

83

a

0

b

0

c

0

0

0

0

0

d

0

e

0

f

0

0

0

0

0

g

0

h

0

i

a

b

c

d

e

f

g

h

i

3x3

5x5

  • Alternative to pooling that preserves input size
  • 9 degrees of freedom
  • 5x5 receptive field

Dilated convolutions

84 of 101

CNN without dilated convolutions

84

Filter

ReLU

Downsample

…

Classify

85 of 101

CNN with dilated convolutions

85

Filter

ReLU

Downsample

…

Even with dilated convolutions, still not full resolution

Classify

Dilated conv.

86 of 101

86

Idea #4: Encoder-decoder models

87 of 101

Encoder-decoder architectures

87

Encoder

Decoder

Convolutions

Deconvolutions

Source: Torralba, Freeman, Isola

88 of 101

Upsampling

88

1

2

3

4

1

1

2

2

1

1

2

2

3

3

4

4

3

3

4

4

  • Often using nearest-neighbor upsampling
  • Can also use interpolation.
  • Produces fewer “checkerboard” artifacts

Convolution

89 of 101

Transposed convolution

89

1

1

2

3

1

2

3

4

  • Weight the filter by the image coefficient and sum.
  • Also sometimes called “upconvolution” or “deconvolution”.

1

2

0

3

4

0

0

0

0

0

1

2

0

3

4

0

0

0

0

0

0

2

4

0

6

8

0

0

0

0

0

3

6

0

9

12

+

1

3

2

5

14

10

6

17

12

I

F

90 of 101

Transposed convolution

90

1

1

2

3

1

2

3

4

  • Weight the filter by the image coefficient and sum.
  • Also sometimes called “upconvolution” or “deconvolution”.

1

2

0

3

4

0

0

0

0

0

1

2

0

3

4

0

0

0

0

0

0

2

4

0

6

8

0

0

0

0

0

3

6

0

9

12

+

1

3

2

5

14

10

6

17

12

I

F

91 of 101

Transposed convolution

91

1

1

2

3

1

2

3

4

[Odena et al. Distill article]

Can lead to “checkerboard” artifacts.

I

F

92 of 101

Encoder-decoder architectures

92

Encoder

Decoder

Convolutions

Deconvolutions

Skip connections

Source: Torralba, Freeman, Isola

Skip connections feed output of earlier layer to later one

​

Enable learning signal in deeper network architectures - reuse features

93 of 101

Encoder-decoder architectures

93

Transposed convolution

“Vanilla” encoder-decoder architecture

Early layers and late layers have

same shape. Concatenate channel-wise!

U-Net

Figures from [Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks”, 2017]

94 of 101

U-net

concat

…

95 of 101

U-net

concat

concat

…

96 of 101

Encoder-decoder architectures

96

Source: Torralba, Freeman, Isola

97 of 101

Other uses for U-nets

97

+

=

Goal: recover the original image

Recall: denoising problem

98 of 101

Denoising

98

Loss:

+

random noise

…

99 of 101

99

Segnet

FCN

Input

Source: Torralba, Freeman, Isola

100 of 101

  • Train a convolutional network to recognize scenes
  • Use PyTorch + autodiff
  • Train on GPU

100

PS6: CNNs

101 of 101

Next class: image generation with GANs

101