Convolutional Autoencoder

Industrial AI Lab.

Prof. Seungchul Lee

Convolutional Autoencoder

• Motivation: image to autoencoder ?
• Convolutional autoencoder extends the basic structure of the simple autoencoder by changing the fully connected layers to convolution layers.
• the network of encoder change to convolution layers
• the network of decoder change to transposed convolutional layers
• A transposed 2-D convolution layer upsamples feature maps.
• This layer is sometimes incorrectly known as a "deconvolution" or "deconv" layer.
• This layer is the transpose of convolution and does not perform deconvolution.

2

downsample

upsample

tf.keras.models.Conv2D

• Encoder
• Padding

3

padding = ‘VALID’

strides = [1, 1, 1, 1]

tf.keras.models.Conv2D

• Encoder
• Padding

4

padding = ‘VALID’

strides = [1, 1, 1, 1]

padding = ‘SAME’

strides = [1, 1, 1, 1]

tf.keras.models.Conv2D

• Encoder
• Stride

5

padding = ‘SAME’

strides = [1, 1, 1, 1]

padding = ‘SAME’

strides = [1, 2, 2, 1]

tf.keras.models.Conv2DTranspose

• Decoder
• Stride

6

padding = ‘VALID’

strides = (1,1)

padding = ‘VALID’

strides = (1,1)

tf.keras.models.Conv2DTranspose

• Decoder
• Stride

7

padding = ‘VALID’

strides = (2,2)

padding = ‘VALID’

strides = (2,2)

tf.keras.models.Conv2DTranspose

• Decoder
• Stride

8

padding = ‘SAME’

strides = (2,2)

padding = ‘SAME’

strides = (2,2)

CAE Implementation

• Fully convolutional
• Note that no dense layer is used

9

CAE Implementation

10

CAE Implementation

11

CAE Implementation

12

Reconstruction Result

13

Fully Convolutional Network (FCN)

Industrial AI Lab.

Prof. Seungchul Lee

Deep Learning for Computer Vision: Review

15

Source: 6.S191 Intro. to Deep Learning at MIT

Segmentation

• Segmentation task is different from classification task because it requires predicting a class for each pixel of the input image, instead of only 1 class for the whole input.
• Segment images into regions with different semantic categories. These semantic regions label and predict objects at the pixel level

​

​

16

Image from http://d2l.ai/

Segmentation

• Segmentation task is different from classification task because it requires predicting a class for each pixel of the input image, instead of only 1 class for the whole input.
• Segment images into regions with different semantic categories. These semantic regions label and predict objects at the pixel level
• Classification needs to understand what is in the input (namely, the context).
• However, in order to predict what is in the input for each pixel, segmentation needs to recover not only what is in the input, but also where.

17

Image from http://d2l.ai/

Semantic Segmentation: FCNs

• FCN uses a convolutional neural network to transform image pixels to pixel categories.

​

• Network designed with all convolutional layers, with down-sampling and up-sampling operations

​

• Given a position on the spatial dimension, the output of the channel dimension will be a category prediction of the pixel corresponding to the location.

18

Image from http://d2l.ai/

From CAE to FCN

19

From CAE to FCN

20

CAE

FCN

Skip Connection

• A skip connection is a connection that bypasses at least one layer.

​

• Here, it is often used to transfer local information by summing feature maps from the downsampling path with feature maps from the upsampling path.
• Merging features from various resolution levels helps combining context information with spatial information.

21

ResNet (Deep Residual Learning)

• He, Kaiming, et al. “Deep residual learning for image recognition.” CVPR. 2016.
• Plain net

22

ResNet (Deep Residual Learning)

• He, Kaiming, et al. “Deep residual learning for image recognition.” CVPR. 2016.
• Residual net
• Skip connection

​

23

- A direct connection between 2 non-consecutive layers

- No gradient vanishing

ResNet (Deep Residual Learning)

24

• If identity were optimal, easy to set weights as 0

​

• If optimal mapping is closer to identity, easier to find small fluctuations

Residual Net

25

Fully Convolutional Networks (FCNs)

• To obtain a segmentation map (output), segmentation networks usually have 2 parts
• Downsampling path: capture semantic/contextual information
• Upsampling path: recover spatial information
• The downsampling path is used to extract and interpret the context (what), while the upsampling path is used to enable precise localization (where).

​

• Furthermore, to fully recover the fine-grained spatial information lost in the pooling or downsampling layers, we often use skip connections.

​

• Network can work regardless of the original image size, without requiring any fixed number of units at any stage.

​

26

Segmented (Labeled) Images

27

input

output

FCN Architecture

28

FCN Architecture

29

Fixed

maxp3

maxp4

fcn4

fcn3

fcn2

fcn1

Trained

FCN Architecture

30

Fixed

maxp3

maxp4

fcn4

fcn3

fcn2

fcn1

Trained

FCN Architecture

31

Fixed

maxp3

maxp4

fcn4

fcn3

fcn2

fcn1

Trained

Segmentation Result

32

maxp3

maxp4

input

Segmentation output

overlapping

Super-resolution and Deblurring

Prof. Seungchul Lee

Industrial AI Lab.

Image Restoration

• The sources of corruption in digital images arise during image acquisition (digitization) and transmission
• Imaging sensors can be affected by ambient conditions
• Interference can be added to an image during transmission
• Image restoration
• To recover original image from degraded one with prior knowledge of degradation process

​

34

Inverse Problem

• Inverse problems involve modeling of degradation and applying the inverse process in order to recover the original image from inadequate observations
• The observations contain incomplete information about the target parameter or data due to physical limitations of the measurement devices
• Consequently, solutions to inverse problems are non-unique

​

35

Image Super-resolution (SR)

• Restore high resolution (HR) image from low resolution (LR) image

​

​

​

​

​

​

​

​

​

• Numerous learning-based SR approaches

​

​

36

Lab: SR on Material Images

37

Build a FCN Model

38

Build a FCN Model

39

Build a FCN Model

40

Training

41

Result

42

Image Deblurring

43

Build a FCN Model

44

Build a FCN Model

45

Build a FCN Model

46

Training

47

Result

48