ME 5990: Introduction to Machine Learning

�

Autoencoder Discussion

Outline

• Autoencoder
• Principle
• Stacked Autoencoder
• Convolutional autoencoder
• Application: Image Denoise
• Application: Variational autoencoder
• Generative Adversarial Network

Introduction

• What does the convolution do?

​

​

​

​

​

​

​

• How can we know the convolution maintains the most valuable information?

​

Basic Idea of Encoding and Decoding

• Encoding and decoding

Encoding

Decoding

Autoencoders

• Encoder: compress input into a latent-space of usually smaller dimension. h = f(x)
• Decoder: reconstruct input from the latent space. r = g(f(x)) with r as close to x as possible

​

https://towardsdatascience.com/deep-inside-autoencoders-7e41f319999f

Autoencoder

• An autoencoder is a network trained to learn the identity function: output = input
• Subnetwork called encoder f(·) maps input to an embedded representation
• Subnetwork called decoder g(·) maps back to input space
• Can be thought of as lossy compression of input
• Need to identify the important attributes of inputs to reproduce faithfully

​

Stacked Autoencoder

Stacked Autoencoder

• We can simplify the training by incremental training
• Cans implify training by starting with single hidden layer H1
• Then, train the 2^nd autoencoder to mimic the output of H1
• Insert this into first network
• Can build by using H1 output as training set for the next phase

Autoencoder example

Transfer learning

• Can use the encoder as the pre-trained network with unlabeled data
• Learn useful features and then train “logic” of FC layers with labeled data

Transfer Learning from Trained Classifier

• Can also transfer from a classifier trained on different task, e.g., transfer a GoogleNet architecture to ultrasound classification
• Often choose existing one from a model zoo

Autoencoders

• Autoencoders are designed to reproduce their input, especially for images.
• Key point is to reproduce the input from a learned encoding.

​

https://www.edureka.co/blog/autoencoders-tutorial/

Convolution Autoencoder

• Convolution: Usually we downsize the image

https://epynn.net/Convolution.html

Convolution Autoencoder

• 2D Transposed Convolution:
• Stride = 1

​

https://www.geeksforgeeks.org/apply-a-2d-transposed-convolution-operation-in-pytorch/

Convolution Autoencoder

• 2D Transposed Convolution:
• Stride = 2

​

https://www.geeksforgeeks.org/apply-a-2d-transposed-convolution-operation-in-pytorch/

Convolution Autoencoder

Denoise using Autoencoder

• Denoising: input clean image + noise and train to reproduce the clean image.

​

Denoise using Autoencoder

• Image Denoise

Dor Bank, Noam Koenigstein, Raja Giryes, Autoencoders: https://arxiv.org/abs/2003.05991

Denoise using Autoencoder

Denoise

Variational Autoencoder

• Variation Autoencoder (VAE) is a generative network.
• The generative model is equivalent to a probabilistic decoder

Variational Autoencoder

• Results for VAE on Minist

Variational Autoencoder

• Face generation

https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf

Generative Adversarial Network

• A powerful generative network by Ian Goodfellow etc.

https://developers.google.com/machine-learning/gan/gan_structure

Generative Adversarial Network

• GAN has to major module: generator (similar to decoder) and discriminator

GAN

• Train the discriminator

GAN

• Train the generator, fix the discriminator!

GAN

• The generator and the discriminator have different training processes. So how do we train the GAN as a whole?
• GAN training proceeds in alternating periods:
1. The discriminator trains for one or more epochs.
2. The generator trains for one or more epochs.
3. Repeat steps 1 and 2 to continue to train the generator and discriminator networks

GAN

• The result is very destructive…

GAN

• The result is very destructive