Generative Adversarial Networks (GAN)

Prof. Seungchul Lee

Industrial AI Lab.

Supervised Learning

• Discriminative model

​

2

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Unsupervised Learning

• Generative model

​

3

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

= Latent space

Model Distribution vs. Data Distribution

4

Probability Distribution

5

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Probability Distribution

6

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Probability Density Estimation Problem

7

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Deterministic Transformation (by Network)

• 1-dimensional example:

​

​

​

​

​

​

​

​

​

​

• Remember
• Network does not generate distribution, but
• It maps known distribution to target distribution

8

Source: Prof. Roger Grosse at U of Toronto

Generative Models from Lower Dimension

9

Source: Prof. Roger Grosse at U of Toronto

Turing Test

• One way to judge the quality of the model is to sample from it.
• GANs are based on a very different idea:
• Model to produce samples which are indistinguishable from the real data, as judged by a discriminator network whose job is to tell real from fake

10

Deterministic Transformation (by Network)

• High dimensional example:

11

Source: Prof. Roger Grosse at U of Toronto

Generative Adversarial Networks (GAN)

• The idea behind Generative Adversarial Networks (GANs): train two different networks
• Generator network: try to produce realistic-looking samples
• Discriminator network: try to distinguish between real and fake data

​

• The generator network tries to fool the discriminator network

12

Autoencoder

• Dimension reduction
• Recover the input data
• Learns an encoding of the inputs so as to recover the original input from the encodings as well as possible

​

​

13

Generative Adversarial Networks (GAN)

• Analogous to Turing Test

​

14

Generated

Generator

Data Generator

Generative Adversarial Networks (GAN)

• Analogous to Turing Test

​

​

15

Intuition for GAN

16

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Discriminator Perspective (1/2)

17

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Discriminator Perspective (2/2)

18

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Generator Perspective

19

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

GAN Implementation in TensorFlow

20

TensorFlow Implementation

21

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Generator

22

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Discriminator

23

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Combined

24

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Training: Discriminator

25

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Training: Generator

26

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

After Training

• After training, use generator network to generate new data

27

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Generated Images

28

DCGAN

29

DCGAN (Deep Convolutional GAN)

• A DCGAN is a direct extension of the GAN, except that it explicitly uses convolutional and convolutional-transpose layers in the discriminator and generator, respectively
• (For example) simplified architecture

​

30

Alec Radford, et al., "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks," ICLR 2016

Conditional GAN

31

Normal Distribution of MNIST

• A standard normal distribution
• This is how we would like points corresponding to MNIST digit images to be distributed in the latent space

32

Generator at GAN

33

Generator

Generator

Conditional GAN

• In an unconditioned generative model, there is no control on modes of the data being generated.

​

• In the Conditional GAN (CGAN), the generator learns to generate a fake sample with a specific condition or characteristics (such as a label associated with an image or more detailed tag) rather than a generic sample from unknown noise distribution.

34

Conditional GAN

• MNIST digits generated conditioned on their class label

35

Conditional GAN

• Simple modification to the original GAN framework that conditions the model on additional information for better multi-modal learning

​

​

​

• Many practical applications of GANs when we have explicit supervision available

36

Conditional GAN

• Simple modification to the original GAN framework that conditions the model on additional information for better multi-modal learning

​

​

​

• Many practical applications of GANs when we have explicit supervision available

37

Conditional GAN

• Simple modification to the original GAN framework that conditions the model on additional information for better multi-modal learning

​

​

​

• Many practical applications of GANs when we have explicit supervision available

38

Generator at GAN

39

Generator

Generator

Generator at Conditional GAN

• Feed a random point in latent space and desired number.
• Even if the same latent point is used for two different numbers, the process will work correctly since the latent space only encodes features such as stroke width or angle

40

Generator

Generator

Generator

41

Discriminator

42

Combined

43

CGAN Implementation

44

Fake MNIST Images Generated by CGAN

45

Adversarial Autoencoder

46

Autoencoder

• Dimension reduction
• Recover the input data
• Learns an encoding of the inputs so as to recover the original input from the encodings as well as possible

​

​

47

Image from https://lilianweng.github.io/

Limitation of Autoencoder

48

Generator from Autoencoder

• Dimension reduction
• Recover the input data
• Learns an encoding of the inputs so as to recover the original input from the encodings as well as possible

​

​

49

Image from https://lilianweng.github.io/

Variational Autoencoder

• Encoder maps each point into a distribution within the latent space

50

1) Reconstruction loss

2) Regularization

Adversarial Autoencoder

51

Adversarial Autoencoder

52

Incorporating Label Information

53

Label Information

Disentangled Latent Representation

54

Disentangled Latent Representation

55

Disentangled Latent Representation

56

Disentangled Latent Representation

57

CycleGAN

58

Image-to-Image Translation

• Change the style of image to another style
• Monet to photos
• Zebras to horses
• Summer to winter

​

59

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,

Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, ICCV 2017

Paired vs. Unpaired Datasets

• Limitation of paired datasets
• Impossible to collect paired datasets in most of the cases.

60

Paired vs. Unpaired Datasets

• Limitation of paired datasets
• Impossible to collect paired datasets in most of the cases.

61

Start from GAN

• Given an image X (Horse), transform it into the target image Y (Zebra)

62

Start from GAN

• Given an image X (Horse), transform it into the target image Y (Zebra)
• Do not preserve the content of the input image
• Discriminator only distinguishes whether it is a horse or a zebra, not the content itself.

63

Start from GAN

• Given an image X (Horse), transform it into the target image Y (Zebra)

64

Adversarial Loss

(real or fake)

CycleGAN

65

Example of CycleGAN

66

Grain Boundary

• Background
• Grain boundaries play an important role in governing mechanical and physical properties of polycrystalline materials

​

• Research objective
• Propose an unsupervised learning-based grain boundary detection
• Only requires virtual micrographs

​

67

J. Na+, J. Lee+, et al., Label-free Grain Segmentation for Optical Microscopy Images via Unsupervised Image-to-Image Translation, Materials Characterization

Methodology

• The appearance of real and virtual micrographs differs, but their underlying grain structure is semantically equivalent

​

​

​

​

​

​

​

​

​

​

• How can we transfer appearance information from the virtual domain to the real domain?
• Image-to-image translation

​

68

Virtual microstructure

Real microstructure

Not paired

CycleGAN

• The target appearance is enforced by an adversarial loss, while the grain structure is preserved by a reconstruction loss

69

Real-to-Virtual Microstructure Translation

Real microstructure

Adversarial loss

Reconstruction loss

Virtual microstructure

Results: Low Carbon Steel and Magnesium

70

Magnesium

Low carbon steel

Input

Ground truth

Ours