Generative Adversarial Networks (GAN)

Supervised Learning

• Discriminative model

​

2

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Unsupervised Learning

• Generative model

​

3

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

= Latent space

Model Distribution vs. Data Distribution

4

Probability Distribution

5

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Probability Distribution

6

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Probability Distribution

7

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Probability Density Estimation Problem

8

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Generative Models from Lower Dimension

9

Source: Prof. Roger Grosse at U of Toronto

Deterministic Transformation (by Network)

• 1-dimensional example:

​

​

​

​

​

​

​

​

​

​

• Remember
• Network does not generate distribution, but
• It maps known distribution to target distribution

10

Source: Prof. Roger Grosse at U of Toronto

Deterministic Transformation (by Network)

• High dimensional example:

11

Source: Prof. Roger Grosse at U of Toronto

Prob. Density Function by Deep Learning

• Generative model of image

12

Source: Prof. Roger Grosse at U of Toronto

Generative Adversarial Networks (GANs)

• In generative modeling, we'd like to train a network that models a distribution, such as a distribution over images.
• GANs do not work with any explicit density function !
• Instead, take game-theoretic approach

​

​

13

Turing Test

• One way to judge the quality of the model is to sample from it.
• GANs are based on a very different idea:
• Model to produce samples which are indistinguishable from the real data, as judged by a discriminator network whose job is to tell real from fake

14

Generative Adversarial Networks (GAN)

• The idea behind Generative Adversarial Networks (GANs): train two different networks
• Generator network: try to produce realistic-looking samples
• Discriminator network: try to distinguish between real and fake data

​

• The generator network tries to fool the discriminator network

15

Autoencoder

• Dimension reduction
• Recover the input data
• Learns an encoding of the inputs so as to recover the original input from the encodings as well as possible

​

​

16

Generative Adversarial Networks (GAN)

• Analogous to Turing Test

​

17

Generated

Generator

Data Generator

Generative Adversarial Networks (GAN)

• Analogous to Turing Test

​

​

18

Intuition for GAN

19

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Discriminator Perspective (1/2)

20

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Discriminator Perspective (2/2)

21

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Generator Perspective

22

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Loss Function of Discriminator

23

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Loss Function of Generator

24

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Non-Saturating Game

25

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Non-Saturating Game

26

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Solving a MinMax Problem

27

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

GAN Implementation in TensorFlow

28

TensorFlow Implementation

29

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Generator

30

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Discriminator

31

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Combined

32

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Training: Discriminator

33

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Training: Generator

34

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

After Training

• After training, use generator network to generate new data

35

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

Generated Images

36

DCGAN

37

DCGAN (Deep Convolutional GAN)

• A DCGAN is a direct extension of the GAN, except that it explicitly uses convolutional and convolutional-transpose layers in the discriminator and generator, respectively
• (For example) simplified architecture

​

38

Alec Radford, et al., "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks," ICLR 2016

CycleGAN

39

Image-to-Image Translation

• Change the style of image to another style
• Monet to photos
• Zebras to horses
• Summer to winter

​

40

Paired vs. Unpaired Datasets

• Limitation of paired datasets
• Impossible to collect paired datasets in most of the cases.

41

Paired vs. Unpaired Datasets

• Limitation of paired datasets
• Impossible to collect paired datasets in most of the cases.

42

Start from GAN

43

Start from GAN

44

Discriminator

45

CycleGAN

46

CycleGAN

47

Example of CycleGAN

48

Grain Boundary

• Background
• Grain boundaries play an important role in governing mechanical and physical properties of polycrystalline materials

​

• Research objective
• Propose an unsupervised learning-based grain boundary detection
• Only requires virtual micrographs

​

49

J. Na+, J. Lee+, et al., Label-free Grain Segmentation for Optical Microscopy Images via Unsupervised Image-to-Image Translation, Materials Characterization

Methodology

• The appearance of real and virtual micrographs differs, but their underlying grain structure is semantically equivalent

​

​

​

​

​

​

​

​

​

​

• How can we transfer appearance information from the virtual domain to the real domain?
• Image-to-image translation

​

50

Virtual microstructure

Real microstructure

Not paired

CycleGAN

• The target appearance is enforced by an adversarial loss, while the grain structure is preserved by a reconstruction loss

51

Real-to-Virtual Microstructure Translation

Real microstructure

Adversarial loss

Reconstruction loss

Virtual microstructure

Results: Low Carbon Steel

52

Input

Ground Truth

CycleGAN

Homework Assignment

• To change the white-colored digit in a black background image to the black-colored digit in a white background image

​

• Assume data is not paired

53

Conditional GAN

54

Conditional GAN

• In an unconditioned generative model, there is no control on modes of the data being generated.

​

• In the Conditional GAN (CGAN), the generator learns to generate a fake sample with a specific condition or characteristics (such as a label associated with an image or more detailed tag) rather than a generic sample from unknown noise distribution.

55

Conditional GAN

• MNIST digits generated conditioned on their class label

56

Conditional GAN

• Simple modification to the original GAN framework that conditions the model on additional information for better multi-modal learning

​

​

​

• Many practical applications of GANs when we have explicit supervision available

57

Normal Distribution of MNIST

• A standard normal distribution
• This is how we would like points corresponding to MNIST digit images to be distributed in the latent space

58

Generator at GAN

59

Generator

Generator

Generator at Conditional GAN

• Feed a random point in latent space and desired number.
• Even if the same latent point is used for two different numbers, the process will work correctly since the latent space only encodes features such as stroke width or angle

60

Generator

Generator

Generator

61

Discriminator

62

Combined

63

CGAN Implementation

64

Fake MNIST Images Generated by CGAN

65

InfoGAN

66

InfoGAN

• In a standard generative model, there is no control on the features of the data being generated.

​

• In the Information Maximizing GAN (InfoGAN), the generator learns to generate a fake sample with latent codes (such as values in the range of -1 to 1) that has interpretable information of the data rather than a generic sample from unknown noise distribution.

​

• The latent code in InfoGAN learns interpretable information from the data using unsupervised learning.

67

InfoGAN

• MNIST digits generated by latent code variation per noise

68

InfoGAN

• MNIST digits generated by latent code variation per noise

69

InfoGAN

70

Generator at Conditional GAN

• Feed a random point in latent space and desired number.
• Even if the same latent point is used for two different numbers, the process will work correctly since the latent space only encodes features such as stroke width or angle

71

Generator

Generator

Generator at InfoGAN

• Feed a random point in latent space and latent code.
• Even if the same latent point is used, the process will work correctly by controlling the interpretable information in the data through latent code.

72

Structure of InfoGAN

73

Generator

74

Discriminator

75

Discriminator

76

Q Net

77

Combined

78

Combined

79

Fake MNIST Images Generated by InfoGAN

80

Example of InfoGAN

81

Generate FFT Spectrum Data

• Background
• The FFT spectrum provides important information for diagnosing equipment health
• The generated data features should be controllable to improve the utility of the generative model

​

• Research objective
• Propose a method of deep learning-based vibration signal generation
• Extract information from FFT spectra with unsupervised learning

82

Methodology

• Let the latent code learn the information about where the peak with the highest diversity occurs in the FFT spectrum.

83

Noise

Low value case

High value case

High information

Low information

Fake Vibration Signals Generated by InfoGAN

84

Normal

Abnormal

Adversarial Autoencoder

85

Limitation of Autoencoder

86

Generate Data from Controlled Latent Space

87

• Encode data into a controllable latent space
• Generate new data from the controlled space

Adversarial Autoencoder

88

Encoder

Adversarial Autoencoder

89

Encoder

Decoder

Adversarial Autoencoder

90

Encoder

Decoder

Generator

Adversarial Autoencoder

91

Encoder

Decoder

Generator

Discriminator

Prior Distribution and Latent Representation

92

Prior Distribution and Latent Representation

93

Prior Distribution and Latent Representation

94

Latent representations of�MNIST dataset

Latent representations of�MNIST dataset

Incorporating Label Information

95

Label Information

Disentangled Latent Representation

96

Disentangled Latent Representation

97

Disentangled Latent Representation

98

Disentangled Latent Representation

99

Disentangled Latent Representation

100

Disentangled Latent Representation

101

Disentangled Latent Representation

102