1 of 33

1

Lecture 16:

Generative Adversarial Networks

​

​

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

​

​

​

Lecture 16 -

Oct 27, 2020

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

2 of 33

Generative Adversarial Networks

2

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

​

​

Problem: Want to sample from complex, high-dimensional training distribution. No direct way to do this!

​

Solution: Sample from a simple distribution, e.g. random noise. Learn transformation to training distribution.

​

​

Q: What can we use to represent this complex transformation?

​

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

3 of 33

Generative Adversarial Networks

3

​

​

Problem: Want to sample from complex, high-dimensional training distribution. No direct way to do this!

​

Solution: Sample from a simple distribution, e.g. random noise. Learn transformation to training distribution.

​

​

z

Input: Random noise

​

Generator Network

Output: Sample from training distribution

​

Q: What can we use to represent this complex transformation?

​

A: A neural network!

​

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

4 of 33

Training GANs: Two-player game

4

Generator network: try to fool the discriminator by generating real-looking images

Discriminator network: try to distinguish between real and fake images

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

5 of 33

Training GANs: Two-player game

5

Generator network: try to fool the discriminator by generating real-looking images

Discriminator network: try to distinguish between real and fake images

z

Random noise

Generator Network

Discriminator Network

Fake Images

(from generator)

Real Images

(from training set)

Real or Fake

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Fake and real images copyright Emily Denton et al. 2015. Reproduced with permission.

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

6 of 33

Training GANs: Two-player game

6

Generator network: try to fool the discriminator by generating real-looking images

Discriminator network: try to distinguish between real and fake images

​

​

​

​

Train jointly in minimax game

​

Minimax objective function:

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

7 of 33

Minimax objectives: Saddle points

7

Train jointly in minimax game

​

Minimax objective function:

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

8 of 33

Training GANs: Two-player game

8

Generator network: try to fool the discriminator by generating real-looking images

Discriminator network: try to distinguish between real and fake images

​

​

​

​

Train jointly in minimax game

​

Minimax objective function:

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

9 of 33

Training GANs: Two-player game

9

Generator network: try to fool the discriminator by generating real-looking images

Discriminator network: try to distinguish between real and fake images

​

​

​

​

Train jointly in minimax game

​

Minimax objective function:

Discriminator output for real data x

Discriminator output for generated fake data G(z)

Discriminator outputs likelihood in (0,1) of real image

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Should be 1 for real data x�

Should be 0 for fake data z.

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

10 of 33

Training GANs: Two-player game

10

Generator network: try to fool the discriminator by generating real-looking images

Discriminator network: try to distinguish between real and fake images

​

​

​

​

Train jointly in minimax game

​

Minimax objective function:

Discriminator output for real data x

Discriminator output for generated fake data G(z)

Discriminator outputs likelihood in (0,1) of real image

  • Discriminator (θd) wants to maximize objective such that D(x) is close to 1 (real) and D(G(z)) is close to 0 (fake)
  • Generator (θg) wants to minimize objective such that D(G(z)) is close to 1 (discriminator is fooled into thinking generated G(z) is real)

​

​

​

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

11 of 33

Training GANs: Two-player game

11

Minimax objective function:

Alternate between:

  1. Gradient ascent on discriminator

​

​

​

2. Gradient descent on generator

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

12 of 33

Training GANs: Two-player game

12

Minimax objective function:

Alternate between:

  • Gradient ascent on discriminator

​

​

​

2. Gradient descent on generator

In practice, optimizing this generator objective does not work well!

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

When sample is likely fake, want to learn from it to improve generator. But gradient in this region is relatively flat!

Gradient signal dominated by region where sample is already good

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

13 of 33

Training GANs: Two-player game

13

Minimax objective function:

Alternate between:

  • Gradient ascent on discriminator

​

​

​

2. Instead: Gradient ascent on generator, different objective

Instead of minimizing likelihood of discriminator being correct, now maximize likelihood of discriminator being wrong.

Same objective of fooling discriminator, but now higher gradient signal for bad samples => works much better! Standard in practice.

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

High gradient signal

Low gradient signal

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

14 of 33

Training GANs: Two-player game

14

Minimax objective function:

Alternate between:

  • Gradient ascent on discriminator

​

​

​

2. Instead: Gradient ascent on generator, different objective

Instead of minimizing likelihood of discriminator being correct, now maximize likelihood of discriminator being wrong.

Same objective of fooling discriminator, but now higher gradient signal for bad samples => works much better! Standard in practice.

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

High gradient signal

Low gradient signal

Aside: Jointly training two networks is challenging, can be unstable. Choosing objectives with better loss landscapes helps training, is an active area of research.

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

15 of 33

Training GANs: Two-player game

15

Putting it together: GAN training algorithm

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

16 of 33

Training GANs: Two-player game

16

Putting it together: GAN training algorithm

Some find k=1 more stable, others use k > 1, no best rule.

​

Recent work (e.g. Wasserstein GAN) alleviates this problem, better stability!

​

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

17 of 33

Training GANs: Two-player game

17

Generator network: try to fool the discriminator by generating real-looking images

Discriminator network: try to distinguish between real and fake images

z

Random noise

Generator Network

Discriminator Network

Fake Images

(from generator)

Real Images

(from training set)

Real or Fake

After training, use generator network to generate new images

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Fake and real images copyright Emily Denton et al. 2015. Reproduced with permission.

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

18 of 33

Generative Adversarial Nets

18

Nearest neighbor from training set

Generated samples

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Figures copyright Ian Goodfellow et al., 2014. Reproduced with permission.

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

19 of 33

Generative Adversarial Nets

19

Nearest neighbor from training set

Generated samples (CIFAR-10)

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Figures copyright Ian Goodfellow et al., 2014. Reproduced with permission.

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

20 of 33

Generative Adversarial Nets: Convolutional Architectures

20

Radford et al, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”, ICLR 2016

Generator is an upsampling network with fractionally-strided convolutions

Discriminator is a convolutional network

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

21 of 33

Generative Adversarial Nets: Convolutional Architectures

21

Radford et al, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”, ICLR 2016

Generator

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

22 of 33

Generative Adversarial Nets: Convolutional Architectures

22

Radford et al,� ICLR 2016

Samples from the model look much better!

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

23 of 33

Generative Adversarial Nets: Convolutional Architectures

23

Radford et al,� ICLR 2016

Interpolating between random �points in latent space

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

24 of 33

Generative Adversarial Nets: Interpretable Vector Math

24

Smiling woman

Neutral woman

Neutral man

Samples �from the �model

Radford et al, ICLR 2016

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

25 of 33

Generative Adversarial Nets: Interpretable Vector Math

25

Smiling woman

Neutral woman

Neutral man

Samples �from the �model

Average Z

vectors, do arithmetic

Radford et al, ICLR 2016

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

26 of 33

Generative Adversarial Nets: Interpretable Vector Math

26

Smiling woman

Neutral woman

Neutral man

Smiling Man

Samples �from the �model

Average Z

vectors, do arithmetic

Radford et al, ICLR 2016

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

27 of 33

Generative Adversarial Nets: Interpretable Vector Math

27

Radford et al, �ICLR 2016

Glasses man

No glasses man

No glasses woman

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

28 of 33

Generative Adversarial Nets: Interpretable Vector Math

28

Glasses man

No glasses man

No glasses woman

Woman with glasses

Radford et al, �ICLR 2016

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

29 of 33

“The GAN Zoo”

2017: Explosion of GANs

“The GAN Zoo”

29

https://github.com/hindupuravinash/the-gan-zoo

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

30 of 33

2017: Explosion of GANs

“The GAN Zoo”

30

https://github.com/hindupuravinash/the-gan-zoo

See also: https://github.com/soumith/ganhacks for tips and tricks for trainings GANs

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

31 of 33

2017: Explosion of GANs

31

Better training and generation

LSGAN, Zhu 2017.

Wasserstein GAN, Arjovsky 2017.

Improved Wasserstein GAN, Gulrajani 2017.

Progressive GAN, Karras 2018.

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

32 of 33

2017: Explosion of GANs

32

CycleGAN. Zhu et al. 2017.

Source->Target domain transfer

Many GAN applications

Pix2pix. Isola 2017. Many examples at https://phillipi.github.io/pix2pix/

Reed et al. 2017.

Text -> Image Synthesis

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019

33 of 33

GANs

33

​

Take game-theoretic approach: learn to generate from training distribution through 2-player game

​

Pros:

  • Beautiful, state-of-the-art samples!

​

Cons:

  • Trickier / more unstable to train
  • Can’t solve inference queries such as p(x), p(z|x)

​

Active areas of research:

  • Better loss functions, more stable training (Wasserstein GAN, LSGAN, many others)
  • Conditional GANs, GANs for all kinds of applications

Erik Learned-Miller and TAs�Adapted from slides of Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 16 -

Nov 21, 2019