WGAN-GP

Alex Lau

Hong Kong Machine Learning Meetup

17/07/2019

Table of Content

• What is GAN
• Pain-points when training GAN
• What is WGAN-GP
• How WGAN-GP can help
• WGAN-GP v.s. GAN
• WGAN-GP Applications

What is GAN?

• Proposed by Ian Goodfellow in 2014
• Generative Model
• Input training data set (aka real data)
• Output data highly resemble real data
• Statistically, it is trying to approximate real data distribution

​

What is GAN?

How do we measure distribution similarity?

• Important metrics!
• Real data distribution v.s. Fake data distribution
• GAN model measure distribution similarity by JS (Jensen-Shannon) divergence
• computed by KL (Kullback Leibler) divergence
• Both measures are commonly used in machine learning

Cost Function of GAN

D stands for Discriminator network

G stands for Generator network

First term considers sample from real data

​

1. Sample from real data, (x1, x2 ….xn)
2. Input sample into discriminator
3. Collect discriminator outputs, D(x1), D(x2)... D(xn)
4. Take log on the outputs, log(D(x1)) … log(D(xn))
5. Take average of the collections

Second term consider sample from fake data

​

1. Sample prior noise
2. Feed to generator to get fake data
3. Feed fake data into discriminator
4. ...

Cost Function of GAN

D stands for Discriminator network

G stands for Generator network

D(x) should output ~1 when input are real data

​

For sample from real data, discriminator aims to higher the chance of classifying them as real data

D(x) should output ~0 when input are generated data

​

For fake data, discriminator aims to higher the chance of classifying them as fake data

Cost Function of GAN

D stands for Discriminator network

G stands for Generator network

G(.) should fool D(.) so as to output ~1

​

Generator aims to produce data so as to higher the chance of discriminator classifying them as real data

Cost Function of GAN

D stands for Discriminator network

G stands for Generator network

Optimizing the cost function is equivalent to minimizing JS divergence

Pain-point when training GAN

Cost Function not Smooth

GAN uses JS divergence (closely related to KL divergence) as measurement of distribution similarity

Both JS divergence and KL divergence gives non-smooth surface when two distribution has a disjoint support

Vanishing Gradient

If we apply KL/ JS divergence on the cost function, it easily leads to vanishing gradient

​

(Vanishing gradient is an issue when optimise comes to a flat surface...

It makes neural network learn almost nothing for each iteration)

source

Mode Collapse

Generated data stuck in a small space with extremely low variety

Equilibrium is reached but the model is bad...

source

Lack of Indicative Metrics for Training Performance

Lack of Indicative Metrics for Training Performance

What is WGAN-GP?

• Variant of GAN proposed in 2017, come after WGAN (both 1000+ citation)
• Wasserstein GAN with Gradient Penalty
• What’s new?
• New cost function with Wasserstein distance
• Enforce optimization constraint with gradient penalty

Introducing Wasserstein Distance

• Another measurement on distribution similarity
• It even works well when two distributions have disjoint support (overcome the difficulty faced by JS divergence)

​

How Wasserstein Distance Measure Similarity?

• View as an optimal transport problem
• Blue: original configuration
• New: target configuration
• How much effort at minimum I need to take to make original configuration into a target configuration
• Factors
• Transport distance
• Mass to be transport

​

v.s.

How Wasserstein Distance Measure Similarity?

Wasserstein Distance

= ⅓ * (5-1) + ⅓ * (6-2) + ⅓ * (7-3) = 4

​

However, WD is Hard to Compute ….

Original form is VERY hard to compute

With math magic, it can be expressed in another form that is easier to compute (under a certain constraint…)

Apply gradient penalty to enforce the constraint (help avoid gradient exploding/ vanishing)

(Where D(x) is 1-Lipschitz function)

I.e. differentiable + derivative norm <= 1

How WGAN-GP Can Help?

Smoothen the Function to be Optimised

​

• Way smoother
• Less flat surface

​

Avoid Mode Collapse

• Cost function with wasserstein distance is
• More smooth
• Less flat surface
• Optimisation works better
• Generator can learn effectively

Loss Curve More indicative of Training Performance

• Loss curve can be a reliable indicator to tell if the training performance is going well
• It goes well when generator’s loss and discriminator’s loss convergence and close to each other

Loss Curve More indicative of Training Performance

WGANGP v.s. GAN

GAN v.s. WGAN-GP in Algorithm

WGAN-GP

GAN

GAN v.s. WGAN-GP in Algorithm

WGAN-GP

GAN

What are somethings in common?

High-level structure

GAN v.s. WGAN-GP in Algorithm

WGAN-GP

GAN

What are somethings in common?

Generator input

GAN v.s. WGAN-GP in Algorithm

WGAN-GP

GAN

What are somethings in common?

Number of training for discriminator

GAN v.s. WGAN-GP in Algorithm

WGAN-GP

GAN

What are the differences?

Number of training for discriminator

GAN v.s. WGAN-GP in Algorithm

WGAN-GP

GAN

What are the differences?

Optimizer

1. Minibatch
2. Take into account momentum (i.e. preceding gradient direction)
3. Take into account diminishing learning rate

• Minibatch

​

GAN v.s. WGAN-GP in Algorithm

WGAN-GP

GAN

What are the differences?

Discriminator output

Discriminator output probability (0 - 1)

Sigmoid as activation function

Critic output any real number

Sigmoid is removed

GAN v.s. WGAN-GP in Algorithm

WGAN-GP

GAN

What are the differences?

Cost Function

Applications of WGAN-GP

Image Inpainting (Yu et al, 2018)

Image Inpainting (Yu et al, 2018)

Recovered by WGAN-GP

Recovered by GAN

Image Inpainting (Yu et al, 2018)

You can try their demo: http://jiahuiyu.com/deepfill/

​

Image Coloring

DeOldify: https://github.com/jantic/DeOldify

​

Reference

1. [Paper] Improved Training of Wasserstein GANs
2. [Paper] Wasserstein GAN
3. [Paper] Generative adversarial nets
4. [Paper] Towards principled methods for training Generative Adversarial Networks
5. [Paper] Generative Image Inpainting with Contextual Attention
6. [Article] From GAN to WGAN
7. [Article] Approximating Wasserstein distances with PyTorch
8. [Article] GAN — Wasserstein GAN & WGAN-GP
9. [Youtube] Explaining the Kullback-Liebler divergence through secret codes
10. [Github] My github repo

Open to Project Collaboration

Proposed Projects

• Counting densely crowd
• DeepFake

​

​

​

​

​

Alex Lau

alexlauwh@gmail.com

​