1 of 45

Generative Models and Style Transfer

Mengdi Fan, Xinyu Zhou

2 of 45

Outlines

Motivation
GAN Generative Adversarial Nets
CycleGAN Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
CUT Contrastive Learning for Unpaired Image-to-Image Translation
MUNIT Multimodal Unsupervised Image-to-Image Translation

Motivation
GAN Generative Adversarial Nets
CycleGAN Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
CUT Contrastive Learning for Unpaired Image-to-Image Translation
MUNIT Multimodal Unsupervised Image-to-Image Translation

2

3 of 45

Autonomous Driving

Motivation

Data generating

Winter ⇄ Summer

Day ⇄ Night

https://github.com/mingyuliutw/UNIT

3

4 of 45

GAN

Generative Adversarial Nets

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio

Département d’informatique et de recherche opérationnelle

Université de Montré al

Montré al, QC H3C 3J7

NIPS 2014

arXiv:1406.2661

4

5 of 45

Generative Adversarial Nets (GAN)

Generator

Discriminator

REAL

FAKE

5

6 of 45

Generative Adversarial Nets (GAN)

Generator

Discriminator

REAL

FAKE

6

7 of 45

Training Discriminator

Generator

Discriminator

REAL

FAKE

Maximize

7

8 of 45

Training Generator

Generator

Discriminator

REAL

FAKE

Minimize

8

9 of 45

Minimax objective function

GAN

https://jonathan-hui.medium.com/gan-why-it-is-so-hard-to-train-generative-advisory-networks-819a86b3750b

Adversarial loss

9

10 of 45

Image-to-Image Translation

Generator

10

11 of 45

CycleCAN

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros

Berkeley AI Research (BAIR) laboratory, UC Berkeley

ICCV 2017

arXiv:1703.10593

11

12 of 45

Unpaired Data

CycleCAN

X

Y

Mapping function G: X → Y

12

13 of 45

Adversarial Loss

CycleCAN

Generator

Discriminator

REAL

FAKE

13

14 of 45

Formulation

CycleCAN

cycle consistent

14

15 of 45

Adversarial Loss

CycleCAN

Generator

Discriminator

REAL

FAKE

Generator

Discriminator

REAL

FAKE

15

16 of 45

Formulation

CycleCAN

16

17 of 45

Formulation

CycleCAN

Forward cycle consistency

Backward cycle consistency

17

18 of 45

Full objective

CycleCAN

18

19 of 45

CUT

Contrastive Learning for Unpaired Image-to-Image Translation

Taesung Park, Alexei A. Efros, Richard Zhang, Jun-Yan Zhu

UC Berkeley, Adobe Research

ECCV 2020

arXiv:2007.15651

19

20 of 45

Formulation

CUT

cycle consistent

20

21 of 45

Formulation

CUT

21

22 of 45

Formulation

CUT

22

23 of 45

Patchwise Contrastive Loss

CUT

23

24 of 45

Patchwise Contrastive Loss

CUT

24

25 of 45

Internal vs External Patches

CUT

25

26 of 45

Identity Loss

CUT

26

27 of 45

Full objective

Adversarial loss

CUT

PatchNCE loss

identity loss

27

28 of 45

Full objective

CUT

identity loss

28

29 of 45

Evaluation

CUT

29

30 of 45

Ablation Study

CUT

30

31 of 45

Ablation Study

CUT

31

32 of 45

Multimodal Unsupervised Image-to-Image Translation

by Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz

32

33 of 45

Image-To-Image Translation

33

34 of 45

Unimodal vs Multimodal translation

Unimodal

Multimodal

34

35 of 45

CycleGAN and UNIT

CycleGAN

UNIT

Forward Consistency:

Backward Consistency

Liu, Ming-Yu, Thomas Breuel, and Jan Kautz. "Unsupervised image-to-image translation networks." Advances in neural information processing systems 30 (2017).

35

36 of 45

UNIT: Training framework

36

37 of 45

Cycle consistency implies deterministic translations

37

38 of 45

Disentangled latent space

38

39 of 45

Loss function

39

40 of 45

Adversarial Loss

D₂ is the discriminator to distinguish the translation image and real image in X₂

40

41 of 45

Overall Training Objective

41

42 of 45

Theoretical result

42

43 of 45

Results:

43

44 of 45

Results:

44

45 of 45

Thank you!

45