Sketch-Guided Text-to-Image Generation

Final Report - Jul 27, by Elliott Wu

Mentor: Hyungjoo Cho

Advisor: Yongyi Lu, Yu-Wing Tai,

Chi-Keung Tang

sText2Image

male, long face, smile with mouth closed, double eyelids, five o'clock shadow…

sText2Image

male, long face, smile with mouth closed, double eyelids, five o'clock shadow…

sText2Image

male, long face, smile with mouth closed, double eyelids, five o'clock shadow…

TEXT

SKETCH

IMAGE

Text2Image

• Generative Adversarial Text-to-Image Synthesis (Reed et al, ICML 2016)
• Stack-GAN: Text to Photo-realistic Image Synthesis (Zhang et al, arxiv)
• …

* retrieved from Stack-GAN

Sketch?

* Jun-yan Zhu, Generative Visual Manipulation on the Natural Image Manifold, ECCV 2016

Sketch?

* collected from volunteers

Sketch?

?

?

?

Sketch?

Joint Representation

male, long face, smile with mouth closed, double eyelids, five o'clock shadow…

TEXT

SKETCH

IMAGE

male, long face, smile with mouth closed, double eyelids, five o'clock shadow…

Joint Space

TEXT

SKETCH

IMAGE

Network Architecture - Training

4x8

8x16

16x32

32x64

32x64

16x32

8x16

4x8

G(z, t)

y

Generator:

Discriminator:

z

linear

replicate

512

fake/wrong

real

replicate

4x8

Network Architecture - Testing

​

z

L_contextual :

Discriminator

L_perceptual :

text

G(z, t)

Generator

Input:

text

sketch

text

Output:

backprop

Data Preparation - Image

Face (CelebA)

Bird (CUB)

Flower (Oxford)

11k

202k

8k

Data Preparation - Image

40 attributes:

1 : "5_o_Clock_Shadow"

2 : "Big_Lips"

3 : "Big_Nose"

4 : "Chubby"

5 : "Double_Chin"

6 : "Eyeglasses"

7 : "Goatee"

8 : "Heavy_Makeup"

9 : "High_Cheekbones"

10 : "Male"

11 : Mouth_Slightly_Open"

12 : "Mustache"

...

For both bird and flower datasets, 10 captions per image provided by char-CNN-RNN (Reed et al, CVPR 2016):

attribute vector OR

text embedding

Face (CelebA)

Bird (CUB)

Flower (Oxford)

Data Preparation - Synthesized Sketch�

Edge detection:

• XDog (Winnemöller et al, Computer & Graphics 2012)
• Photoshop photocopy effect

Simplification (synthesized sketches):

• Sketch simplification (Simo-Serra & Iizuka, SIGGRAPH 2016)

Image

Edge

Simplified

Data Preparation - Freehand Sketch�

* collected from volunteers

Experiments - Face

male, long face, smile with mouth closed, double eyelids, five o'clock shadow…

ATTRIBUTES

SKETCH

IMAGE

Experiments - Failures

Experiments - Failures

Experiments - Finally…

Experiments - Face

1

Attributes Match Sketch

2

Attributes Mismatch Sketch

3

Freehand Sketch

Experiments - Match (Mustache)

Experiments - Match (Eyeglasses)

Experiments - Match (Lipstick)

Female, Heavy_Makeup, Wearing_Lipstick

Experiments - Mismatch

Female, Heavy_Makeup, Smiling, Wearing_Lipstick

Experiments - Mismatch

Male, Chubby, Double_Chin, High_Cheekbones, Mouth_Open

Male

Experiments - Mismatch

Female, High_Cheekbones, Smiling, Wearing_Lipstick, No_Eyeglasses

Female, Heavy_Makeup, High_Cheekbones, Pointy_Nose, Smiling, Wearing_Lipstick, No_Eyeglasses

Experiments - Freehand

Experiments - Freehand

Experiments - Failure Cases (Eyeglasses)

Timeline

Before Mar

Ideation

Mar

Submitted to ICCV on Sketch-to-Image

Jul

Extension on Sketch-

Guided Text-to-Image

Aug

Run experiments on bird and flower datasets

Sept - Oct

Refine results and paper write-up

Nov

Submit to CVPR

THANK YOU!

Shangzhe (Elliott) Wu

Email: swuai@ust.hk

GitHub: elliottwu