1 of 46

Adversarial Infill

2 of 46

Image Infilling

Classical infilling

Damage restoration
Image editing
Noise removal

3 of 46

Image Infilling

Semantic infilling

Large region completion
Painting Restoration
Facial reconstruction

4 of 46

Approaches

Local

Exemplar-based inpainting
Seam carving
Structure propagation

NIKHIL

-So all inpainting methods can be bucketed into one of two distinct groups.

-The first is local inpainting methods. These are inpainting approaches that only use the information present in the input photo to infill a missing region.

-These methods are usually math intensive, and borrow ideas from flow dynamics and image editing to methodically fill in a missing region using only textures and information in the given photo

-Exemplar-based inpainting or EBI, uses a iterative pixel filling algorithm to infill pixel blocks with similar textures from the surrounding pixels

-seam carving carves the image to find the intersection of 8 different lines that best infills the missing area

-structure propogation is used for image removal, and requires the user to specify anchor points that the algorithm uses to replace a region

5 of 46

Approaches

Non-Local

Multi-output estimators
PatchMatch
GANs!

6 of 46

Project Goals

Explore local and non-local methods for semantic infilling
Successfully infill masked 64x64 CelebA images with various masks
Implement and improve on GAN infilling method proposed by Yeh et al.

7 of 46

Exemplar-Based Inpainting

8 of 46

Exemplar-Based Inpainting

9 of 46

Deep Convolutional GANs

10 of 46

GAN Outputs

11 of 46

Approach

Input

Input + Mask

Generate

Infill

12 of 46

Model + Training

Generator: 4 conv layers
Discriminator: 4 conv layers
Trained on ~200k celebA images
20 epochs ~ 8 hours

13 of 46

Infilling

Real or Fake

G

D

z

G(z)

14 of 46

Contextual Loss

Take advantage of remaining input data
Why not pure L1 or L2 norm?

G

z

G(z)

15 of 46

Perceptual Loss

Want to penalize unrealistic images
Can we reuse the discriminator?

Real or Fake

G

D

z

G(z)

16 of 46

Blending Loss

Smooth infilling boundary
Penalize boundary pixel differences

G

z

G(z)

17 of 46

Results

Input

Input + Mask

Generate

Infill

18 of 46

Infill - 1/5

19 of 46

Infill - 2/5

20 of 46

Infill - 3/5

21 of 46

Infill - 4/5

22 of 46

Infill - 5/5

23 of 46

Center Masks

Original

Masked

Result (no blending loss)

Result (with blending loss)

24 of 46

Random Masks

25 of 46

Beyond CelebA

Input

Input + Mask

Generate

Infill

26 of 46

Ricky Infill - 1/6

27 of 46

Ricky Infill - 2/6

28 of 46

Ricky Infill - 3/6

29 of 46

Ricky Infill - 4/6

30 of 46

Ricky Infill - 5/6

31 of 46

Ricky Infill - 6/6

32 of 46

Final Result

33 of 46

GIF

34 of 46

Next Steps

Poisson Blending
Conditional GAN
Expanding beyond CelebA/faces

35 of 46

Thank you!

James Wexler
Isaac Kontomah
6.s198 Staff!
Yeh et. al, Brandon Amos

36 of 46

37 of 46

38 of 46

JUNK PICTURES

39 of 46

40 of 46

41 of 46

42 of 46

43 of 46

44 of 46

Results

Before

45 of 46

Results

Masked

46 of 46

Multi-Output Estimators

math

Extra trees
k nearest neighbors