Dongliang Cao, Trung Nguyen

1

03/16/2021

Unsupervised Learning of Occlusion Map

Video from [Zhao, Shen, etc.]MaskFlowNet

Content

1. Background
2. Motivation
3. Related works
4. Our method
5. Experiment Results
6. Conclusion

Dongliang Cao, Trung Nguyen

2

03/16/2021

Dongliang Cao, Trung Nguyen

3

03/16/2021

Background: Optical Flow Estimation

Pictures from [Butler, D. J., etc.]MPI Sintel Flow Dataset

Dongliang Cao, Trung Nguyen

4

03/16/2021

Background: Brightness Constancy Assumption

Scene point moving through image sequence

Assumption: Brightness of the point will remain the same

Dongliang Cao, Trung Nguyen

5

03/16/2021

Background: Brightness Constancy Assumption

Dongliang Cao, Trung Nguyen

6

03/16/2021

Motivation: Occlusion invalids the assumption

Pictures from [Butler, D. J., etc.]MPI Sintel Flow Dataset

Methods based on brightness constancy assumption fail at occluded regions.

Occlusion estimation is necessary.

Dongliang Cao, Trung Nguyen

7

03/16/2021

Picture from [Meister, Hur, Roth] UnFlow

Related Work: Occlusion estimation

Picture from [Wang, Yang, etc] OAFlow

Occlusion estimation based on the predicted forward and backward optical flow.

1. The quality of predicted optical flow affects the occlusion estimation.
2. The network needs to predict the backward optical flow.

Dongliang Cao, Trung Nguyen

8

03/16/2021

Our method: Unsupervised learning of occlusion map

Our framework with two stages

Dongliang Cao, Trung Nguyen

9

03/16/2021

Our method: Network architecture

Network architecture for optical flow estimation

Flow prediction at one pyramid level

Pictures from [Jonschkowski, Stone, etc] UFlow

Network architecture for occlusion map estimation

Dongliang Cao, Trung Nguyen

10

03/16/2021

Our method: Network architecture

Network architecture for image inpainting

Gated convolution

Pictures from [Yu, Lin, etc] Gated Convolution

Dongliang Cao, Trung Nguyen

11

03/16/2021

Our method: Loss terms

Dongliang Cao, Trung Nguyen

12

03/16/2021

Experiment results

1. Unsupervised training of optical flow

​

• Training of inpainting network

​

• Unsupervised training of occlusion map

​

Dongliang Cao, Trung Nguyen

13

03/16/2021

Unsupervised training of optical flow

Advantage:

1. The photometric error only takes the non-occluded areas into account.
2. There is a more clear distinguish between the photometric error in occluded areas and in non-occluded areas.

Dongliang Cao, Trung Nguyen

14

03/16/2021

Unsupervised training of optical flow

Dataset:

MPISintel Clean

Result

​

Unsupervised training of optical flow

Quantitative results over the MPI-Sintel dataset.

Average EPE: 0.39

Average EPE: 4.70

Next step: train the optical flow network over the FlyingChair dataset in this unsupervised way.

Training of inpainting network

Train dataset

FlyingChairs2

_44k images

_with ground truth occlusion map

_convenient for later joint training with occlusion map

Metric

_Structural Similarity Index Measure(SSIM)

_Frechet Inception Distance(FID)

_Peak Signal to Noise Ratio(PSNR)

16

Training of inpainting network

Generate mask randomly so that inpainting network can learn effectively the structure of images.

17

Rectangular mask

Random mask

Training of inpainting network

Loss function

_ L1: simplest function

_ VGG : take into account deep spatial structure

_ Adversarial : generate realistic image

18

Training of inpainting network

_Choose Gated Convolution with VGG loss and use it as pretrained model for next training phase

19

L1 loss

VGG+L1 loss

VGG loss (best model)

Adversarial loss

Unsupervised training of occlusion map

21

Unsupervised training of occlusion map

1. Fix pretrained inpainting network, overfit occlusion network to single pair of image
2. Jointly train inpainting and occlusion network from scratch
3. Fix pretrained inpainting network, warm-up occlusion to 33 epochs, then train both of them together

22

Overfit occlusion network

23

Ground truth occlusion mask

Predicted hard occlusion mask

Using hard occlusion mask

Using soft

occlusion mask

Using soft

occlusion mask, add smoothness loss

Unsupervised training of occlusion map

Jointly train inpainting and occlusion network from scratch

24

Photometric loss goes down to 0, reconstruction loss increases significantly

=> the network collapse, all pixels are predicted as occluded

Unsupervised training of occlusion map

Increase the weight for reconstruction loss to 2 (previous setting: 1)

25

Reconstruction loss goes down to 0, photometric loss increases significantly

=> the network collapse, all pixels are predicted as non-occluded

Analysis

26

1st image

2nd image

warped image

masked image

reconstructed image

Some thoughts:

1. Training both inpainting network and occlusion net is very tricky, it is important that the balance point is found
2. The training curve goes down to 0 abruptly, suggesting that the learning rate is too high
3. As the inpainting network is trained from scratch with occlusion network, the quality of reconstructed image is very low at the beginning. If the occlusion net collapse quickly, all pixels are occluded

⇒ inpainting network cannot learn anything helpful

⇒ set low learning rate

Unsupervised training of occlusion map

Decrease learning rate to 5e-6

27

Photometric loss still goes down to 0, but with a smoother curve, reconstruction loss increases significantly

=> the network collapse, all pixels are predicted as occluded

Analysis

28

1st image

2nd image

warped image

masked image

reconstructed image

The network still collapses, but the inpainting net is slightly better

Unsupervised training of occlusion map

Finetune pretrained inpainting network with occlusion net

_Fix pretrained inpainting network, warm-up occlusion to 33 epochs, then train both of them together

29

Green curve: warm-up stage

Blue curve: finetune stage

Unsupervised training of occlusion map

30

Dongliang Cao, Trung Nguyen

31

03/16/2021

Conclusion

_The general framework base on the balance between the photometric error in the non-occluded area and the reconstruction error in the occluded area, which is possible .

_ The joint training between occlusion and inpainting is tricky, as we dont know where the balance point is

_ Training both inpainting and occlusion network from scratch is very hard, since a very weak inpainting net is one big disadvantage

_ It is better to have a pretrained inpainting network trained together with occlusion net

​

Dongliang Cao, Trung Nguyen

32

03/16/2021

Thanks for your attention!

Question?

33