1 of 25

Online Adaptive Integration of Observation and Inpainting for Diminished Reality with �Online Surface Reconstruction

Taiki Kato1, Naoya Isoyama1, Norihiko Kawai2, �Hideaki Uchiyama1, Nobuchika Sakata3, Kiyokawa Kiyoshi1

​

1Nara Institute of Science and Technology

2Osaka Institute of Technology

3Ryukoku University

2 of 25

Diminished reality

Erase obstacles visually

Shohei Mori. A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects. IPSJ. 2017

Input

Augmented reality

Diminished reality

2

2

3 of 25

Existing methods

  • Observation-based DR
    • Offline methods
    • Online methods

​

  • Inpainting-based DR
    • Patch-based methods
    • Learning-based methods

3

3

4 of 25

Offline-observation-based DR

Use background information that was observed offline

Obstacle

Background

Offline observation

Offline observation

User’s view

4

4

5 of 25

Online-observation-based DR

Use background that is observed from other viewpoint(s) in real-time

Siim Meerits and Hideo Saito. Real-time diminished reality for dynamic scenes. In IEEE International Symposium on Mixed and Augmented Reality Workshops, pp. 53–59, 2015

5

5

6 of 25

Inpainting-based DR

Infer background texture and shape from its surroundings

N. Kawai, T. Sato, and N. Yokoya:"Diminished reality based on image inpainting considering background geometry",IEEE Transactions on Visualization and Computer Graphics, vol. 22, no. 3, pp. 1236-1247, 2016

6

6

7 of 25

Comparison of existing methods

​

Texture

Completeness

Dynamic Scene

Offline observation

Real

Imcomplete�for unseen area

No

Online observation

Real

Imcomplete�for unseen area

Yes

Inpainting

Synthesized

Complete

Yes

Combine online observation and inpainting methods

7

7

8 of 25

Proposed method

Integrate online observation and inpainting adaptively

Obstacle

Background

User’s view

Online�observation

Inpainting

User’s view

Moving

8

8

9 of 25

Process flow

Input RGB-D image

Compute ROI�with previous pose

Output DR result

Fill ROI with�reconstructed surface

Fill remaining ROI with patch-based inpainting

Perform Kinect fusion

ROI computation

ROI completion

9

9

10 of 25

Process flow

Input RGB-D image

Compute ROI�with previous pose

Output DR result

Fill ROI with�reconstructed surface

Fill remaining ROI with patch-based inpainting

Perform Kinect fusion

ROI computation

ROI completion

Projected 3D ROI

10

10

11 of 25

Process flow

Input RGB-D image

Compute ROI�with previous pose

Output DR result

Fill ROI with�reconstructed surface

Fill remaining ROI with patch-based inpainting

Perform Kinect fusion

ROI computation

ROI completion

Fill ROI with

reconstructed surface

Remaining ROI

11

11

12 of 25

Process flow

Input RGB-D image

Compute ROI�with previous pose

Output DR result

Fill ROI with�reconstructed surface

Fill remaining ROI with patch-based inpainting

Perform Kinect fusion

ROI computation

ROI completion

Fill ROI with

inpainting

Fill ROI with inpainting

12

12

13 of 25

Fill ROI with reconstructed surface

13

13

14 of 25

Fill ROI with reconstructed surface & inpainting

14

14

15 of 25

Results for a scene with flat surfaces

Original

Inpainting only

Observation only

Proposed

15

15

16 of 25

Result for a scene with general 3D structure

Original

Inpainting only

Observation only

Proposed

16

16

17 of 25

Conclusion

We proposed a DR method combining online observation and inpainting methods

  • Use online observation once observed
  • Use inpainting for invisible/not observed regions

​

Future work

  • Real time processing
  • Color adjustment

17

17

18 of 25

19 of 25

Related work

Integrate pre-captured observation and inpainting

​

Kunert, Christian, Tobias Schwandt, and Wolfgang Broll. "An efficient diminished reality approach using real-time surface reconstruction." 2019 International Conference on Cyberworlds (CW). IEEE, 2019.

19

19

20 of 25

ROI selection

Generate 3D cuboid manually

20

20

21 of 25

21

21

22 of 25

Process flow

Input RGB-D image

Compute ROI�with previous pose

Output DR result

Fill ROI with�reconstructed surface

Fill remaining ROI with patch-based inpainting

Perform Kinect fusion

ROI computation

ROI completion

22

22

23 of 25

Process flow

Input RGB-D image

Compute ROI�with previous pose

Output DR result

Fill ROI with�reconstructed surface

Fill remaining ROI with patch-based inpainting

Perform Kinect fusion

ROI computation

ROI completion

Projected 3D ROI

23

23

24 of 25

Process flow

Input RGB-D image

Compute ROI�with previous pose

Output DR result

Fill ROI with�reconstructed surface

Fill remaining ROI with patch-based inpainting

Perform Kinect fusion

ROI computation

ROI completion

Fill ROI with

reconstructed surface

Remaining ROI

24

24

25 of 25

Process flow

Input RGB-D image

Compute ROI�with previous pose

Output DR result

Fill ROI with�reconstructed surface

Fill remaining ROI with patch-based inpainting

Perform Kinect fusion

ROI computation

ROI completion

Fill ROI with

inpainting

25

25