1 of 61

Geomagical Labs - 21 October 2022

AGENDA

  1. Overview of DR systems & solutions by Shohei Mori.

How Far Can We Go for Diminished Reality Without Neural Networks?

​

  1. Geomagical’s end-to-end solution by Prakhar Kulshreshtha & Nektarios Lianos.

RGB image inpainting for indoor scenes.

Room layout estimation for the real world.��Demo (5min) & Break (15min)�

  1. Alternative Texture Inpainting Techniques by Kiyoshi Kiyokawa.

Online Adaptive Integration of Observation and Inpainting for Diminished Reality with Online Surface Reconstruction.�

  1. End-to-end system for DR using spherical panoramas by Dimitrios Zarpalas & Vasileios Gkitsas

Diminished reality for indoor spherical panoramas.�

  1. Photometric Consistency for Diminished Reality by Salma Jiddi.

Inverse Rendering using RGBD cameras with application to Shadow Removal.

​

Closing Remarks

© 2022 Geomagical Labs, Inc. All Rights Reserved.

2 of 61

The Great Title of Your Presentation

Name of presenter / DD Month YYYY / Other info

Inverse Rendering using RGBD cameras with application to Shadow Removal.

Image assets

​

https://drive.google.com/drive/folders/1XcgvW2MB-j3sDPE18i6Ct6SERttTiCi8?usp=sharing

​

Salma Jiddi

© 2022 Geomagical Labs, Inc. All Rights Reserved.

3 of 61

Geomagical Labs - 21 October 2022

Diminished Reality: realism.

Inpainted Image

Input Image

© 2022 Geomagical Labs, Inc. All Rights Reserved.

4 of 61

Geomagical Labs - 21 October 2022

Diminished Reality: realism.

Synthesized Intrinsic Highlights

Erased Object’s Shadow

© 2022 Geomagical Labs, Inc. All Rights Reserved.

5 of 61

Geomagical Labs - 21 October 2022

Diminished Reality: realism.

Erased Object’s Shadow

Inpainted Image

Input Image

© 2022 Geomagical Labs, Inc. All Rights Reserved.

6 of 61

Geomagical Labs - 21 October 2022

Diminished Reality: realism.

Inpainted Image

Input Image

Erased Object’s Shadow

© 2022 Geomagical Labs, Inc. All Rights Reserved.

7 of 61

Geomagical Labs - 21 October 2022

Diminished Reality: realism.

Inpainted Image

Input Image

Erased Object’s Shadow

© 2022 Geomagical Labs, Inc. All Rights Reserved.

8 of 61

Geomagical Labs - 21 October 2022

Outline

  1. Introduction to Inverse Rendering��
  2. Overview of existing approaches to Inverse Rendering��
  3. End-to-end Inverse Rendering system with Application to AR & DR��
  4. Demo

© 2022 Geomagical Labs, Inc. All Rights Reserved.

9 of 61

Inverse Rendering? �Let’s define rendering first.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

10 of 61

Inverse Rendering? �Let’s define rendering first.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

11 of 61

Inverse Rendering

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

12 of 61

Inverse Rendering

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

13 of 61

Inverse Rendering

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

14 of 61

Phong Model [Phong, 1975] - why this choice?

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

15 of 61

Phong Model: ambient.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

16 of 61

Phong Model: diffuse.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

17 of 61

Phong Model: diffuse.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

18 of 61

Phong Model: specular.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

19 of 61

Phong Model: specular.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

20 of 61

Lighting Model: unknowns.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

21 of 61

Overview of existing approaches to Inverse Rendering

© 2022 Geomagical Labs, Inc. All Rights Reserved.

22 of 61

Lighting Estimation: probes.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

23 of 61

Lighting Estimation: probeless.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

24 of 61

Intrinsic Image Decomposition (IID) - �Where does it fit?

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

25 of 61

Geomagical Labs / ISMAR Workshops - W15

Intrinsic Image Decomposition (IID) - �Where does it fit?

reflectance x shading

© 2022 Geomagical Labs, Inc. All Rights Reserved.

26 of 61

Geomagical Labs / ISMAR Workshops - W15

Intrinsic Image Decomposition (IID) - �Where does it fit?

© 2022 Geomagical Labs, Inc. All Rights Reserved.

27 of 61

End-to-end system Inverse Rendering� with Application to AR & DR

© 2022 Geomagical Labs, Inc. All Rights Reserved.

28 of 61

Core Idea.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

29 of 61

Core Idea.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

30 of 61

Shadow Detection: challenge.

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

31 of 61

Inputs & Assumptions

Geomagical Labs / ISMAR Workshops - W15

[ISMAR17, ICIP18]

S. Jiddi, P. Robert, E. Marchand. Estimation of position and intensity of dynamic light sources using cast shadows on textured real surfaces. In IEEE International Conference on Image Processing, Greece, October 2018.

© 2022 Geomagical Labs, Inc. All Rights Reserved.

32 of 61

Texture and illumination separation

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

33 of 61

Geomagical Labs / ISMAR Workshops - W15

Texture and illumination separation

© 2022 Geomagical Labs, Inc. All Rights Reserved.

34 of 61

Light Sources Position Estimation

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

35 of 61

Light Sources Position Estimation

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

36 of 61

Light Sources Intensity Estimation

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

37 of 61

Results – Light Sources Position

Geomagical Labs / ISMAR Workshops - W15

[ISMAR17, ICIP18]

S. Jiddi, P. Robert, E. Marchand. Estimation of position and intensity of dynamic light sources using cast shadows on textured real surfaces. In IEEE International Conference on Image Processing, Greece, October 2018.

© 2022 Geomagical Labs, Inc. All Rights Reserved.

38 of 61

Results – Light Sources Intensity

Geomagical Labs / ISMAR Workshops - W15

[ISMAR17, ICIP18]

S. Jiddi, P. Robert, E. Marchand. Estimation of position and intensity of dynamic light sources using cast shadows on textured real surfaces. In IEEE International Conference on Image Processing, Greece, October 2018.

© 2022 Geomagical Labs, Inc. All Rights Reserved.

39 of 61

Results – Dynamic Light Sources

Geomagical Labs / ISMAR Workshops - W15

[ISMAR17, ICIP18]

S. Jiddi, P. Robert, E. Marchand. Estimation of position and intensity of dynamic light sources using cast shadows on textured real surfaces. In IEEE International Conference on Image Processing, Greece, October 2018.

© 2022 Geomagical Labs, Inc. All Rights Reserved.

40 of 61

Mixed Reality Demonstrator

Geomagical Labs / ISMAR Workshops - W15

[Best demo award, APMAR18]

S. Jiddi, P. Robert, E. Marchand, A. Laurent, M. Fradet, C. Baillard. Realistic Mixed Reality Scenarios under Dynamic Lighting and Moving Geometry. In APMAR’18, Taiwan, April 2018. Best Demo Award.

© 2022 Geomagical Labs, Inc. All Rights Reserved.

41 of 61

Limitations and Critical Scenarios

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

42 of 61

Core Idea: multiple cues.

Geomagical Labs / ISMAR Workshops - W15

[TVCG 2020]

S. Jiddi, P. Robert, E. Marchand. Using Specular Reflections and Cast Shadows to Recover Surface Reflectance and Illumination properties in Dynamic Indoor Scenes. In TVCG 2020..

© 2022 Geomagical Labs, Inc. All Rights Reserved.

43 of 61

Core Idea: multiple cues.

Geomagical Labs / ISMAR Workshops - W15

[TVCG 2020]

S. Jiddi, P. Robert, E. Marchand. Using Specular Reflections and Cast Shadows to Recover Surface Reflectance and Illumination properties in Dynamic Indoor Scenes. In TVCG 2020..

© 2022 Geomagical Labs, Inc. All Rights Reserved.

44 of 61

Illumination maps

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

45 of 61

Illumination maps

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

46 of 61

Light Sources direction

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

47 of 61

Light Sources 3D Position

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

48 of 61

Light Sources 3D Position

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

49 of 61

Specular Reflectance Estimation

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

50 of 61

Reflectance & Light Colors - Results

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

51 of 61

Light Positions - Results

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

52 of 61

Light Positions - Results

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

53 of 61

Mixed Reality Scenarios

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

54 of 61

Geomagical Labs - 21 October 2022

Where to?�

The most expensive part is shadow detection.

© 2022 Geomagical Labs, Inc. All Rights Reserved.

55 of 61

Visual cues detection using deep learning

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

56 of 61

Visual cues detection using deep learning

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

57 of 61

Visual cues detection using deep learning

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

58 of 61

Visual cues detection using deep learning

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

59 of 61

Visual cues detection using deep learning

Geomagical Labs / ISMAR Workshops - W15

© 2022 Geomagical Labs, Inc. All Rights Reserved.

60 of 61

Conclusion

Geomagical Labs / ISMAR Workshops - W15

Short Term

​

  • Photometric Consistency can help improve drastically inpainting results
    • Lighting estimation using deep learning approaches
      • Shadow detection is an interesting DL task
      • 3D parameters recovery can still be handled with classical techniques
      • Comparison between classical and DL results
    • Joint Specularity and Shadow Detection (JSSD) dataset
      • Extend to outdoor scenes
      • Availability to the research community

​

Long Term�

  • Scene analysis using advanced reflection models
    • Spot and area lights
    • Indirect lighting and inter-reflections
  • Realism assessment
    • Unified dataset to evaluate photometric registration approaches
    • User studies (accuracy vs perception)

​

© 2022 Geomagical Labs, Inc. All Rights Reserved.

61 of 61

Geomagical Labs / ISMAR Workshops - W15

​

Salma Jiddi

www.geomagical.com/

salma@geomagical.com

​

Thank you!

Geomagical Labs - 21 October 2022

© 2022 Geomagical Labs, Inc. All Rights Reserved.