1 of 35

Workshop on Inpainting Techniques�for Object Removal in Indoor Scenes

Diminished reality for indoor spherical panoramas

Vasileios Gkitsas

Dimitrios Zarpalas

2 of 35

Spherical panoramas

  • How to get 360° images?
    • 360° cameras (Ricoh, Insta360, GoPro)
    • Stitched from a set of perspective images
    • Renderings from 3D models (Stanford 2D-3D, Matterport)

​

www.polyhaven.com

3 of 35

Spherical panoramas

  • How to get 360° images?
    • Stitched from a set of perspective images
    • Renderings from 3D models (Stanford 2D-3D, Matterport)

​

​

www.cmsc426.github.io

4 of 35

Spherical panoramas

  • How to get 360° images?
    • Renderings from 3D models (Stanford 2D-3D, Matterport)

​

5 of 35

Drawbacks in using 360° images�

  • Distortion from the equirectangular projection
    • The distortion shows up in the longitude since it preserves distances between lines of latitude across the image

https://meder411.medium.com/

6 of 35

What is diminished reality?

  • Removal of objects from the perceived environment
  • Replace holes with content that preserves:
    • Photorealism
    • Structural coherency

https://atlantis-ar.eu

7 of 35

How can diminished reality benefit from 360° images?

  • The wide field of view enhances the Deep Neural Networks’ capabilities
  • Users can navigate the scene

8 of 35

How can diminished reality benefit from 360° images?

  • Perspective images provide a limited field of view

Perspective image

Panorama

www.usabilitypartners.se

9 of 35

Diminished reality targeted datasets

  • Creating datasets with full-empty configuration is challenging and laborious task
    • Only Structured3D (S3D) available

​

​

Zheng, J., Zhang, J., Li, J., Tang, R., Gao, S. and Zhou, Z., 2020, August. Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling. In 2020 European Conference on Computer Vision.

10 of 35

10

Masked image

Direct composition from empty to full

S3D Dataset adaptations

11 of 35

11

Masked image

Direct composition from empty to full

Dataset adaptations

Photometric Inconsistency

12 of 35

12

Masked image

Direct composition from empty to full

Augmentation from full to empty

Dataset adaptations

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

13 of 35

13

Masked image

Direct composition from empty to full

Augmentation from full to empty

Dataset adaptations

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

14 of 35

Important factors for developing diminished reality models

​

Structural awareness

Adaptability

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

Surrounding Context Encoder

128

256

128

128

64

3

Structure Encoder

Background Image

128x256

256x512

Masked Input Image

Input Image

Diminished

Instance Normalization

Gated Convolution (GC)

Sean Residual Block

(Dilated GC & IN) x4

Upsample & GC

 

Predicted Image

Structure-aware

Decoder

64x128

128

64

256

256

256

256

256

256

256

15 of 35

Important factors for developing diminished reality models

​

Structural awareness

  • ✅ background hallucination
  • ❌ Generated content heavily

relies on predicted layout

​

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

16 of 35

Important factors for developing diminished reality models

​

Structural awareness

  • ✅ background hallucination
  • ❌ Generated content heavily

relies on predicted layout

​

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

Predicted layout

Diminished panorama

Input panorama

17 of 35

Important factors for developing diminished reality models

​

Structural awareness

  • ✅ background hallucination
  • ❌ Generated content heavily

relies on predicted layout

​

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

Predicted layout

Diminished panorama

Input panorama

18 of 35

Important factors for developing diminished reality models

​

Structural awareness

  • ✅ background hallucination
  • ❌ Generated content heavily

relies on predicted layout

​

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

Predicted layout

Diminished panorama

Input panorama

19 of 35

Important factors for developing diminished reality models

​

Adaptability

✅ Leverage the style of the surrounding context

via SEAN blocks

​

​

​

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

20 of 35

Mitigating spherical distortion

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

Spherical attention mask

 

*

21 of 35

Applicability of Inpainting methods to DR

[1] Li, J., Wang, N., Zhang, L., Du, B., & Tao, D. (2020). Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7760-7768)

[2] Zheng, C., Cham, T. J., & Cai, J. (2019). Pluralistic image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1438-1447).

Li et al.

Zheng et al.

PanoDR

Input panorama

22 of 35

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

23 of 35

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

24 of 35

Real-world results

360° DR

Perspective view

25 of 35

Convergence

✅

Epochs required to converge

PanoDR

+166%

+216%

Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., & Zarpalas. PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes. OmniCV CVPRW, 2021

26 of 35

Shortcomings

  • Hybrid Image Inpainting & Image-to-Image translation

- Complex training (no end-to-end)

​

​

​

27 of 35

Instant Automatic Emptying of Panoramic Indoor Scenes

Pintore, G., Agus, M., Almansa, E., & Gobbetti, E. (2022). Instant Automatic Emptying of Panoramic Indoor Scenes. IEEE Transactions on Visualization and Computer Graphics.

28 of 35

Instant Automatic Emptying of Panoramic Indoor Scenes

Pintore, G., Agus, M., Almansa, E., & Gobbetti, E. (2022). Instant Automatic Emptying of Panoramic Indoor Scenes. IEEE Transactions on Visualization and Computer Graphics.

Image-to-image translation: Supervision depends solely on empty room!

29 of 35

Results

Input Ground Truth PanoDR Pintore et al.

Pintore, G., Agus, M., Almansa, E., & Gobbetti, E. (2022). Instant Automatic Emptying of Panoramic Indoor Scenes. IEEE Transactions on Visualization and Computer Graphics.

30 of 35

Results

Input Ground Truth Input Pintore et al.

Pintore, G., Agus, M., Almansa, E., & Gobbetti, E. (2022). Instant Automatic Emptying of Panoramic Indoor Scenes. IEEE Transactions on Visualization and Computer Graphics.

31 of 35

Instant Automatic Emptying of Panoramic Indoor Scenes

Pintore, G., Agus, M., Almansa, E., & Gobbetti, E. (2022). Instant Automatic Emptying of Panoramic Indoor Scenes. IEEE Transactions on Visualization and Computer Graphics.

32 of 35

Instant Automatic Emptying of Panoramic Indoor Scenes

Pintore, G., Agus, M., Almansa, E., & Gobbetti, E. (2022). Instant Automatic Emptying of Panoramic Indoor Scenes. IEEE Transactions on Visualization and Computer Graphics.

33 of 35

Instant Automatic Emptying of Panoramic Indoor Scenes

  • Image-to-image translation models are prone to generating images with hue discrepancies

Pintore, G., Agus, M., Almansa, E., & Gobbetti, E. (2022). Instant Automatic Emptying of Panoramic Indoor Scenes. IEEE Transactions on Visualization and Computer Graphics.

34 of 35

Conclusion

​

  • Wide field of view provides rich content and ameliorates the limited receptive field of convolution kernels
    • Adequate performance with small convolution kernels (5x5 and 3x3)
  • Geometry information (depth, layout) is crucial for DR
  • Absence of 360° datasets with full-empty configuration
  • Regular convolutions restrict the field of view of the network.
    • Fourier convolutions?
  • Image-to-image translation models are vulnerable to hue inconsistencies

35 of 35

Thank you!