Replicating Interactive 3D Scene �using Shape & Structural Priors

Jeonghyun Kim, M.Sc

_�contact: mildjay16@gmail.com

KISTI OXR 융합연구단 기술세미나, 2023.02.17

CAD-Represented Digital Twin

Replicating 3D Indoor Scene

Realistic Digital Replication

3D Simulation

Building-level Management

Replicating 3D Indoor Scene: Digital Twin

@NVIDIA Omniverse Replicator

synthetic data generation

physics-based simulation

@MatterPort 3D tour

Large-scale RGB-D Scanning�Virtual tour in 3D & 360-view

Replicating 3D Indoor Scene via RGB-D Scanning

Noisy, Occlusion, Non-interactable

RGB-D Sensor

Point Cloud (+ vertex color)

Replicating 3D Indoor Scene + Shape Prior

H. Muzhi et al., Interactive Scene Reconstruction, ICRA 2021

Noisy, Occlusion, �Non-interactable

Clean, Fine-grained, �Pre-defined Interaction

Deep learning-based Approaches!

S. Hampali et al., Monte Carlo Scene Search for 3D Scene Understanding, CVPR 2021

A. Avetisyan et al., Scan2CAD, CVPR 2019

H. Izadinia et al., IM2CAD, CVPR 2017

Framework Overview

RGB-D Sensor

Optimization Problem�(i.e. ICP-like method)

Deep-learning!

Parsing 3D Scene into Semantics & Instances

Encoder-�Decoder

Point Cloud�(N, 3)

3D Semantic Instances �(K, points, semantics)

Parsing 3D Scene into Semantics & Instances

ScanNetv2 Benchmark

Traditional Shape Retrieval from Raw 3D Shape

Shape DB

Geometric �Distance

Time-consuming,�Missing Structural Information

Query shape

Retrieved�Shape

(i.e., Chamfer Distance)

Shape Retrieval via Learned Shape Code

Structural details are still not preserved

Shape Codes�(n-dim vector)

Shape Code

Feature�Distance

Feature�Space�(K, n)

Shape DB�(K)

Reconstructed�Shape DB

Minimize Geometric Error

Shape DB

Query shape

Retrieved�Shape

Structural Analysis on 3D Shape

K. Mo et al., StructureNet: Hierarchical Graph Networks for 3D Shape Generation. ACM Transactions on Graphics (ToG), 2019.

K. Mo et al., StructureNet, ToG 2019

A X. Chang et al., ShapeNet, arXiv 2015

Seg&Struct – Structure Prediction on Raw 3D shapes

J. Kim et al., Seg&Struct, WACV 2023

Predicting Part Structure from Raw 3D shapes �using structure & segmentation priors

Seg&Struct: Structure-aware Shape Retrieval

Shape DB�w/ Structure�Info.

Structural�Difference

Intermediate�Representation�(i.e., Part Structure)

Query shape

Retrieved�Shape

Encoder-�Decoder

Using a predicted part structure as an intermediate representation �for fast/accurate shape retrieval

Seg&Struct: Structure-aware Shape Retrieval

Seg&Struct: Structure-aware Shape Retrieval

Next Breakthroughs (Future Work)

• 3D Surface Scan (full region)
• chair, table, cabinet
• Supervised Learning
• pre-defined structure

M. Sung et al., Data-driven Structural Priors for Shape Completion, ToG 2015

T. Luo et al., Learning-to-Group, ICRA 2020

Source Code / Dataset / etc.

• Scan2CAD [CVPR 2019]
• https://github.com/skanti/Scan2CAD (Dataset)
• Interactive Scene Reconstruction [ICRA 2021]
• https://github.com/hmz-15/Interactive-Scene-Reconstruction
• Monte Carlo Scene Search for 3D Scene Understanding [CVPR 2022]
• https://github.com/montescene/demo
• Deformation-aware Shape Retrieval [ECCV 2020]
• https://github.com/mikacuy/deformation_aware_embedding
• Scratch code for Neural shape retrieval!
• SoftGroup [CVPR 2022 Oral]
• https://github.com/thangvubk/SoftGroup
• Almost SOTA in 3D Semantic Instance Benchmark (ScanNet v2)