1 of 14

Zhigang Jiang¹ Zhongzheng Xiang² Jinhua Xu¹ Ming Zhao^2�

¹East China Normal University ²Yiwo Technology

LGT-Net: Indoor Panoramic Room Layout Estimation with Geometry-Aware Transformer Network

2 of 14

Goal of 3D Room Layout Estimation Task

3 of 14

Room Layout Representation: Floor Boundary + Room Height

3D Room Layout Representation

Ceiling-Boundary

Floor-Boundary

Room Height

Vertical Direction

Horizontal Direction

Floor-Boundary

2D-Shape

Room Height

4 of 14

Floor-Boundary Representation

Vertical Direction

Horizontal Direction

Camera Center

Horizon-Depth

Ceiling-Boundary

Floor-Boundary

Room Height

Camera Height

Floor-Boundary

Camera Height (Set to 1)

Horizon-Depth

5 of 14

Room Layout Representation: Horizon-Depth + Room Height

Room Height

Ceiling-Boundary

Floor-Boundary

Room Height

Horizon-Depth

Motivation: Omnidirectional-Geometry Awareness

6 of 14

Motivation: Planar-Geometry Awareness

Ceiling-Boundary

Floor-Boundary

Room Height

2D-Shape

Wall

Corner

Model needs to be constrained to get a more regular 2D shape that reflects the flat walls

Floor-Boundary

7 of 14

MSA

Feature Sequence

G-RPE

Global Block

Feature Sequence

Window Partitioning

MSA

W-RPE

Window Merging

Window Block

MSA

Feature Sequence

Rolling

Window Partitioning

Window Merging

Reverse Rolling

W-RPE

Shifted Window Block

Add & Norm

Feed Forward

Add & Norm

Feature Sequence

Basic Transformer Encoder

SWG-Transformer Block

Network Architecture

512x1024x3

256x1024

Global Block

Shifted Window Block

Global Block

Window Block

x 2

SWG-Transformer

1x1

Post-Processing

3D Visualizing

256x1

256x1024

Feature Sequence

Input

Output

Feature Extractor

ResNet-50

There is the overall architecture of our proposed network.

#The feature extractor, like HorizonNet, is based on ResNet-50 to obtain a feature sequence.

#Then SWG-Transformer further processes the feature sequence.

#Finally, two branches output a horizontal depth sequence and a room height value.

#It is worth noting that we can calculate the normal and gradient by horizon depth. They are used to calculate the planar-geometry awareness loss.

#Here are the various blocks in SWG-Transformer, and the difference between them is the operation before and after MSA.

#The window blocks are used to reduce computation and enhance local modeling ability.

#However, each token corresponds to a horizon-depth value, so the global block is necessary to enhance the global awareness of each token.

#In the figure, W-RPE and G-RPE represent relative position embedding in window block and global block.

8 of 14

Relative Position Encoding

Feature Sequence

Circular Structure

W-RPE

G-RPE

Learnable Scalar Table

Relative Position Bias Matrix

9 of 14

Wall

Corner

Wall

Ceiling-Boundary

Floor-Boundary

Room Height

Vertical Direction

Horizontal Direction

Horizon-Depth Loss：

Room Height Loss:

Normal Loss：

Gradient Loss：

Normal：

Gradient of Normal：

3D Point：

Total Loss：

Omnidirectional-Geometry

Awareness:

Planar-Geometry

Awareness:

Loss Function

Camera Height = 1

10 of 14

Qualitative Results on MatterportLayout Dataset

HorizonNet

LED²-Net

Ours

11 of 14

Qualitative Results on ZInd Dataset

HorizonNet

LED²-Net

Ours

12 of 14

Quantitative Results and Ablation Study

13 of 14

Quantitative Results and Ablation Study

14 of 14

Online Demo: https://huggingface.co/spaces/zhigangjiang/3D-Room-Layout-Estimation_LGT-Net

GitHub:

https://github.com/zhigangjiang/LGT-Net