TensoIR:

Tensorial Inverse Rendering

Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang,

Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, Hao Su,

Zhejiang University | UC San Diego | Kingstar Technology Inc. | Adobe Research

Present by Yihang Liu

Inverse Rendering

Reconstructing physical attributes from captured images

• Geometry / Shape
• Surface material
• Lighting

​

We can just understand it literally -

Inverse the process of rendering

​

2

Inverse Rendering - Why So Difficult?

• Different combinations could create the same image
• Hard to separate material color vs. lighting effects
• Light bounces multiple times and creates shadows
• Unknown lighting conditions
• Visibility is not differentiable
• High computational cost
• …

​

​

Inverse Rendering - NeRF Solution

• NeRF is better than old methods like meshes, volumes, and point clouds.
• More flexible, more accurate geometry, and better rendering
• Problem: Limited capacity & High computational costs (MLPs form)
• Recent works are making NeRF more efficient
• TensoRF - Fast and compact reconstruction, allow unknown lighting

​

​

Method - Overview

Input Images:

Multi-view w. unknown lighting conditions

MLPs

Loss Functions:

Rendering Loss

Normal Regularization

BRDF Smoothness

Other constraints

Method - Rendering

Radiance field rendering

• Similar to NeRF, uses ray marching
• Camera ray r(t) = o + td

​

Physically-based rendering

• Physically-based BRDF model
• Deal with direct illumination, shadows, indirect lighting

Integrates over hemisphere for incoming light

Computes surface intersection points from volume rendering weights

Method - TensoRF-Based Representation

• Two separate VM-factorized tensors
• Density tensor (G_σ): For scene geometry
• Appearance tensor (G_a): For appearance properties

​

​

​

• Multiple MLP decoders
• Radiance Net
• Normal Net
• Material Net (BRDF parameters)

Method - Illumination and Visibility

Compute secondary effects Efficiently

• Computes visibility using ray marching
• Calculates indirect illumination from radiance field
• Uses environment map (parameterized by spherical Gaussians) for direct lighting

Method - Multi-Light Representation

• Add an extra dimension to appearance tensor for multiple lighting conditions
• 5D tensor (handles multiple lighting conditions efficiently)
• Share geometry and material properties across lighting conditions
• Only view-dependent colors vary with lighting

Method - Joint Reconstruction and Training

• Rendering losses

​

​

• Normal regularization

​

​

• BRDF smoothness & additional regularization terms

Experiment

Experiment

Experiment - Ablations

Thank you!

And questions…