Extracting Triangular 3D Models, Materials, and Lighting From Images
CVPR 2022
1 Intro
Multi-view reconstruction
Share similar architecture to NeRF
NeRF output density+color
Disentangled representation
Input: multi-view images
Output: mesh, textures and light probe
More flexible application
Benefits and Applications
01
02
03
Editing
Re-lighting
Simulation
In Contrast, NeRF is not editable
Editable NeRF
Relation to our wire art
02
03
01
NeRF
Learning Radiance
This paper
Disentangle topology, material, lighting
NeuS (our)
Disentangle geometry from color
Related Works
3D Reconstruction
Classical methods
Depth estimation/ Multiview stereo
Neural represetations
NeRF and follow-ups
Unisurf, NeuS
Related Works
BRDF
Lighting Estimation
Bidirectional Reflectance Distribution Function
2 Method
Overview
Rendering
Scene
Rendered Images
Scene Update
Volume rendering
-
Ground Truth
Loss
Part 1
Part 2
Part 3
Rasterize
Step 1: Topology
Initial: Regular tetrahedra grid
Predict SDF
Find surface (Marching Tetrahedra)
Step 1: Topology
Refinement: Volume subdivision
Extract surface
Step 2: Shading
Extracted surface
Predict Material: PBR
kd: base color
korm: r(roughness), m(metalness)
Step 2: Shading
Predict texture parameterization
1 kd: base color
2 korm: r(roughness), m(metalness)
3 normals
x
Texture map
Step 3: Lighting
Predict an image for image based lighting
Rendering
Render the scene using the following equation
Lighting
Material
Viewing angle
Use many simplification techniques: split sum approximation, specular approximation
Loss
Train the model using image loss
3 Experiments
Output
Good re-lighting quality
OKay novel view synthesis
Lighting reconstruction result
Better in real-world reconstruction
holes
floaters
Limitation
Quality vs speed tradeoff�Our main limitation is the simplified shading model, not accounting for global illumination or shadows.
Conclusion
Collection of various methods.
This technology is quite mature.
Useful in practice / Compatible with existing software.