NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, Wenping Wang

Lecturer

Wenqi Jia

Problem Statement: Why NeuS?

Goal: Recover accurate 3D surfaces from multi-view 2D images

Challenges:

• NeRF and volumetric models struggle with detailed surface recovery
• Traditional 3D reconstruction methods lack generalizability
• The need for a method that can represent surfaces explicitly while leveraging volume rendering

Key Contribution of NeuS

• NeuS: A novel framework for learning neural implicit surfaces using volume rendering
• Combines signed distance functions (SDFs) for surface representation
• Uses differentiable volume rendering for training
• Handles both surface geometry and appearance from multi-view images

Implicit SDF surface representation

+

volume rendering scheme

Signed Distance Function Basics (SDF)

NeuS encodes the 3D surface using an SDF neural network as implicit surface representation

• Represents a surface by measuring the distance of a point to the nearest surface
• Positive outside the surface, negative inside
• Zero on the surface itself

Volume Rendering

Differentiable Rendering:

• NeuS integrates SDF with volume rendering, allowing for training using 2D image supervision
• The network learns to render the scene with implicit surfaces by minimizing rendering errors

Uncertainty Handling:

• Multiple sampled points along the ray: Accounts for uncertainties in surface estimations during rendering to improve robustness

​

Training Procedure

Input: Multi-view 2D images

Output: 3D surface geometry and texture

Optimization Target: Minimize the difference between the rendered pixel colors and the ground truth pixel colors, without any 3D supervision

Dataset: DTU[1], BlendedMVS[2]

[1] Yariv, Lior, et al. "Multiview neural surface reconstruction by disentangling geometry and appearance." Advances in Neural Information Processing Systems 33 (2020): 2492-2502.

[2] Yao, Yao, et al. "Blendedmvs: A large-scale dataset for generalized multi-view stereo networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

Qualitative Results: NeuS v.s. IDR & NeRF

NeuS captures fine details like edges and smooth surfaces

Qualitative Results: NeuS v.s. NeRF & COLMAP

​

Archaeologist 1�Reconstruction/Rendering in the Past

Junkun Chen

​

https://docs.google.com/presentation/d/1FbA29d21cz8CczlFYq-xqqgplqQn2J1hi3mT0xdBd8Y/edit?usp=sharing

Trails of Development

• Traditional Multi-View reconstruction:
• Cat 1: Point-based and Surface-based
• Cat 2: Volume-based

​

• Neural-based representation
• Explicit representation v.s. Implicit representation
• Surface rendering v.s. Volume rendering

​

Traditional Methods: Point/Surface-Based

• Estimate pixel-wise depth maps by finding inter-view correspondences
• Unproject depth maps into a point cloud, then apply mesh construction

Correspondence,

Correspondence,

Correspondence!

Takeo Kanade

Traditional Methods: Volume-Based

• Estimate occupancy/opacity and color in each voxel grid
• Evaluate color consistency of each voxel

Correspondence,

Correspondence,

Correspondence!

Takeo Kanade

Neural Representation

• Representation types
• Explicit: Uses explicit 3D structure to represent the 3D shape�E.g., mesh, point clouds, voxels, hash grids, OCTrees, …
• Implicit: Use neural network to represent an “abstract” 3D-input function�E.g., DeepSDF, NeRFs, …
• Rendering modes
• Surface rendering: color is only at the intersection of ray and surface�E.g., traditional mesh rendering
• Volume rendering: color is at every “aether” in the space with an opacity�E.g., NeRF

Neural Representation

• Connect with what we learned from previous classes

Inspirations towards NeuS

• Combine the advantages of both surface and volume rendering

NeuS ('21)

Archaeologist 2

Jackie Lin

Huge Impact in surface reconstruction

• 1400+ citations since 2021

​

Concurrent work on volume rendering surfaces

VolSDF (NeurIPS 2021 Dec)

Yariv, L., Gu, J., Kasten, Y., & Lipman, Y. (2021). Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34, 4805-4815.

​

UNISURF (IVCC 2021 Oct)

Oechsle, M., Peng, S., & Geiger, A. (2021). Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5589-5599).

-> Occupancy networks (replace NeRF volume density with occupancy network)

Building off NeuS

Sparseneus (ECCV 2022)

Long, X., Lin, C., Wang, P., Komura, T., & Wang, W. (2022, October). Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In European Conference on Computer Vision (pp. 210-227). Cham: Springer Nature Switzerland.

151 citations

1) a multi-level geometry reasoning framework to recover the surfaces in a coarse-to-fine manner; 2) a multi-scale color blending scheme for more reliable color prediction; 3) a consistency-aware fine-tuning scheme to control the inconsistent regions caused by occlusion and noise

Also based on MVSnerf (734 citations)

Direct Follow-up: NeuS2

NeuS2 - from same authors.

Fast scene reconstruction and dynamic scene reconstruction from multiview videos. Per frame reconstruction in 20 seconds

Private Investigator

Hao Zhang

• The author's background and research experience

​

• The research environment at the time

(e.g., joining new group, finishing QE, and the emergence of foundational work.

​

​

Volume Rendering, SDF:

Great minds think alike

Prior Works:

Volume Rendering:

SDF, Ray Tracing:

Industrial Practitioner

Hao-Yu Hsu

Omni-Reconstruct

Reconstruct Anything, Anywhere, Anytime

Product Overview

Reconstruct everything from a single video captured by your mobile phones!

We offer a SaaS platform that enables seamless video streaming to our cloud, where it processes videos and returns the scene reconstruction results.

Competitor: Object Capture (Apple)

Object-level reconstruction with mobile captures

Omni-Reconstruct:

We have scene-level reconstruction!

Framework Details

Input RGB frame

Current scene meshes

Video Streaming

Omni-Reconstruct

Surface Reconstruction &

Intrinsic Decomposition (Optional)

Updated scene meshes

Residual Mesh Streaming

Dynamic Object Reconstruction

(ex: Humans, Vehicles)

Human Annotations (Optional)

Off-the-shelf

Seg. Model

Input depth (Optional)

Powered by:

Product Workflow

I want 3D geometry fr!!!

Omni-Reconstruct

Now I can do many cool application!!!

You

Mobile Captures

Scene Geometry

Application

• Indoor furniture arrangement
• Real life Pokemon battle
• Sky's the limit!

Where creativity comes reality

Omni-Reconstruct

• A toolkit allows user to reconstruct mesh from a video
• Why explicit geometry is KEY?
• Physical Simulation
• Illumination Modeling
• Exportable asset, suitable for most real-time graphics pipeline
• Easy to manipulate
• Application:
• Game dev
• Building modeling
• AR (like pokemon GO, indoor scene planning)
• Competitor: Apple Object Capture (https://developer.apple.com/augmented-reality/object-capture/)

Critic

Al Smith

Core critiques of NeuS:

​

Computational Complexity

• Inference and training

Handling of sparse views and incomplete geometry

Lack of camera pose refinement

Limited to unit-sphere bounding region

Textureless objects (obviously)

Computational Complexity

NeuS is very slow to train (on the order of ½ day)

Inference time is also slow (order of minutes)

Nowhere near real-time, unable to handle moving/deforming geometries

Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3295–3306, 2023.

Handling Sparse Views and Incomplete Geometry

NeuS does not handle incomplete viewing of the scene well.

Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. 2022. SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII. Springer-Verlag, Berlin, Heidelberg, 210–227. https://doi.org/10.1007/978-3-031-19824-3_13

Pose Refinement

NeuS assumes accurate camera pose is already given, and does not resolve errors in camera pose (left to future work)

NeRF also assumes pose is given, but IDR simultaneously resolves pose and structure

​

Some methods have attempted to improve this.

​

Shi-Sheng Huang, Zi-Xin Zou, Yi-Chi Zhang, and Hua Huang. Sc-neus: Consistent neural surface reconstruction from sparse and noisy views. arXiv preprint arXiv:2307.05892, 2023.

Geometric Limitations

Assumes the scene is bounded within a sphere, limiting NeuS to only bounded objects.

Cannot handle forward-facing scenes like NeRF, which used normalized device coordinates space to handle forward-facing views (like an unbounded wall)

• This is a limitation of the SDF as a surface representation, maybe UODF could be used?

​

​

Lu, Yujie, Long Wan, Nayu Ding, Yulong Wang, Shuhan Shen, Shen Cai, and Lin Gao. "Unsigned Orthogonal Distance Fields: An Accurate Neural Implicit Representation for Diverse 3D Shapes." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20551-20560. 2024.

Final Note… (Potentially Sensitive Topic)

Naming is important… be careful.

​

​

​

​

BARF is another example… Make your papers accessible to all!

Check with native speakers to make sure acronyms are culturally appropriate

…

Graduate Student

Christopher Conway

Issues to Explore:

• Neural Surface is very powerful, but it is computationally complex and has requirements for input resolution and quality
• Research Question 1: How can we implement NeuS on edge devices with lower computational power, and pursue real-time application?
• Additionally, scenes and scene conditions are assumed static whereas reality is often dynamic
• Research Question 2: How can we account for scene changes such as moving objects or altered lighting conditions?

Idea 1: Lightweight Implementation

• Much work needs to be done to achieve real-time or near real-time performance
• Coarse-to-Fine methods have seen success in lightweighting a variety of CV algorithms
• NeuS2 demonstrates reconstruction at up to 20 s per frame through multi-resolution hash encodings from coarse-to-fine, but this is still relatively slow

​

Topics to Further Explore:

1. Preconditioning - initializing NeuS with a traditional reconstruction or known data can speed up convergence time
2. Smarter Sampling - increased sampling at regions of interest and decreased samples elsewhere could reduce the computation size
3. Further Data Compression - compression will be required for use on lower memory devices

Idea 1: Lightweight Implementation

• Much work needs to be done to achieve real-time or near real-time performance
• Coarse-to-Fine methods have seen success in lightweighting a variety of CV algorithms
• NeuS2 demonstrates reconstruction at up to 20 s per frame through multi-resolution hash encodings from coarse-to-fine, but this is still relatively slow

​

Topics to Further Explore:

• Preconditioning - initializing NeuS with a traditional reconstruction or known data can speed up convergence time
• Smarter Sampling - increased sampling at regions of interest and decreased samples elsewhere could reduce the computation size
• Further Data Compression - compression will be required for use on lower memory devices

Idea 2: Hybrid NeuS

• NeuS2 addresses dynamic scenes through incremental training, but it does not use dense correspondences between frames, or meshing of SDF like in BakedSDF
• The addition of physics grounding or integrated rendering such as lighting can greatly improve the utility of NeuS methods for real world applications
• NeuS-PIR (Pre-Integrated Rendering) shows a similar idea with integration of lighting rendering - radiance field is factorized into material field and lighting representation

Topics to Further Explore:

• Mechanics Simulation - combine NeuS with traditional simulation based methods to generate novel views further into a mechanical test
• Opaque Materials - integrate subsurface scattering models with NeuS

NeuS-PIR

Idea 2: Hybrid NeuS

• NeuS2 addresses dynamic scenes through incremental training, but it does not use dense correspondences between frames or meshing of SDF like in BakedSDF
• The addition of physics grounding or integrated rendering such as lighting can greatly improve the utility of NeuS methods for real world applications
• NeuS-PIR (Pre-Integrated Rendering) shows a similar idea with integration of lighting rendering - radiance field is factorized into material field and lighting representation

Topics to Further Explore:

• Mechanics Simulation - combine NeuS with traditional simulation based methods to generate novel views further into a mechanical test
• Opaque Materials - integrate subsurface scattering models with NeuS

NeuS-PIR

Graduate Student

Christopher Conway

Issues to Explore:

• Neural Surface is very powerful, but it is computationally complex and has requirements for input resolution and quality
• Research Question 1: How can we implement NeuS on edge devices with lower computational power, and pursue real-time application?
• Additionally, scenes and scene conditions are assumed static whereas reality is often dynamic
• Research Question 2: How can we account for scene changes such as moving objects or altered lighting conditions?

Idea 1: Lightweight Implementation

• Much work needs to be done to achieve real-time or near real-time performance
• Coarse-to-Fine methods have seen success in lightweighting a variety of CV algorithms
• NeuS2 demonstrates reconstruction at up to 20 s per frame through multi-resolution hash encodings from coarse-to-fine, but this is still relatively slow

​

Topics to Further Explore:

• Preconditioning - initializing NeuS with a traditional reconstruction or known data can speed up convergence time
• Smarter Sampling - increased sampling at regions of interest and decreased samples elsewhere could reduce the computation size
• Further Data Compression - compression will be required for use on lower memory devices

Idea 1: Lightweight Implementation

• Much work needs to be done to achieve real-time or near real-time performance
• Coarse-to-Fine methods have seen success in lightweighting a variety of CV algorithms
• NeuS2 demonstrates reconstruction at up to 20 s per frame through multi-resolution hash encodings from coarse-to-fine, but this is still relatively slow

​

Topics to Further Explore:

• Preconditioning - initializing NeuS with a traditional reconstruction or known data can speed up convergence time
• Smarter Sampling - increased sampling at regions of interest and decreased samples elsewhere could reduce the computation size
• Further Data Compression - compression will be required for use on lower memory devices

Idea 2: Hybrid NeuS

• NeuS2 addresses dynamic scenes through incremental training, but it does not use dense correspondences between frames, or meshing of SDF like in BakedSDF
• The addition of physics grounding or integrated rendering such as lighting can greatly improve the utility of NeuS methods for real world applications
• NeuS-PIR (Pre-Integrated Rendering) shows a similar idea with integration of lighting rendering - radiance field is factorized into material field and lighting representation

Topics to Further Explore:

• Mechanics Simulation - combine NeuS with traditional simulation based methods to generate novel views further into a mechanical test
• Opaque Materials - integrate subsurface scattering models with NeuS

NeuS-PIR

Idea 2: Hybrid NeuS

• NeuS2 addresses dynamic scenes through incremental training, but it does not use dense correspondences between frames or meshing of SDF like in BakedSDF
• The addition of physics grounding or integrated rendering such as lighting can greatly improve the utility of NeuS methods for real world applications
• NeuS-PIR (Pre-Integrated Rendering) shows a similar idea with integration of lighting rendering - radiance field is factorized into material field and lighting representation

Topics to Further Explore:

• Mechanics Simulation - combine NeuS with traditional simulation based methods to generate novel views further into a mechanical test
• Opaque Materials - integrate subsurface scattering models with NeuS

NeuS-PIR

Learning Neural Implicit 2D shape by Volume Rendering

​

Jiahua Dong

Learning Neural Implicit 2D shape by Volume Rendering

Task: Reconstruct 2D shapes by multi-view 1-D images.

Our steps to generate the setting:

2D Shape (Circle)

Build cameras

Learning Neural Implicit 2D shape by Volume Rendering

Task: Reconstruct 2D shapes by multi-view 1-D images.

Our steps to generate the setting:

2D Shape (Circle)

Build cameras

However, solving such task is impossible if there’s no texture information

Learning Neural Implicit 2D shape by Volume Rendering

Task: Reconstruct 2D shapes by multi-view 1-D images.

Our steps to generate the setting:

Add texture

Learning Neural Implicit 2D shape by Volume Rendering

Task: Reconstruct 2D shapes by multi-view 1-D images.

Our steps to generate the setting:

Collect GroundTruth

Learning Neural Implicit 2D shape by Volume Rendering

Task input:

• Camera parameters
• Multi-view images

​

Output:

• SDF values (or final 2D shape)
• Novel view renderings （Optional)

Input

Results on simple shapes

|Predicted SDF|

Predicted Shape

GT Shape

Difference

Results on more complicated shapes

|Predicted SDF|

Predicted Shape

GT Shape

Difference

Results on the MNIST dataset

Novel-view rendering results

Limitations

Time consuming:

• Need about 100K iterations to converge
• The voluming sampling needs to be dense (512 samples)

​

Quality is not perfect

• The reconstructed shape is slightly smaller than GT
• Can’t solve the holes or areas not observable

2D Poisson Reconstruction

Yuqun Wu

Poisson Reconstruction

• Find the indicator function with gradients fitting the sampled surface normals of points
• Transfer to Poisson Problem via divergence operator
• ​

Simple Shape with naive implementation

Simple Shape with SDF regularization

Complex Shape

Sphere Tracing with SDFs

Hacker 3

Sphere Tracing

• Ray marching algorithm used to render implicit surfaces defined by SDFs
• How it works: -
• Cast rays from a camera, marching along each ray by the distance to the nearest surface
• Rays stop when they hit a surface (within a small tolerance) or reach a max distance.
• Application: Ideal for visualizing complex surfaces that would be hard to model explicitly

Demo Overview

Rendering Process:

• Define SDFs for basic shapes (e.g., circles, boxes).
• Use sphere tracing to render these implicit surfaces.
• Each pixel represents a ray cast into the scene, and the closest surface hit is shaded.

Key Functions:

• sphere_trace: Traces rays and detects surface hits.
• scene_sdf: Defines the geometry of the scene using SDFs.

Results and Comparison to Ray Marching

Vanilla Ray Marching Sphere Tracing

3.5s 0.4s