JavaScript isn't enabled in your browser, so this file can't be opened. Enable and reload.

1 of 35

PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar

Presenter: Yiduo Hao

Nov. 18, 2024

Tzofi Klinghoffer, Xiaoyu Xiang*, Siddharth Somasundaram*, Yuchen Fan

Christian Richardt, Ramesh Raskar, Rakesh Ranjan

MIT, Meta, Codec Avatars Lab

2 of 35

Motivation

3 of 35

Motivation

4 of 35

Motivation

5 of 35

Current Vision Based Solutions

Hallucinate

Shadow

6 of 35

Motivation

7 of 35

Background: LiDAR

Our method, PlatoNeRF, takes inspiration from Plato's allegory of the cave. Much like prisoners discerning reality through shadows, our approach uses indirect, ‘shadow-like’ information from two-bounce lidar signals to infer occluded scene geometry. This method combines lidar’s unique strengths with Neural Radiance Fields—or NeRF—to reconstruct 3D scenes accurately and robustly, even under challenging conditions.

"To address these, we used lidar's two-bounce light. Unlike first-bounce lidar systems, which only map visible surfaces, two-bounce lidar captures reflections from occluded areas, offering a robust signal for reconstruction.”

"Single-photon lidar uses pulsed lasers to measure the time of flight (ToF) of light, offering precise depth information. Unlike traditional lidar systems, which only utilize the first bounce of light, our method incorporates two-bounce light—light that reflects off two surfaces before returning to the sensor.

Two-Bounce Lidar Integration: Captures indirect light paths to reveal occluded geometry.
Single-View Capability: Reconstructs the scene from a single perspective without hallucination.
Robustness: Handles challenges like low albedo, ambient light, and reduced sensor resolution, making it viable for consumer-grade devices."

8 of 35

Background: Single-Photon Two-Bounce LiDAR

Step 1: Shoot a beam

9 of 35

Background: Single-Photon Two-Bounce LiDAR

Step 2: First Bounce

10 of 35

Background: Single-Photon Two-Bounce LiDAR

Step 3: Second Bounce

11 of 35

Background: Single-Photon Two-Bounce LiDAR

Step 4: Shadow?

12 of 35

Pipeline Overview

13 of 35

Method: Single-Photon Two-Bounce LiDAR

Step 1: Shoot a laser beam from a fixed laser position

Camera position at

Distance from laser source to first bounce

14 of 35

Method: Single-Photon Two-Bounce LiDAR

Step 2: The first contact of the laser becomes a virtual light source at position

The virtual light source returns to SPAD

The virtual light source illuminate another surface at position

Distance travelled

15 of 35

Method: Single-Photon Two-Bounce LiDAR

The second illuminated surface returns to SPAD, with a distance travelled

The virtual light source can illuminate multiple second surface

16 of 35

Method: Single-Photon Two-Bounce LiDAR

17 of 35

Method: Single-Photon Two-Bounce LiDAR

Primary Ray

18 of 35

Method: Single-Photon Two-Bounce LiDAR

19 of 35

Method: Single-Photon Two-Bounce LiDAR

Shadow?

20 of 35

Method: Single-Photon LiDAR

21 of 35

Method: Single-Photon LiDAR

Known Values from LiDAR measurement:

sensor location & direction

laser location

distance from first bounce

GT two-bounce time of flight

if the pixel is in shadow

？

？

22 of 35

Method: Volumetric Lidar Rendering

23 of 35

Method: Primary Ray Rendering

Modeling

24 of 35

Method: Single-Beam Two-Bounce LiDAR

25 of 35

Method: Primary Ray Rendering

Training

26 of 35

Method: Secondary Ray Rendering

Modeling

27 of 35

Method: Secondary Ray Rendering

Training

28 of 35

Method: Volumetric Lidar Rendering - Inference

29 of 35

Baselines

Bounce-Flash Lidar: Our work is inspired by BounceFlash (BF) Lidar, which models two-bounce lidar analytically to estimate visible depth and occluded geometry from a single view, using geometric constraints and shadow carving, respectively. BF Lidar’s output is one point cloud (PC) for visible and one for occluded geometry, which we combine for our comparisons.

S 3 - NeRF is a recent method for learning neural scene representations using shadows. Using single-view RGB images captured under varying illumination, it trains a neural SDF model by exploiting shadow and shading information. A sphere is initialized at the origin where the object is assumed to be and known camera and light positions are used to model the scene’s bidirectional reflectance distribution function. S 3 -NeRF reconstructs both the object casting shadows and all other background scene geometry, making it a suitable comparison.

30 of 35

Results - Depth evaluation

31 of 35

Results - Depth evaluation

32 of 35

Results - Depth evaluation

33 of 35

Ablations

34 of 35

Ablations

35 of 35

Highlights and Limitations

Novel method to alleviate single-view
New framework for Lidar neural rendering

Only model Lambertian reflectance.
Built on top of vanilla NeRF, and, as a result, occasionally has floaters. However, our method is agnostic to the flavor of NeRF and can be integrated into others in the future.