Instant-3D: Instant Neural Radiance Field Training Towards On-Device AR/VR 3D Reconstruction

Sixu Li*, Chaojian Li*, Wenbo Zhu, Boyang Yu, Yang Zhao,

Cheng Wan, Haoran You, Huihong Shi, and Yingyan (Celine) Lin

^�Georgia Institute of Technology

The 50^th International Symposium on

Computer Architecture (ISCA 2023)

Have you experienced 3D reconstruction to create anything in the digital world?

[Source: https://jonbarron.info/mipnerf360]

Background: What is 3D Reconstruction?

3D

Reconstruction

[Source: https://jonbarron.info/mipnerf360]

• Input: Sparsely sampled images
• Output: 2D images from any new view of the same scene

…

3D Reconstruction Demand has Surged

3D Reconstruction Demand has Surged

[Source: y2u.be/TX9qSaGXFyg]

[Source: y2u.be/afdnbXXbBTg]

[Source: y2u.be/XXXMrD7aWNs]

Virtual Telepresence

Metaverse

Rescue Robots

• Estimated market value: > $1.8 billion by 2028

[Source: shorturl.at/aDHY6]

On-Device 3D Reconstruction: Not Yet Possible 

• Instant on-device 3D reconstruction is highly desirable

On-Device 3D Reconstruction: Not Yet Possible 

[Müller et. al., SIGGRAPH’22]

Contribution 1: Identify the Bottleneck

Contribution 1: Identify the Bottleneck

Contribution 1: Identify the Bottleneck

Contribution 1: Identify the Bottleneck

Contribution 1: Identify the Bottleneck

Contribution 1: Identify the Bottleneck

Contribution 1: Identify the Bottleneck

Contribution 2: Instant-3D Algorithm

​

• We leverage: the reconstruction quality has different sensitivities on the color and density branches

​

Contribution 2: Instant-3D Algorithm

​

• We leverage: the reconstruction quality has different sensitivities on the color and density branches

​

Contribution 2: Instant-3D Algorithm

​

• Experiments on the two branches w/ different grid sizes

* Training time is measured on an edge GPU [Source: shorturl.at/rFMS0]

**PSNR/accuracy is measured on NeRF-Synthetic Dataset [Mildenhall et. al., ECCV’20]

Color Branch

Density Branch

Winning Configuration

👍

Contribution 2: Instant-3D Algorithm

​

• Experiments on the two branches w/ different update freq.

* Training time is measured on an edge GPU [Source: shorturl.at/rFMS0]

**PSNR/accuracy is measured on NeRF-Synthetic Dataset [Mildenhall et. al., ECCV’20]

Color Branch

Density Branch

Winning Configuration

👍

Contribution 2: Instant-3D Algorithm

• Our Instant-3D algorithm allocates different model complexities for the color and density branches

Larger grid size

Smaller grid size

More frequent weights update

Less frequent weights update

• We leverage: the reconstruction quality has different sensitivities on the color and density branches

​

Contribution 3: Instant-3D Accelerator

• We observe: the memory access pattern during embedding grid interpolation is predictable

​

Contribution 3: Instant-3D Accelerator

• We observe: the memory access pattern during embedding grid interpolation is predictable

​

Memory Address

Avg. Inter-Group Distance: 60,000

Avg. Intra-Group Distance: 2

(Correlate to their positions in the 3D grid)

Contribution 3: Instant-3D Accelerator

• Our Instant-3D accelerator reorganizes memory accesses to reduce data movement

• We observe: the memory access pattern during embedding grid interpolation is predictable

​

Instant-3D Accelerator: Overview

• Instant-3D accelerator leverages the properties of Instant-3D algorithm and observations on the memory access

Idle banks

Low-utilization in read requests to SRAM

Observations

Timestep

Instant-3D Accelerator: Overview

Memory Address

…

…

Low-utilization in read requests to SRAM

Frequent write requests to the same address

Observations

• Instant-3D accelerator leverages the properties of Instant-3D algorithm and observations on the memory access

Instant-3D Accelerator: Overview

Low-utilization in read requests to SRAM

Frequent write requests to the same address

Different model sizes for different applications

Observations

• Instant-3D accelerator leverages the properties of Instant-3D algorithm and observations on the memory access

Instant-3D Accelerator: Overview

• Instant-3D accelerator further reduces the dominant memory accesses based on the observations

Pre-fetches & executes memory accesses

Accumulates requests into necessary ones

Fuse processing cores for flexibility

Proposed Techniques

Low-utilization in read requests to SRAM

Frequent write requests to the same address

Different model sizes for different applications

Observations

Instant-3D Acc.: Feed-Forward Read Mapper

• We observe:
• The large inter-group distance results in low bandwidth utilization when accessing multi-bank SRAM

​

Instant-3D Acc.: Feed-Forward Read Mapper

Memory Address

Avg. Inter-Group Distance: 60,000

Avg. Intra-Group Distance: 2

…

SRAM Bank to be Accessed

Idle banks cause low-utilization

• We observe:
• The large inter-group distance results in low bandwidth utilization when accessing multi-bank SRAM

​

Instant-3D Acc.: Feed-Forward Read Mapper

…

SRAM Bank to be Accessed

• We observe:
• The large inter-group distance results in low bandwidth utilization when accessing multi-bank SRAM
• Sequential read requests access different SRAM banks

​

…

Timestep

…

…

Instant-3D Acc.: Feed-Forward Read Mapper

• We observe:
• The large inter-group distance results in low bandwidth utilization when accessing multi-bank SRAM
• Sequential read requests access different SRAM banks

​

​

• Our Feed-Forward Read Mapper (FRM) pre-fetches and pre-executes memory access if idle banks exist

Feed-Forward Read Mapper (FRM) Unit

Instant-3D Acc.: Back-Prop. Update Merger

• We observe: there exist frequent memory write requests to the same address during the back-propagation process

​

Instant-3D Acc.: Back-Prop. Update Merger

• We observe: there exist frequent memory write requests to the same address during the back-propagation process

​

Timestep of a Sliding Window of 1000 Continuous Accesses

Instant-3D Acc.: Back-Prop. Update Merger

• We observe: there exist frequent memory write requests to the same address during the back-propagation process

​

Memory Address

…

…

Temporal Locality

Instant-3D Acc.: Back-Prop. Update Merger

• We observe: there exist frequent memory write requests to the same address during the back-propagation process

​

• Our Back-Propagation Update Merger (BUM) accumulates write requests with a buffer for making only necessary SRAM accesses

Instant-3D Acc.: Back-Prop. Update Merger

• We observe: there exist frequent memory write requests to the same address during the back-propagation process

​

• Our Back-Propagation Update Merger (BUM) accumulates write requests with a buffer for making only necessary SRAM accesses

Instant-3D Acc.: Back-Prop. Update Merger

• We observe: there exist frequent memory write requests to the same address during the back-propagation process

​

• Our Back-Propagation Update Merger (BUM) accumulates write requests with a buffer for making only necessary SRAM accesses

Instant-3D Acc.: Back-Prop. Update Merger

• We observe: there exist frequent memory write requests to the same address during the back-propagation process

​

• Our Back-Propagation Update Merger (BUM) accumulates write requests with a buffer for making only necessary SRAM accesses

Instant-3D Acc.: Back-Prop. Update Merger

• We observe: there exist frequent memory write requests to the same address during the back-propagation process

​

• Our Back-Propagation Update Merger (BUM) accumulates write requests with a buffer for making only necessary SRAM accesses

Instant-3D Acc.: Multi-Core Fusion Scheme

• We observe: different grid sizes are necessary to accommodate the Instant-3D algorithm and scenes of varying scales

​

Instant-3D Acc.: Multi-Core Fusion Scheme

• We observe: different grid sizes are necessary to accommodate the Instant-3D algorithm and scenes of varying scales

​

Different Grid Sizes in Instant-3D Algorithm

Instant-3D Acc.: Multi-Core Fusion Scheme

• We observe: different grid sizes are necessary to accommodate the Instant-3D algorithm and scenes of varying scales

​

Different Grid Sizes in Instant-3D Algorithm

Different Grid Sizes in Scenes with Varying Scales

Instant-3D Acc.: Multi-Core Fusion Scheme

• We observe: different grid sizes are necessary to accommodate the Instant-3D algorithm and scenes of varying scales

• Our Multi-Core Fusion Scheme fuses processing cores with a fixed grid size to support flexible grid sizes

Instant-3D Acc.: Multi-Core Fusion Scheme

• Our Multi-Core Fusion Scheme fuses processing cores with a fixed grid size to support flexible grid sizes

Instant-3D Acc.: Multi-Core Fusion Scheme

Instant-3D Acc.: Multi-Core Fusion Scheme

Instant-3D Acc.: Multi-Core Fusion Scheme

Evaluation Setup

• Consider 13 scenes from 3 datasets
• Commonly-used NeRF-Synthetic dataset

​

​

​

• Large-scale SILVR dataset

​

​

​

• Real-world-captured ScanNet dataset

[Mildenhall et. al., ECCV’20]

[Courteaux et. al., MMSys’22]

[Dai et. al., CVPR’17]

Evaluation Setup

• Consider 13 scenes from 3 Datasets
• Commonly-used NeRF-Synthetic dataset
• Large-scale SILVR dataset
• Real-world-captured ScanNet dataset
• Benchmark w/ 3 baselines with various computing resources

[Mildenhall et. al., ECCV’20]

[Courteaux et. al., MMSys’22]

[Dai et. al., CVPR’17]

Instant-3D’s Speedup Over Baselines

Instant-NGP on Edge GPU

Our Instant-3D

[Müller et. al., SIGGRAPH’22]

Key Insights in Instant-3D

Observation:

Different quality sensitivities on color and density branches 

Allocate different model complexities for color/density

Observation:

Predictable memory access patterns during dominant steps

Dedicated design to reorganize memory accesses and fuse cores

Instant-3D Algorithm

Instant-3D Accelerator

Key Insights in Instant-3D

Our Instant-3D has delivered the first instant on-device Neural Radiance Fields (NeRF)-based 3D reconstruction

Instant-3D Algorithm

Instant-3D Accelerator

Allocate different model complexities for color/density

Dedicated design to reorganize memory accesses and fuse cores

Instant-3D: Instant Neural Radiance Field Training Towards On-Device AR/VR 3D Reconstruction

The 50^th International Symposium on

Computer Architecture (ISCA 2023)

This work was supported by the NSF through the CCF program and CoCoSys, one of the seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA.

Sixu Li*, Chaojian Li*, Wenbo Zhu, Boyang Yu, Yang Zhao,

Cheng Wan, Haoran You, Huihong Shi, and Yingyan (Celine) Lin

^�Georgia Institute of Technology

FRM & BUM Overheads

• As shown in Fig. 15b, the FRM and BUM units takes 25% of the area and energy but provide 3.1x speedup.

Speedup Breakdown

Compare with NeRF Inference Accelerator

Why On-The-Fly 3D Reconstruction

• Although offloading to the cloud is possible, it may cause undesired-latency and privacy-concerns
• On-device 3D reconstruction offers
• Smaller communication volume (20 MB reconstructed model vs. 120 MB jpeg images)
• Enhanced privacy
• An alternative to offloading under unstable/ unavailable-internet.

[Dai et. al., CVPR’17]

Implementation Details

• Technology Node: 28nm HPC+
• Corner: TT 25℃
• Instant-3D accelerator is implemented in RTL
• Toolchain:
• Synthesis: Synopsys Design Compiler
• Place & Route: Cadence Innovus
• IP used: DesignWare Floating Point Units

What If the Hash Table Size is Larger

• For larger SRAMs, the required time will be larger depending on the size
• For example, for a 2 MB hash table, it need to be processed twice

50%

Table

3D Points

Array A

Left 50%

Table

3D Points

Array A

Step 1: Load first 50% table, process the input

Step 2: Load left 50 % table, process the same input

Why Temporal Locality Exists

• Different training rays can pass through the same sampled point, corresponding to the same memory address

Only Compare with Commercial Devices

• We have tried our best to ensure the fairness in terms of computing resources

Only Compare with Commercial Devices

• We have tried our best to ensure the fairness in terms of computing resources
• As the first accelerator, we can only compare with the best commercial devices

Only Compare with Commercial Devices

• We have tried our best to ensure the fairness in terms of computing resources
• As the first accelerator, we can only compare with the best commercial devices
• We believe our Instant-3D can be the starting point for efforts from multiple communities to build the next 3D reconstruction accelerator

SOTA Efficient Algorithm: How Does it Work?

• Volume rendering 1) emits a ray from the origin for each pixel and 2) aggregates the queried features of points along the ray

[Mildenhall et. al., ECCV’20]

SOTA Efficient Algorithm: How Does it Work?

[Mildenhall et. al., ECCV’20]

SOTA Efficient Algorithm: How Does it Work?

• The queried features are generated by a MLP with the direction of the point and an embedding from a hash table as input

[Müller et. al., SIGGRAPH’22]

Reason for the Observed Memory Access Pattern

• We observe:
• The large inter-group distance results in low bandwidth utilization when accessing multi-bank SRAM
• Sequential read requests access different SRAM banks

​

• Unique observations on the SOTA efficient algorithm because of the adopted random hash mapping

In the same cell

Access different addresses randomly

[Müller et. al., SIGGRAPH’22]

Why Not Directly Frequent Update SRAM?

​

​

​

• SRAMs are needed for large memory such as hash tables
• SRAMs need 1 clock to read, 1 clock to write
• If we directly access SRAM, we need 3 clocks
• 1 for read, 1 for computation, 1 for write
• If we utilize register-based buffers
• Using combinational logic, we can use 1 cycle for read, compute, write

We Already Have GPUs, Why Accelerator?

• GPUs cannot satisfy the requirement of Instant (< 5 seconds) 3D reconstruction on edge devices

​

We Already Have GPUs, Why Accelerator?

• GPUs cannot satisfy the requirement of Instant (< 5 seconds) 3D reconstruction on edge devices
• It is difficult for GPUs to perform the required precise/fine-grained memory-access in our proposed Instant-3D

​

A Good Time for Building Accelerator?

• One next-technology disruptor, calling for both algorithm and hardware communities' mutual efforts
• Instant on-device NeRF training enables multiple applications
• The SOTA efficient algorithm has attracted 340+ citations and 11k+ GitHub stars within one year, and is widely adopted
• Our efforts contribute to the critical stage of efficient NeRF development.