Grid-guided Neural Radiance Fields for Large Urban Scenes

University of Hong Kong, Shanghai AI Laboratory, Max Planck Institute, Zhejiang University, and Adobe Research

Video too large to upload to Google slides

Motivation

• NeRF-based methods show good results for object level scenes but ability to represent scenes depends heavily on model capacity
• At large scales can only produce blurred renderings without fine details
• Grid-based methods can represent local detail but suffer from tradeoff between scene-scale and granularity
• Existing methods such as BlockNeRF use grids of sub-NeRFs, gets decent balance but suffers from discontinuities between cells

Motivation

• Main idea instead is to use a multi-resolution voxel-grid to assist in NeRF training
• Two key benefits to guiding NeRF with a voxel-grid
• Pretrained voxel-grid serves as an approximation to where surfaces are which significantly improves ray-point sampling for NeRFs, both improves the quality of sampling and reduces the number of samples needed since points are guaranteed to be near the true surface
• Provide learned coarse color and opacities to the NeRF which reduces what the NeRF needs to learn. NeRF no longer needs to memorize the color/opacities over the positions, just needs to learn how to make small adjustments

Framework

Two-stage Training Pipeline

Grid Branch

• In conventional 3D voxel grids, store a feature vector at vertices
• However, observe that many urban scenes can be mostly described by xy-location
• Buildings, roads, trees tend to be NSEW axis-aligned and have a prominent vertical dimension
• Grid can be compressed by factoring into a multi-resolution ground feature plane (xy-axes), and global vertical feature vector (z-axis)
• Features corresponding to a certain point are recovered via outer product of channels, similar to TensoRF

Grid Branch

Grid Branch (Stage 1)

Grid-guided NeRF

• Pretrained grid already provides a decent approximation of the scene, guides NeRF’s point sampling and provides intermediate features along with positional/directional encodings

​

• Relieves the fitting burden of NeRF’s PE so it can concentrate on refining details of the scene

​

Why two stage training pipeline?

• Randomly initialized feature grid does not provide any useful information
• Optimization of coarse grid only during stage 1 is significantly faster than jointly optimizing both because gradients don’t have to flow through the NeRF before back propagating to the feature grid
• Previous methods (Direct Voxel Grid Optimization, Sun et al., 22) freeze voxel grid, before feeding to NeRF, but feature grid can be further refined while jointly learning with NeRF, supervising reconstruction loss of both grid and NeRF

​

Grid-guided NeRF Branch (Stage 2)

• Feature grid can easily be upsampled by adding another layer via (bi)linear interpolation to the multi-resolution pyramid without having to retrain the pretrained coarse grid
• Supervision contains two losses: reconstruction loss of the fine coarse grid, and reconstruction loss of NeRF with the fine coarse grid as additional input
• Note that the coarse grid receives gradients from two pathways: directly from its reconstruction loss, and gradients backpropped from the NeRF reconstruction loss (not bootstrapped)

​

​

Training details

• Coarse grid is pretrained with Adam optimizer with initial learning rate 0.02 and exponential decay, only trained for 10000 iterations
• Used hierarchical point sampling, 4096 rays per batch, 64 coarse points and 128 fine points
• During joint training, MLPs are trained with Adam learning rate of 0.01, but trained for 100000 iterations, ratio of time spent training is around 1:4
• Reconstruction losses of two branches are weighted equally

​

​

Results (benchmarks)

Results (xy feature grid)

Pretrained

Refined

Density

Appearance

Results (xy feature grid)

TensoRF

Fine grid

Results (NeRF branch)

Results

Questions?