1 of 14

Dynamic 3D Gaussian Fields for Urban Areas

Presenter: Yiduo Hao

Sep. 23, 2024

Tobias Fischer, Jonas Kulhanek, Samuel Rota Bulò, Lorenzo Porzi, Marc Pollefeys, Peter Kontschieder

ETH Zürich, Meta Reality Labs, CTU Prague

2 of 14

Problem

Large-Scale Dynamic Urban Areas from heterogeneous, multi-sequence data.

3 of 14

Background

Dynamic

NeRF: Very Slow
3DGS: Explicit SH cannot model transient geometry and appearance variations (seasonal/weather)

Large Urban Areas

NeRF: Very Slow
3DGS: High Memory Requirements

4 of 14

Contribution

Hybrid representation approach

Geometry scaffold: 3D Gaussian primitives
Appearance: Fixed-size neural fields

Model scene dynamics with a graph-based representation
Model non-rigid deformations in this canonical space with neural fields

In particular, we use 3D Gaussian primitives as an efficient geometry scaffold.

However, we do not store appearance as a per-primitive attribute, thus avoiding more than 80% of its memory footprint. Instead, we use fixed-size neural fields as a compact and flexible alternative. This allows us to model drastically different appearances and transient geometry which is essential to reconstructing urban areas from heterogeneous data.

We use neural fields to incorporate scene-specific transient geometry and appearances into the rendering process of 3D Gaussian splatting, overcoming its limitation to static, homogeneous data sources while benefitting from its efficient rendering

Finally, we model scene dynamics with a graph-based representation that maps dynamic objects to canonical space for reconstruction. We model non-rigid deformations in this canonical space with our neural fields to cope with articulated dynamic objects common in urban areas such as pedestrians and cyclists. This decomposed approach further enables a flexible scene composition suitable to downstream applications.

5 of 14

Overview

6 of 14

Graph-Based Scene Configuration

7 of 14

Graph-Based Scene Configuration

Model the time-varying sequence appearance and geometry

As and Gs are appearance and geometry modulation matrices.

γ(·) is a 1D basis function of sines and cosines.

For objects, use both an object code and a time encoding

8 of 14

3D Gaussians and Neural Representation

Do not Hold any Appearance Information (Reduce Memory footprint by 80%)

9 of 14

3D Gaussians

10 of 14

Results

11 of 14

Results

12 of 14

Results

13 of 14

Results�

14 of 14

Ablation