1 of 34

3D Gaussian Splatting for Real-Time Radiance Field Rendering �Bernhard Kerbl*, Georgios Kopanas*, Thomas Leimkühler, George Drettakis (*indicates equal contribution)�ACM Transactions on Graphics, 2023

Daniel Alexander (alexdan@seas.upenn.edu)

September 9, 2024

2 of 34

3D Gaussian Splatting

2

Figure 1: Gaussian Splat in Towne 311

3 of 34

History

3

Photogrammetry

A reconstruction technique by analyzing their geometric relationships.

Limitation: details and lighting, open-ended scenes

NeRF (Neural Radiance Fields)

Solves these problem by introducing deep learning to handle dynamic and open-ended scenes

Photogrammetry vs. NeRF

Photogrammetry is a technique that reconstructs 3D structures from 2D images by analyzing their geometric relationships. It involves creating 3D models by analyzing multiple images taken from different perspectives, often achieved by photographing an object from various angles, such as rotating it on a turntable while capturing image. However, it has limitations such as difficulties in capturing intricate details, handling non-rigid scenes, and being sensitive to lighting conditions. Neural Radiance Fields (NeRF) address these drawbacks by using deep learning to model complex 3D scenes directly from images, overcoming limitations in detail capture and handling dynamic or deformable objects. NeRFs provide a more flexible and accurate representation of 3D scenes, making them particularly effective in scenarios where traditional photogrammetry faces challenges.

NeRF vs. 3D Gaussian Splatting

Neural Radiance Fields (NeRF) is a method in computer graphics that uses neural networks to model and render complex 3D scenes directly from 2D images. By capturing both geometry and appearance in a unified model, NeRF achieves high-quality scene representation.

4 of 34

Problem with NeRFs

4

Computationally expensive!

5 of 34

Enter Gaussian Splatting!

5

Figure 2: Starting from a sparse Structure-from-Motion (SfM) point cloud, the optimization process used a fast tile-based renderer and generates a set of 3D Gaussians, and their density is adaptively controlled. (Source: Image taken from [1])

6 of 34

What is a 3D Gaussian?

6

A 3D gaussian is defined by:

Position (Mean μ): location (XYZ)
Covariance Matrix (Σ): rotation and scaling
Opacity (𝛼): Transparency
Color (RGB) or Spherical Harmonics (SH) coefficients

7 of 34

Training Process

7

Structure from Motion (SfM) initialization
Gradient Descent for Parameter Optimization
Adaptive Densification

8 of 34

Training Process

8

9 of 34

1st step: Structure from Motion

9

Structure from Motion (SfM) is a computer vision technique that reconstructs a three-dimensional scene from a set of two-dimensional images or video frames. The process involves camera motion estimation and 3D structure reconstruction.

It achieves this by first identifying and tracking distinct features across images, determining how they move between frames. Using this information, SfM then estimates the relative poses of the cameras at different points in time. Next, the technique triangulates the 3D positions of these features by finding where the corresponding rays from different camera viewpoints intersect. Finally, a refinement process, known as bundle adjustment, optimizes the camera poses and 3D points to minimize any discrepancies between the actual and projected feature locations.

Instead of random initialization which can affect the optimization process, SfM initialization of 3D points can be a good approach.

The rendering approach focuses on achieving fast overall rendering and sorting for efficient 𝛼-blending without having strict limits on the number of splats. In short, the process involves sorting of gaussians by depth and grouping the tiles.

The approach employs a tile-based rasterizer which divides the screen into 16x16 tiles to manage rendering tasks.
Gaussians are culled against the view frustum and each tile, retaining only those with a 99% confidence interval intersecting the frustum.
A guard band helps reject Gaussians at extreme positions.
Each Gaussian is instantiated based on the number of tiles they overlap, and a key is assigned considering view space depth and tile ID.
Sorting of Gaussians is performed based on these keys using a fast GPU Radix sort.
One thread block is launched for each tile independently, and each block loads gaussians packets into shared memory.
For each pixel, color and 𝛼 values are accumulated by traversing lists of Gaussians front-to-back.
Saturation of 𝛼 is the termination criterion (i.e., 𝛼 goes to 1), and threads are queried at intervals to stop processing when all pixels have saturated.
During the backward pass, the traversal of per-tile lists occurs back-to-front. To compute gradients, intermediate opacities are obtained by dividing the final accumulated opacity by each point’s 𝛼 in the back-to-front traversal.
This method allows an unlimited number of blended primitives to receive gradient updates, accommodating scenes with varied depth complexity without specific parameter tuning.

22 of 34

Fast Differentiable Rasterizer

22

Source: xoft (YouTube)

The rendering approach focuses on achieving fast overall rendering and sorting for efficient 𝛼-blending without having strict limits on the number of splats. In short, the process involves sorting of gaussians by depth and grouping the tiles.

The approach employs a tile-based rasterizer which divides the screen into 16x16 tiles to manage rendering tasks.
Gaussians are culled against the view frustum and each tile, retaining only those with a 99% confidence interval intersecting the frustum.
A guard band helps reject Gaussians at extreme positions.
Each Gaussian is instantiated based on the number of tiles they overlap, and a key is assigned considering view space depth and tile ID.
Sorting of Gaussians is performed based on these keys using a fast GPU Radix sort.
One thread block is launched for each tile independently, and each block loads gaussians packets into shared memory.
For each pixel, color and 𝛼 values are accumulated by traversing lists of Gaussians front-to-back.
Saturation of 𝛼 is the termination criterion (i.e., 𝛼 goes to 1), and threads are queried at intervals to stop processing when all pixels have saturated.
During the backward pass, the traversal of per-tile lists occurs back-to-front. To compute gradients, intermediate opacities are obtained by dividing the final accumulated opacity by each point’s 𝛼 in the back-to-front traversal.
This method allows an unlimited number of blended primitives to receive gradient updates, accommodating scenes with varied depth complexity without specific parameter tuning.

23 of 34

2nd step: Optimization

23

Source: xoft (YouTube)

Loss function is calculated by combining L1 and D-SSIM (Structural Dissimilarity Index)

24 of 34

2nd step: Optimization

24

Source: xoft (YouTube)

25 of 34

3rd step: Adaptive Densification

25

After every 100 iterations: Densify!

26 of 34

3rd step: Adaptive Densification

26

Figure 4: Top row (under reconstruction) and Bottom row (over-reconstruction) (Source: Image taken from [1])

After generating an initial set of sparse points using Structure from Motion (SfM), adaptive densification approach dynamically adjusts the number and density of 3D Gaussians to enhance scene representation during free-view synthesis.

After an initial optimization warm-up, it consistently densifies every 100 iterations, eliminating Gaussians with nearly transparent opacity values. The adaptive control addresses both under-reconstructed regions and over-reconstructed areas with large Gaussian coverage by leveraging view-space positional gradients.

Densification involves cloning small Gaussians in under-reconstructed regions and splitting large Gaussians in high-variance zones. To balance the system’s total volume and Gaussian count, 𝛼 values are occasionally reset, allowing controlled growth or reduction. This strategic approach, combined with periodic removal of excessively large Gaussians, maintains control over the total Gaussian count without the need for space compaction or warping. This method ensures Gaussians retain their representation in Euclidean space throughout the optimization process.

27 of 34

Results and Analysis

27

A side-by-side comparison of previous high-quality representations and Gaussian Splatting (marked as “Ours”) in terms of rendering speed (fps), training time (min), and visual quality (Peak signal-to-noise ratio, the higher the better) [Source: taken from [1]]

28 of 34

Results and Analysis

28

Very fast!!!!! Reached 197 FPS with similar losses among other methods. [Source: taken from [1]]

29 of 34

Pros

29

Very fast and accurate!

Despite the huge improvement compared to previous radiance field methods, this approach still faces limitations because of its simplicity.

Artifacts like elongated or “splotchy” Gaussians can occur, especially in challenging scenes. Popping artifacts may arise when creating large Gaussians, particularly in regions with view-dependent appearances.

In regions with inadequate coverage from training views, the random initialization method tends to introduce “floaters” that cannot be

removed by optimization.

The rasterizer’s guard band and visibility algorithm may contribute to popping artifacts and sudden depth/blending order changes. Also, antialiasing is not implemented, leaving room for improvement in addressing abrupt depth/blending order changes.

GPU memory consumption, while more compact than previous methods, can peak over 20 GB during training for large scenes. There is still room for improvement in reducing memory usage.

30 of 34

Limitations

30

Struggles with smooth surfaces
Artifacts like elongated or “splotchy” Gaussians
Popping artifacts when creating large Gaussians, particularly in regions with view-dependent appearances.
Introducing “Floaters” that cannot be removed by optimization.
GPU memory consumption

Despite the huge improvement compared to previous radiance field methods, this approach still faces limitations because of its simplicity.

Artifacts like elongated or “splotchy” Gaussians can occur, especially in challenging scenes. Popping artifacts may arise when creating large Gaussians, particularly in regions with view-dependent appearances.

In regions with inadequate coverage from training views, the random initialization method tends to introduce “floaters” that cannot be

removed by optimization.

The rasterizer’s guard band and visibility algorithm may contribute to popping artifacts and sudden depth/blending order changes. Also, antialiasing is not implemented, leaving room for improvement in addressing abrupt depth/blending order changes.

GPU memory consumption, while more compact than previous methods, can peak over 20 GB during training for large scenes. There is still room for improvement in reducing memory usage.

31 of 34

Short Demo

31

32 of 34

References

32

[1] Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. SIGGRAPH 2023.

33 of 34

Resources

33

34 of 34

3D Gaussian Splatting for Real-Time Radiance Field Rendering �Bernhard Kerbl*, Georgios Kopanas*, Thomas Leimkühler, George Drettakis (*indicates equal contribution)�ACM Transactions on Graphics, 2023

Daniel Alexander (alexdan@seas.upenn.edu)

September 9, 2024