1 of 55

A tutorial on Gaussian Splatting

Vitor Pereira Matias and Daniel Perazzo

​

Vinícius da Silva, Alberto Raposo, Luiz Velho, Afonso Paiva, Tiago Novello

2 of 55

Problem Statement

  • Given set of views
    • Reconstruct 3D Object
  • We will use gaussians!
  • 3D Gaussian Splatting (3DGS)

2

3 of 55

Problem Statement

  • Given set of views
    • Reconstruct 3D

Salvador

​

zumbi dos palmares

Elevador Lacerda (esquerda)

3

4 of 55

Impact

  • ​

4

Superman (2025)

3DGS Citations

5 of 55

Impact

  • Great companies
    • Meta
    • Google
    • NVIDIA
  • Startups
    • Luma-AI

5

6 of 55

Representation

  • What type of representation?
    • Mesh?
    • Point cloud?

6

7 of 55

Representation

7

8 of 55

Representation

  • We will use Gaussians
    • Volumetric!
    • **Point-based**

8

9 of 55

Overview

9

Volume

Renderer

10 of 55

Outline

  • Preliminaries
  • Volume Rendering
  • EWA Splatting
  • 3D Gaussian Splatting
  • Extensions and applications

10

11 of 55

Camera

11

12 of 55

Structure from Motion

  • We don’t have the camera
  • Use SfM!
    • COLMAP
  • Obtain camera parameters
    • Coarse point cloud

12

Johannes et al. Structure-from-motion revisited. CVPR 2016.

Set of unposed images

13 of 55

Structure from Motion

Johannes et al. Structure-from-motion revisited. CVPR 2016.

13

Set of unposed images

3D representation of the scene

14 of 55

DEMO

Johannes et al. Structure-from-motion revisited. CVPR 2016.

14

Set of unposed images

3D representation of the scene

15 of 55

Volume Rendering

  • Using volumetric representation
    • Volume Rendering

Drebin et al. Volume rendering. SIGGRAPH 1988

15

16 of 55

Volume Rendering

  • Using volumetric representation
    • Volume Rendering

16

Fong et al. Production volume rendering. SIGGRAPH 2017 course

17 of 55

Volume Rendering Definitions

Drebin et al. Volume rendering. SIGGRAPH 1988

17

  • Find color at pixel p
  • Analyze ray passing through it:

18 of 55

Volume Rendering Definitions

Max. Optical models for direct volume rendering. TVCG 2002

18

Emission-absorption model

​

What is the light intensity of ray?

​

​

​

​

19 of 55

Volume Rendering Definitions

​

​

Max. Optical models for direct volume rendering. TVCG 2002

19

20 of 55

Volume Rendering Equation

20

21 of 55

Volume Rendering Equation

21

22 of 55

Volume Rendering Equation

  • A little different
    • World to camera
    • However…

22

23 of 55

Volume Rendering Equation

  • From Scalar Line Integrals…
    • We can invert the rays!
      • Principle behind path tracing

23

24 of 55

Volume Rendering Equation

  • From Line Integrals…
    • We can invert the rays!
      • Principle behind path tracing
  • Obtain final equation:

24

25 of 55

Volume Rendering Equation

25

26 of 55

Volume Rendering Quadrature

​

​

26

27 of 55

Volume Rendering Quadrature

​

​

27

28 of 55

Volume Rendering Quadrature

​

​

  • Idea behind NeRFs!
    • Model with MLPs

Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM.

28

29 of 55

Volume Rendering Quadrature

​

​

  • Idea behind NeRFs!
    • Model with MLPs
    • Increasingly faster! (but slow)

Mülleret al. Instant neural graphics primitives with a multiresolution hash encoding. SIGGRAPH 2022.

29

30 of 55

Volume Rendering Quadrature

​

​

  • Started optimizing representation
    • Hashgrids

Mülleret al. Instant neural graphics primitives with a multiresolution hash encoding. SIGGRAPH 2022.

30

31 of 55

Volume Rendering Quadrature

​

​

  • Optimized samplers
    • Mipnerf
    • NeRFAcc
    • Zipnerf

Barron al. Zip-nerf: Anti-aliased grid-based neural radiance fields. CVPR 2023.

31

32 of 55

Volume Rendering Quadrature

​

​

  • However… still is slow
    • Specially on big scenes
    • So… what is possible to do?
  • Biggest problem is volume rendering
    • Slow to solve
    • Dependent on sampling

Barron al. Zip-nerf: Anti-aliased grid-based neural radiance fields. CVPR 2023.

32

FPS

33 of 55

EWA Splatting Discretization

  • Discretize integral
    • Global modelling
      • Expensive!

​

  • How to solve?
    • Discretize object space

Zwicker, et al. EWA splatting. TVCG 2002.

33

34 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

  • Use 3D gaussians as discretization

​

​

​

  • Assume non-overlapping gaussians
  • Constant colors

​

Zwicker, et al. EWA splatting. TVCG 2002.

34

Mean:

Covariance:

Opacity:

Color :

Define the new density

35 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

​

​

​

Substituting for one gaussian

Zwicker, et al. EWA splatting. TVCG 2002.

35

36 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

​

​

​

Substituting for K gaussians

Zwicker, et al. EWA splatting. TVCG 2002.

36

37 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

​

​

​

Substituting for K gaussians

Zwicker, et al. EWA splatting. TVCG 2002.

37

38 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

​

​

​

Substituting for K gaussians

Zwicker, et al. EWA splatting. TVCG 2002.

38

39 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

​

How to solve this integral faster?

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

39

40 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

​

How to solve this integral faster?

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

40

41 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

We are projecting the gaussians onto the screen!

  1. From object space to cam space

Affine Transform

  • Perspective transformation

Not affine

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

41

After

Before

42 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

We want to project the gaussians onto the screen!

  • From object space to cam space

Affine Transform

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

42

Gaussian property!

43 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

We want to project the gaussians onto the screen!

  1. ​
  2. Perspective transformation

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

43

Make it affine

44 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

We want to project the gaussians onto the screen!

  • ​
  • ​

​

Apply both:

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

44

Before

After

45 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

We want to project the gaussians onto the screen!

  • ​
  • ​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

45

Integrate over this axis

46 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

We want to project the gaussians onto the screen!

  • ​
  • ​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

46

Integrate over this axis

47 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

We want to project the gaussians onto the screen!

  • ​
  • ​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

47

Integrate over this axis

Actually we do not need to integrate!

Truncate J to 2x3

48 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

We want to project the gaussians onto the screen!

  • ​
  • ​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

48

We now have a bunch of 2D gaussians on the screen plane

​

​

​

Their contributions is just their value at a pixel

Actually we do not need to integrate!

Truncate J to 2x3

49 of 55

EWA Splatting Discretization

Our volume rendering equation

​

​

bunch of 2D gaussians on the screen plane

​

​

​

Sort them by depth respective to camera center

​

Evaluate each at pixel

​

Perform Alpha blending

​

​

​

​

​

​

​

​

​

​

​

Zwicker, et al. EWA splatting. TVCG 2002.

49

Sorted

50 of 55

EWA Splatting Aliasing

  • We can already render image
  • The problem is already solved
    • Actually no
    • Aliasing problems
    • Low sampling rate
    • Solution: pre-filter high frequencies

50

51 of 55

EWA Splatting Aliasing

  • We can already render image
  • The problem is already solved
    • Actually no
    • Aliasing problems
    • Low sampling rate
    • Solution: pre-filter high frequencies

​

Gaussian filter to the projected gaussians

Is the same as updating

​

They do not update Mip-Splatting does

51

52 of 55

EWA Splatting Pipeline

  • This method is extremely fast
  • We project the gaussians + evaluate each pixel
  • Each pixel can be computed in parallel

​

  • Similar to common graphics pipelines
    • That rely on projection
  • Open to rasterization procedures

52

53 of 55

EWA Splatting Results

  • Get nice results from Gaussians

53

54 of 55

Overview

54

EWA

Splatting

!!!!!!!!

This is fully differentiable (?)

55 of 55

A tutorial on Gaussian Splatting

Vitor Pereira Matias and Daniel Perazzo

​

Vinícius da Silva, Alberto Raposo, Luiz Velho, Afonso Paiva, Tiago Novello