Volume Rendering
1
Volume Rendering
2
Where we are
2:00 - 2:30 Birds Eye View & Background - Angjoo Kanazawa
2:30 - 3:00: Deep Dive into the Volumetric Rendering Function - Ben Mildenhall
3:00 - 3:30: Encoding and Representing 3D Volumes - Matt Tancik
3:30 - 4:00: 30 min break ☕️
4:00 - 4:30: Signal Processing Considerations - Pratul Srinivasan
4:30 - 5:00: Challenges - Angjoo Kanazawa
5:00 - 5:30: Tutorial + Q&A
Neural Volumetric Rendering
4
Neural Volumetric Rendering
5
computing color along rays through 3D space
What color is this pixel?
Neural Volumetric Rendering
6
continuous, differentiable rendering model without concrete ray/surface intersections
Neural Volumetric Rendering
7
using a neural network as a scene representation, rather than a voxel grid of data
Scene properties
Cameras and rays
8
Cameras and rays
9
Camera
Ray
Pixel
Coordinate frames + Transforms: world-to-camera
World coordinates
Camera coordinates
Image coordinates
Figure credit: Peter Hedman
Extrinsics (R, T)
Orientation + Location of the camera in the World
Intrinsics (K)
How the camera maps a point in 3D to image
Coordinate frames + Transforms: camera-to-world
World coordinates
Camera coordinates
Image coordinates
Figure credit: Peter Hedman
Extrinsics (R, T)
Orientation + Location of the camera in the World
Intrinsics (K)
How the camera maps a point in image to 3D
Camera pose - pixel to camera
12
Y
X
Z
Camera pose - pixel to camera
13
Y
X
Z
3D view
Top view (looking along Y)
Side view (looking along X)
Camera pose - pixel to camera
14
Y
X
Z
3D view
Top view (looking along Y)
Side view (looking along X)
X
Z
Z
Y
Camera pose - pixel to camera
15
Y
X
Z
3D view
Top view (looking along Y)
Side view (looking along X)
distance =
distance =
Camera pose - pixel to camera
16
Y
X
Z
3D view
Top view (looking along Y)
Side view (looking along X)
Camera pose - pixel to camera
17
Y
X
Z
3D view
Top view (looking along Y)
Side view (looking along X)
Pixel (i, j)
distance =
distance =
Coordinates are in pixel space
Camera pose - intrinsic
18
Y
X
Z
3D view
Top view (looking along Y)
Side view (looking along X)
Pixel (i, j)
If image maps to then this pixel sits at where is the image resolution.
Camera pose - pixel to camera
19
Y
X
Z
3D view
Top view (looking along Y)
Side view (looking along X)
Pixel (i, j)
distance =
distance =
Recenter using pixel coordinates of image center
Camera pose - pixel to camera
20
Y
X
Z
3D view
Top view (looking along Y)
Side view (looking along X)
Pixel (i, j)
Rescale frustum by focal length so that image plane is at distance 1
distance =
distance =
Camera pose - intrinsic
21
Y
X
Z
3D view
Top view (looking along Y)
Side view (looking along X)
Pixel (i, j)
Image maps to where focal length controls the amount of “zoom"
Camera pose - pixel to camera
22
Full mapping is to get 3D coordinates for a point on the image plane.
Camera space ray points from origin toward this point.
Y
X
Z
Camera pose - pixel to camera
23
Camera pose - camera to world
24
Apply rigid transformation
Calculating points along a ray
25
Scalar controls distance along the ray
Neural Volumetric Rendering
26
Neural Volumetric Rendering
27
continuous, differentiable rendering model without concrete ray/surface intersections
Surface vs. volume rendering
28
Ray
Camera
Scene representation
Want to know how ray interacts with scene
Surface vs. volume rendering
29
Ray
Camera
Scene representation
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
Surface rendering — loop over geometry, check for ray hits
Surface vs. volume rendering
30
Ray
Camera
Scene representation
Volume rendering — loop over ray points, query geometry
?
?
?
?
?
History of volume rendering
31
Early computer graphics
32
Kajiya 1984, Ray Tracing Volume Densities
Chandrasekhar 1950, Radiative Transfer
Ray tracing simulated cumulus cloud [Kajiya]
Alpha compositing
33
Porter and Duff 1984, Compositing Digital Images
Alpha compositing [Porter and Duff]
Volume rendering for visualization
34
Levoy 1988, Display of Surfaces from Volume Data
Max 1995, Optical Models for Direct Volume Rendering
Kajiya 1984, Ray Tracing Volume Densities
Chandrasekhar 1950, Radiative Transfer
Porter and Duff 1984, Compositing Digital Images
Medical data visualisation [Levoy]
Volume rendering for surfaces
Geometry and materials can be stored per-voxel and used with standard surface rendering methods
35
Computer vision — space carving?
36
Power of Analysis-by-Synthesis
Kultulakos and Seitz, A Theory of Shape by Space Carving IJCV 2000
Volume rendering derivations
38
Real graphics formulation for path tracing
39
40
Slide credit: Novak et al 2018, Monte Carlo methods for physically based volume rendering
http://commons.wikimedia.org
Absorption
http://wikipedia.org
Scattering
Emission
Simplify
41
Slide credit: Novak et al 2018, Monte Carlo methods for physically based volume rendering
http://commons.wikimedia.org
Absorption
http://wikipedia.org
Scattering
Emission
Probabilistic interpretation
42
Derivation of T, density eqns
43
Quadrature
44
Weight pdf is important quantity
45
Trivially differentiable
46
Expected value of non-color quantities (depth, etc)
47
Other stats of weight pdf (median depth?)
48
Volumetric formulation for NeRF
49
Scene is a cloud of tiny colored particles
Max and Chen 2010, Local and Global Illumination in the Volume Rendering Integral
Volumetric formulation for NeRF
50
If a ray traveling through the scene hits a particle at distance along the ray, we return its color
Camera
Ray
What does it mean for a ray to “hit” the volume?
51
This notion is probabilistic: chance that ray hits a particle in a small interval around is .
is called the “volume density”
Probabilistic interpretation
52
To determine if is the first hit along the ray, need to know : the probability that the ray makes it through the volume up to .
is called “transmittance”
Probabilistic interpretation
53
The product of these probabilities tells us how much you see the particles at :
Probabilistic interpretation
54
To determine if is the first hit along the ray, need to know : the probability that the ray doesn’t hit any particles earlier.
is called “transmittance”
We assume is known and want to use it to calculate
Calculating given
55
If is known, T can be computed… How?
Calculating given
56
and are related by the probabilistic fact that
Calculating transmittance
57
and are related by the probabilistic fact that
Calculating transmittance
58
and are related by the probabilistic fact that
Solve for
59
Split up differential
Rearrange
Integrate
Exponentiate
Solve for
60
Rearrange
Integrate
Exponentiate
Taylor expansion for T
Solve for
61
Taylor expansion for T
Rearrange
Integrate
Exponentiate
Solve for
62
Rearrange
Taylor expansion for T
Integrate
Exponentiate
Solve for
63
Taylor expansion for T
Rearrange
Integrate
Exponentiate
Solve differential equation
64
Taylor expansion for T
Rearrange
Integrate
Exponentiate
Density is score function!
65
An interesting observation is that density is actually the negative score function of :
This is why it doesn’t need normalization!
Volumetric formulation for NeRF
66
Volumetric formulation for NeRF
67
Food for thought #1: for a constant density medium and , we have
,
which is the exponential distribution CDF
Volumetric formulation for NeRF
68
Food for thought #2: From our derivation,
,
so acts as the score function of transmittance.
Interesting: depends on the current camera ray, but does not.
PDF for ray termination
69
Finally, we can compute the probability that a ray terminates at :
This is the product P[ray hits nothing before ] x P[ray hits something in interval ]
PDF for ray termination
70
Finally, we can compute the probability that a ray terminates at :
This is the product P[ray hits nothing before ] x P[ray hits something in interval ]
(This PDF is important, we will come back to it later)
PDF for ray termination
71
Finally, we can write the probability that a ray terminates at as a function of only sigma
FIX
PDF for ray termination
72
Finally, we can write the probability that a ray terminates at as a function of only sigma
PDF for ray termination
73
Finally, we can write the probability that a ray terminates at as a function of only sigma.
Note: the corresponding CDF is
(probability that the ray does hit something between 0 and )
Expected value of color along ray
74
This means the expected color returned by the ray will be
Note the nested integral!
Approximating the nested integral
75
We use quadrature to approximate the nested integral,
Approximating the nested integral
76
We use quadrature to approximate the nested integral,
splitting the ray up into segments with endpoints
Approximating the nested integral
77
We use quadrature to approximate the nested integral,
splitting the ray up into segments with endpoints
with lengths
Approximating the nested integral
78
We assume volume density and color are roughly constant within each interval
Deriving quadrature estimate
79
This allows us to break the outer integral into a sum of analytically tractable integrals
Deriving quadrature estimate
80
This allows us to break the outer integral into a sum of analytically tractable integrals
Deriving quadrature estimate
81
Caveat: piecewise constant density and color do not imply constant transmittance!
Deriving quadrature estimate
82
Caveat: piecewise constant density and color do not imply constant transmittance!
Important to account for how early part of a segment blocks later part when is high
Evaluating for piecewise constant density
83
For ,
We need to evaluate at continuous values that can lie partway through an interval
Evaluating for piecewise constant density
84
For ,
“How much light is blocked by all previous segments?”
Evaluating for piecewise constant density
“How much light is blocked partway through the current segment?”
For ,
Deriving quadrature estimate
86
Deriving quadrature estimate
87
Substitute
Deriving quadrature estimate
88
Integrate
Deriving quadrature estimate
89
Cancel
Connection to alpha compositing
90
segment opacity
Connection to alpha compositing
91
segment opacity
Connection to alpha compositing
92
Connection to alpha compositing
93
Connection to alpha compositing
94
Summary: volume rendering integral estimate
Rendering model for ray :
How much light is blocked earlier along ray:
How much light is contributed by ray segment i:
95
3D volume
Camera
Ray
colors
weights
Volume rendering is trivially differentiable
Rendering model for ray :
How much light is blocked earlier along ray:
How much light is contributed by ray segment i:
96
3D volume
Camera
Ray
colors
weights
differentiable w.r.t.
Further points on volume rendering
97
Alpha mattes and compositing
98
Alpha mattes and compositing
99
Alpha mattes and compositing
100
Alpha mattes and compositing
101
FIX MAKE GRAPHICS - 2 clouds and box out two halves of ray
Can compute accumulated weights over some interval along ray to get an alpha mask for compositing part of the NeRF together with other assets:
If the NeRF is an isolated object in space, then can be the full ray.
Alpha mattes and compositing
102
Mildenhall*, Srinivasan*, Tancik* et al 2020, NeRF
Poole et al 2022, DreamFusion
Tang et al 2022, Compressible-composable NeRF via Rank-residual Decomposition
Rendering weight PDF is important
103
Remember, expected color is equal to
and are “rendering weights” — probability distribution along the ray
(continuous and discrete, respectively)
Visual intuition — rendering weights not just 3D function
104
3D volume
Camera
Ray
104
Rendering weights are not a 3D function — depends on ray, because of tranmisttance!
Visual intuition — rendering weights not just 3D function
105
3D volume
Camera
Ray
105
Rendering weights are not a 3D function — depends on ray, because of tranmisttance!
Rendering weight PDF is important — depth
106
We can use this distribution to compute expectations for other quantities, e.g. “expected depth”:
This is often how people visualise NeRF depth maps.
Alternatively, other statistics like mode or median can be used.
Rendering weight PDF is important — depth
107
Mean depth
Median depth
Rendering weight PDF is important — depth
108
Mean depth
Median depth
Volume rendering other quantities
109
This idea can be used for any quantity we want to “volume render” into a 2D image. If lives in 3D space (semantic features, normal vectors, etc.)
can be taken per-ray to produce 2D output images.
Volume rendering other quantities
110
Kobayashi et al 2022, Decomposing NeRF for Editing via Feature Field Distillation
Various recent works have used this idea to render higher-level semantic feature maps (e.g., Feature Field Distillation and Neural Feature Fusion Fields).
Density as geometry
111
Normal vectors (from analytic gradient of density)
Note about density being 3d but trans being light-field equiv?
112
Eg autoint is light field
Applications/optimizing differentiable volume rendering
113
Alpha compositing model in ML/computer vision
114
Tulsiani et al 2017, Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency
Differentiable ray consistency work used a forward model with “probabilistic occupancy” to supervise 3D-from-single-image prediction.
Same rendering model as alpha compositing!
2015s CV/ML (Neural vols, multiplane imgs)
115
Volume rendering for view synthesis
116
Neural Volumes
(Lombardi et al. 2019)
Direct gradient descent to optimize an RGBA volume, regularized by a 3D CNN
Multiplane image methods
Stereo Magnification (Zhou et al. 2018)
Pushing the Boundaries… (Srinivasan et al. 2019)
Local Light Field Fusion (Mildenhall et al. 2019)
DeepView (Flynn et al. 2019)
Single-View… (Tucker & Snavely 2020)
Typical deep learning pipelines - images go into a 3D CNN, big RGBA 3D volume comes out
Next: how to represent scene -> Matt
117