Differentiable Rendering

黃慧光

Homepage: https://shuangz.com/projects/psdr-sg20/

• Rendering pipeline
• OpenDR(2014)
• Soft Rasterizer(2019)
• pytorch3D(2020)

​

​

Rendering pipeline

https://learnopengl-cn.github.io/01%20Getting%20started/04%20Hello%20Triangle/

Rasterization

• “The process of finding all the pixels in an image that are occupied by a geometric primitive is called rasterization”

​

Triangle Rasterization

https://zhuanlan.zhihu.com/p/449289345

Line drawing

Test and Blending : Z-buffering

OpenDR:an approximate differentiable renderer�(ECCV 2014)

GT

GT& initial scene

difference

final result

Forward rendering

Appearance (A)：per-vertex brightness

Geometry (V )：vertex locations

Camera (C)：camera parameters

​

​

​

intermediate variable

Projection coordinates（U）： The coordinates of the vertex in screen space after coordinate transformation (can be understood as the output of the vertex shader)

​

https://zhuanlan.zhihu.com/p/584510853

https://zhuanlan.zhihu.com/p/631670088

Limitations:

1. Inconsistency between forward and reverse processes

​

2.Unable to pass gradient into occluded triangle

​

3. Vertices can only receive gradients from adjacent

pixels within a short distance

https://zhuanlan.zhihu.com/p/585346760

Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning�(ICCV 2019)

https://zhuanlan.zhihu.com/p/585346760

https://zhuanlan.zhihu.com/p/540768426

Differentiable formulation

Probability Map Computation

​

​

​

​

​

​

• Probability maps of a triangle under Euclidean metric. (a) definition of pixel-to-triangle distance; (b)-(d) probability maps generated with different σ.

Aggregate Function

Experiments: Image-based 3D Reasoning

Single-view Mesh Reconstruction�

Color Reconstruction

​

• Instead of directly regressing the color value, our color generator formulates color reconstruction as a classification problem that learns to reuse the pixel colors in the input image for each sampling point.

​

​

​

​

​

Color Reconstruction

​

​

​

Accelerating 3D Deep Learning with PyTorch3D�(2020 Facebook AI Research )

3D operators

Chamfer loss is a common metric that quantifies agreement between point clouds P and Q. Formally

​

​

​

​

PyTorch3D avoids this inefficiency (and supports heterogeneity) by using our efficient KNN to compute neighbors. Figure 1a compares ours against the naïve approach with B = 32, |P | = 1000, and varying |Q|. The naïve approach runs out of memory for |Q| > 10k, while ours scales to large point clouds and reduces time and memory use by more than 12×.

https://medium.com/maochinn/pytorch3d-%E4%BB%8B%E7%B4%B9-4dae8b8db6ad

Differentiable mesh renderer

​

​

​

Rasterizer

Pytorch3D’s rasterizer departs from [Soft Rasterizer] in three ways to improve efficiency and modularity.

• In [Soft Rasterizer], pixels are influenced by every face they intersect in the xy-plane; in contrast we constrain pixels to be influenced by only the nearest K faces along the z-axis, computed using per-pixel priority queues.

​

​

Shaders

This design is highly modular, as users can easily implement new shaders to customize the renderer.

​

Performance

​

Differentiable point cloud renderer

PyTorch3D also provides an efficient and modular point cloud renderer following the same design as the mesh renderer.

Each point is splatted to a circular region in screen-space whose opacity decreases away from the region's center

In our experiments we consider two blending methods: Alpha-compositing and Normalized weighted sums

​

​

​

​

​

Alpha-compositing uses the depth ordering of points , Norm ignores the depth order

Experiments

Thanks