1 of 35

Differentiable Rendering: Overview

2020-09-18

Presenter: Frieda Rong

CogAI Reading Group

2 of 35

Modeling the physical world: the rise of 3D vision (3DV)

Today’s talk: Kato et. al. [2006.12057] Differentiable Rendering: A Survey

3 of 35

Integrating graphics and computer vision

1998

2019

differentiable rendering

4 of 35

Differentiable rendering

5 of 35

Applications of differentiable rendering

Inverse rendering

Estimating scene parameters
Pose optimization

Reconstruction

Novel view synthesis

Rerendering, relighting

6 of 35

Overview of algorithms

Point cloud

Lack of surface information
Lightweight
Too sparse

Differentiable Point Clouds
SynSin
DSS
DRWR

Mesh

Common representation format
Good tradeoff between cost and quality

DiffRT/redner
OpenDR
NMR
SoftRas
DIB-R

Voxel

Supports 3D convolutions
Memory inefficient

RenderNet
DRC
VON, HoloGAN
DeepVoxels
Neural Voxel Renderer

Implicit functions

Flexible representation
Harder to model discontinuities

DIST
SRNs
Differentiable Volumetric Rendering
NeRF
IDR

7 of 35

[01] Meshes

Most common representation

Vertices and faces (usually triangles)

Model vertex-specific properties

Use interpolation to determine values within face

Can predict deformations from a canonical or category-specific mesh

Like predicting residuals in linear regression

8 of 35

[DiffRT/redner] Differentiable Monte Carlo Ray Tracing through Edge Sampling (SIGGRAPH 2018)

Analytic derivatives

Edge sampling to handle discontinuities

Monte Carlo ray tracing

Tzu-Mao Li, Miika Aittala, Fredo Durand, Jaakko Lehtinen (MIT, Aalto University, NVIDIA)

9 of 35

[NMR] Neural 3D Mesh Renderer (CVPR 2018)

Approximate gradients

Hand-designs gradients for backward pass

Used in Learning Category-Specific Mesh Reconstruction from Image Collections

Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada (University of Tokyo)

10 of 35

[NMR] Neural 3D Mesh Renderer (CVPR 2018)

11 of 35

SoftRasterizer (ICCV 2019 Oral)

Approximate rendering

Replaces non-differentiable rasterization operation with probabilistic generalization + differentiable aggregation

Allows long-range propagation of gradients

Shichen Liu, Tianye Li, Weikai Chen and Hao Li (USC)

12 of 35

SoftRasterizer (ICCV 2019 Oral)

describes the influence of face j on pixel i

color map

background

For silhouettes:

where

and control sharpness of probability map and aggregation respectively

Probability Map Computation

Aggregation

13 of 35

SoftRasterizer (ICCV 2019 Oral)

3D Unsupervised Single-view Mesh Reconstruction

Pose Optimization

Mesh Deformation

Non-rigid Shape Fitting

14 of 35

Loss functions and regularizers

Photometric loss: L1 (typically), adversarial visual appearance

Silhouette loss mask

Mesh smoothness flattening effect

Mesh Laplacian averaging/smoothing effect

Chamfer distance point cloud loss

Edge length regularization, Normal loss, etc.

Refs: [1804.01654] Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images, [1901.05567] Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstructi on

15 of 35

[DIB-R] Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (NeurIPS 2019)

Approximate rendering

Decomposes rasterization-based rendering into foreground (interpolation-based) and background (similar to SoftRas)

Supports multiple lighting models (none, Lambertian reflectance, Phong shading, Spherical Harmonics)

Wenzheng Chen, Jun Gao^*, Huan Ling^*, Edward J. Smith^*, Jaakko Lehtinen, Alec Jacobson, Sanja Fidler (NVIDIA)

16 of 35

[DIB-R] Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (NeurIPS 2019)

use Barycentric interpolation

use probabilistic influence + aggregation similar to SoftRas to get a silhouette (alpha channel)

describes the influence of face j on pixel i

is the aggregated alpha channel / silhouette opacity; same as SoftRas silhouette aggregation

Foreground pixels (covered by a face)

Background pixels (not covered by any face)

Lighting

Phong shading

Lambertian reflectance

Spherical Harmonics

Loss

17 of 35

[DIB-R] Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (NeurIPS 2019)

Differentiable vertex attributes and rendering models

DIB-R

NMR

w/ lighting

texture only

lighting (Lambertian)

18 of 35

[02] Voxels

“Volume” “pixel”
Occupancy grid of probabilities
Supports 3D convolutions
Memory inefficient

Oct-trees, Binary space partitioning (BSP) trees

Can extract mesh using Marching Cubes algorithm

19 of 35

[DRC] Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency (CVPR 2017)

Inspiration from “A Theory of Shape by Space Carving”

Model probabilistic ray termination events

Rays should be consistent with available observations

Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, Jitendra Malik (UC Berkeley)

20 of 35

[DRC] Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency (CVPR 2017)

Event probabilities

Event cost functions

21 of 35

[DRC] Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency (CVPR 2017)

22 of 35

DeepVoxels: Learning Persistent 3D Feature Embeddings (CVPR 2019 Oral)

Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer (Stanford University, Technical University of Munich, Princeton University)

23 of 35

Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Tool (CVPR 2020)

Assumes voxel input

Learned neural rendering and re-rendering

Rasterization, shading

Uses RenderNet as the backbone

Konstantinos Rematas, Vittorio Ferrari (Google)

24 of 35

[03] Point Clouds

Lightweight

Need to deal with sparsity

Splatting
Use some radius (hyperparameter)

25 of 35

SynSin: End-to-end View Synthesis from a Single Image (CVPR 2020)

Olivia Wiles, Georgia Gkioxari, Richard Szeliski, Justin Johnson (University of Oxford, Facebook AI Research, Facebook, University of Michigan)

26 of 35

[DSS] Differentiable Surface Splatting for Point-based Geometry Processing (SIGGRAPH 2019)

High-fidelity differentiable renderer for point clouds; approximated gradients

Forward splatting based on Elliptical Weighted Average

Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, Olga Sorkine-Hornung (ETH Zurich, Disney Research)

27 of 35

[04] Implicit functions

Typically: signed distance function describes distance to nearest surface

Isosurface of the shape = \{ x\in\R^3 : f(x) = 0 \}

Sometimes other variations, eg. surface is at not
May output other properties, eg. color [NeRF]

Can extract mesh via Marching Cubes algorithm; can obtain implicit function from normals via (Screened) Poisson Surface Reconstruction

Fine detail Discontinuities

Computational cost - querying a point involves an entire NN inference pass

Employ neural rendering or differentiable rendering

28 of 35

[04] Implicit functions

Recent trend towards implicit representations

Deep Equilibrium Networks
Gradient Origin Networks
Diffusion probabilistic models, energy based models
Slides by yours truly

29 of 35

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

(No rendering; uses ground truth)

Samples points and evaluates implicit function, texture losses

Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, Hao Li (USC, UC Berkeley)

30 of 35

DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing (CVPR 2020)

Optimizes over a learned latent shape space (eg. from DeepSDF)

Makes rendering differentiable by formulating differentiable sphere tracing

Employs a number of technical tricks to improve performance

Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, Zhaopeng Cui (ETH Zurich, Google, Tsinghua University, Peking University, Microsoft)

31 of 35

Sphere tracing

March along the ray until convergence

32 of 35

Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision (CVPR 2020)

Represent surfaces as 3D occupancy fields

Numerically find surface intersection for each ray

Calculate exact derivatives

Predict diffuse color for each point using neural 3D texture field

Michael Niemeyer, Lars Mescheder, Michael Oechsle, Andreas Geiger (Max Planck Institute for Intelligent Systems, University of Tubingen, Amazon, Bosch Group)