Differentiable Rendering

Oct 17

Lecturer

Wei-Cheng Huang

Modular Primitives for High-Performance Differentiable Rendering (NVdiffrast)

Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, Timo Aila

​

ACM Transactions on Graphics 39(6) (proc. SIGGRAPH Asia 2020)

Lecture Adapted from NVdiffrast Presentation @ SIGGRAPH Asia 2020

Start with Rendering in General …

Why Differentiable Rendering？

Must be differentiable !

Two Main-Stream Rendering: Rasterization vs. Ray tracing

​

• Light Transport Simulation /Ray Tracing

• Traces rays to determine visibility and intersection points in the scene
• Rays can go anywhere! All shaders and geometry must be available at the same time

• Rasterization

• Determines visibility by projecting geometry onto a 2D screen and resolving depth for each pixel
• Each object can be a draw call

from: SIGGRAPH 2024 real-time ray tracing tutorial

from: CMU 15-462

Keenan Crane

Two Main-Stream Differentiable Rendering

• Light transport simulation (Ray Tracing)

• redner [Li et. al 2018]
• Mitusba2 [Nimier-David et al. 2019]
• Mitusba3 [Jakob et al. 2022]

Favor quality over speed

• Rasterization

• SoftRas [Liu et. al 2019]
• PyTorch3D [Ravi et al. 2020]
• Nvdiffrast [Laine et al. 2020]

Favor speed over quality

​

​

Coverage (Visibility) Gradient Problem

Z discontinuity problem

XY discontinuity problem

from: PyTorch3D @ SIGGRAPH ASIA 2020 Course

Coverage (Visibility) Gradient Problem

Essentially: Moving a vertex has no 1st order effect on coverage

​

​

Common Tricks: blurring the geometry, make partial transparency, aggregate/ composite in an ad-hoc fashion

Problem: 1. Occlusions become fuzzy 2. must be shaded before composition

​

Shading with Blurred Triangles

Shade internally

​

Shade through memory

​

Composition of transparency and blur just causes overshading

This work

• Pipeline Overview

• Important Primitive Operations

All differentiable and accelerated with CUDA

This work

• Pipeline Overview

• Important Primitive Operations

All differentiable and accelerated with CUDA or hardware graphics pipeline

• Target: efficiency, flexibility (freedom), modularity, quality

• Input:

Vertex positions (homogeneous coordinates) in Clip Space

Triangular index buffer

• Output:

(𝑢, 𝑣, 𝑧/𝑤, triangle ID) per pixel

Jacobian buffer 𝐽 = 𝜕{𝑢, 𝑣 }/𝜕{𝑥,𝑦}

Extra Notes:

Independent of shading, textures etc. (Modularity)

Accelerated using OpenGL hardware graphics pipeline (efficiency)

Backward pass gradient easy to get through chain rule (perspective mapping)

​

​

Dynamic mapping between world coordinates and discrete pixel coordinates.

• Input:

Per-vertex attributes

Triangular index buffer

(𝑢, 𝑣) and Jacobian buffer 𝐽 = 𝜕{𝑢,𝑣}/𝜕{𝑥,𝑦} from rasterization

• Output:

Interpolated attributes per pixel

Extra notes:

barycenter Jacobians is used for screen-space (attribute pixel) derivatives calculation using chain rule

​

​

Backward pass gradient also easy to get (interpolation)

Creating a mapping

between the pixels and the attributes

𝑢 = 𝑢(𝑥,𝑦)，𝑣=𝑣(𝑥,𝑦)， 𝑖th vertex by 𝐴_𝑖 ,

𝐴 = 𝑢𝐴_𝑖0 + 𝑣𝐴_𝑖1 + (1 − 𝑢 − 𝑣)𝐴_𝑖2

𝜕𝐴/𝜕{𝑥,𝑦} = [𝜕{𝑢,𝑣}/𝜕{𝑥,𝑦}][𝜕{𝐴}/𝜕{𝑢,𝑣}]

• Input:

Per-pixel texture coordinates

Texture image (arbitrary # channels)

Attribute pixel derivatives 𝜕𝐴/𝜕{𝑥,𝑦}

• Output:

Sampled texture per pixel

Standard operation in a shading system, Closely resemble attribute interpolation, differs by the multiscale nature, scale level determined by attribute pixel derivatives

​

• Input:

Point sampled images, vertices, triangles

• Output:

Antialiased image

Ket step, solve the coverage gradient problem in this step!

Insights:

1. Antialiasing converts the discontinuities to smooth changes, thus gradients can be computed
2. Modify pixel colors based on geometry

Modify pixel colors based on geometry

Similar idea as Distance-to-Edge Antialiasing (DEAA) [Malan 2010]) and Geometric Post-Process Antialiasing (GPAA) [Persson 2011]

Estimate occlusion ratio based on geometry

Blend in color from neighboring pixel

Extra Notes:

Approximate occlusion ratio, linear function of the location of the crossing point—from zero at midpoint to 50% at pixel center

Essentially approximates the exact surface coverage per pixel [Jalobeanu et al. 2004] using an axis-aligned slab, only perfect for axis aligned geometry

​

​

​

Performance comparison

15 ms/ frame

​

5 s/ frame

​

Qualitative Experiments

visibility gradients provide useful information even for small triangles

Qualitative Experiments

Modular design and flexibility of the framework can handle complex rendering pipeline

Qualitative Experiments

Cube pose optimization: blurring and transparency before shading is not really necessary

SoftRas [Liu et. al 2019]: 63.57°

Avg error:

NVdiffrast: 48.52°

+Two stage optimization: 22.49°

+symmetry noise 2.61°

Applications: Facial Performance Capture

Formulate facial performance capture as an inverse rendering problem:

Find a global texture and per-frame mesh, so that rendering them produces input images

• Input:

sequences from 9 cameras with camera positions

Base mesh

Applications: Facial Performance Capture

Single GPU, 100000 iterations, 1 hour to convergence, better than commercial (DI4D)

Archeologist

Chaitanya

Differentiable rendering

• Fast, correct, generalizable, flexibility, high resolution

Past Works

• Linear combination of select faces

​

• Hard to generalize

• Morphable Model For The Synthesis Of 3D Faces (SIGGRAPH ’99)

​

• Single image

​

• Shape + Texture vectors

Common Variants

High Quality

​

High Performance

• Differentiable Monte Carlo RT through Edge Sampling

​

• Mitsuba 2

• OpenDR

​

• SoftRas

​

• DIB-R

​

• NVDiffRast

Differentiable Monte Carlo RT through Edge Sampling (SIGGRAPH ’18)

Physically based

What is the gradient

When a triangle moves?

approximate integral with MC

Differentiable Monte Carlo RT through Edge Sampling

OpenDR

Limited shading model

Approximate gradients

General purpose

fast

• Local shading

​

• Spatial gradients
• Interior pixel → within object
• Boundary pixel → between objects

What is the gradient of a pixel for when the cylinder is rotated in plane

​

Neural Mesh Rendering (CVPR ’18)

Not scalable

Limited flexibility

Hallucinated gradients

What is the gradient at the pixel p w.r.t to the vertex x ?

Noise free

SoftRas (ICCV ’19)

• differentiable aggregating process

​

• computes influence of each triangle on

each pixel

SoftRas

Good gradients

Noise free

Approximate rendering

Cannot scale well wrt parameters

What is the right blur?

Accuracy comparison wrt NVDiffRast

Comparison

Archeologist:

Impact and Future Work

Ozgur Kara

https://github.com/sicxu/Deep3DFaceRecon_pytorch

Magic3d: High-resolution text-to-3d content creation

Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation

Extracting Triangular 3D Models, Materials, and Lighting From Images (CVPR 2022 Oral) (NVdiffRec)

Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising (Neurips 2022) (NVDiffRecMC)

Differentiable Rendering Packages

• Kaolin
• NVIDIA Kaolin library provides a PyTorch API for working with a variety of 3D representations and includes a growing collection of GPU-optimized operations such as modular differentiable rendering…
• fVDB
• Developed by NVIDIA, fVDB is an open-source deep learning framework for sparse, large-scale, high-performance spatial intelligence. It builds NVIDIA-accelerated AI operators on top of OpenVDB to enable reality-scale digital twins, neural radiance fields, 3D generative AI, and more.
• Mitsuba 3
• Mitsuba 3 is a research-oriented retargetable rendering system, written in portable C++17 on top of the Dr.Jit Just-In-Time compiler. It is developed by the Realistic Graphics Lab at EPFL.

Private Investigator

Guang Yin

Timo Aila

Timo Aila joined NVIDIA Research in 2007 from Helsinki University of Technology, where he led the computer graphics research group. His expertise ranges from real-time rendering in computer games to to high-quality image synthesis, with contributions to the PantaRay rendering system used in Avatar, Tintin, and Hobbit. He also gained expertise in mobile graphics as the chief scientist of Hybrid Graphics, which was acquired by NVIDIA in 2006. Timo is currently working on machine learning, with a special focus on generative models such as StyleGAN. Previously he had a central role in NVIDIA's research efforts on ray tracing, including the design of the RTX hardware units.

​

Computer Graphics

[2000 - Now] Main topic: occlusion culling, ray tracing

Machine Learning

[2017 - Now] Main topic: Generative Adversarial Networks

GPU Hardware Design

[2000] [Real-time Graphics]

Master’s thesis ➡️ the first commercial occlusion culling library Umbra

[2000] [Real-time Graphics]

Master’s thesis ➡️ the first commercial occlusion culling library Umbra

[2009] [Ray Tracing]

In 2007, joined NVIDIA Research and led the computer graphics research group

How GPUs can be optimized for ray tracing

[2010] [Ray Tracing]

Simple!

Oh no…

Bounding Volume Hierarchies (BVH)

Used in Avatar, Tintin, and the Hobbit

[2016] [Machine Learning]

Computer Graphics

Machine Learning

[2017] [Machine Learning for Computer Graphics]

Lead to Deep learning super sampling (DLSS)?

[2017] [Machine Learning] [8676 citations]

[2019] [Machine Learning] [12062 citations]

Makes GAN practical

Widespread use in creative industries

[2020] [Computer Graphics for Machine Learning]

Computer Graphics

Machine Learning

GPU hardware design

Highly optimized differentiable rendering pipeline

Timo Aila

Broad and deep understanding of the whole system

Computer Graphics

Machine Learning

GPU Hardware Design

Diverse research directions provide a variety of tools to address bottlenecks across different domains.

​

Industrial Applications matter!

Critic

Harris Nisar

Raytracing vs. Rasterization

Raytracing and Global Illumination Intro. to Computer Graphics, CS180, Fall 2008 UC Santa Barbara – https://images.slideplayer.com/25/7723008/slides/slide_2.jpg

Raytracing vs. Rasterization

Raytracing and Global Illumination Intro. to Computer Graphics, CS180, Fall 2008 UC Santa Barbara – https://images.slideplayer.com/25/7723008/slides/slide_2.jpg

Resources

Renderer

Image

https://youtu.be/brDJVEPOeY8?si=mycNk7T7Wb1xJxZ9

Rasterization at a high level

Rasterization

Step 1: World to Screen Space + Clipping

transforming 3D object vertices from object space to screen space

Limitations

• Based on camera’s view; can not handle off-screen reflections or indirect lighting.
• Triangle number must be very high to approximate smooth curves.

https://imagecomputing.net/old_storage/old_website2/teaching/2019_2020/semester_1/INF630_refresher/class/content/014_projection_rasterization/projection.svg

https://img.clipart-library.com/2/clip-computer-graphicss/clip-computer-graphicss-1.jpg

Rasterization

Step 2: Triangles -> Pixels

• Which pixels are covered by which triangle (scan-line algo)
• Which triangles are in front (z buffer)

Limitations

• Aliasing
• Cannot handle transparency
• Complex geo -> slower performance

​

https://youtu.be/brDJVEPOeY8?si=mycNk7T7Wb1xJxZ9

https://www.salomonsson.se/img/math_rasterizer_what_is.png

Rasterization

Step 3: Shading

• Applying:
• Color
• Texture
• Lighting

Limitations

• Inability to handle global illumination
• Inability to handle shadows and reflections (requires tricks like shadow maps and screen space reflections)

​

​

https://www.fragmentstorm.com/assets/images/2019-03-17-overview-of-the-graphics-pipeline/fragment-shading.png

https://miro.medium.com/v2/resize:fit:900/0*lOiF1XoVkXlWYvrB.jpg

https://learnopengl.com/Advanced-Lighting/Shadows/Shadow-Mapping

https://sugulee.wordpress.com/2021/01/16/performance-optimizations-for-screen-space-reflections-technique-part-1-linear-tracing-method/

• Rasterization is:
• Fast 😀
• But inaccurate 😢

• Rasterization is:
• Fast 😀
• But inaccurate 😢
• This is definitely a “limitation” but I would not call it a full blown critique – the authors acknowledge them and deciding on using this system should be based on your application requirements 🤔

​

Discontinuities rendering pipeline

Geometric Post-process Anti-Aliasing (GPAA)

Graduate Student

Possible Improvements And Applications

Junkun Chen

​

https://docs.google.com/presentation/d/1FbA29d21cz8CczlFYq-xqqgplqQn2J1hi3mT0xdBd8Y/edit?usp=sharing

Idea of Improvements

• “Rasterization” → 3D Gaussian Splatting

• “Meshes” → Tetrahedron → Differentiable Marching Tetrahedra

Idea 1: Apply with advanced approaches from 3DGS

• NVDiffRast cannot, but 3DGS can:
• Cannot deal with semi-transparency
• Cannot dynamically add or delete vertices, can only move
• No native anti-aliasing
• Needs lots of “soften” operations to make it differentiable

Idea 1: Apply with advanced approaches from 3DGS

• First, enable transparency/volume density through alpha maps
• Combine mesh triangles with Gaussians
• For each edge, model the opacity as a Gaussian distribution w.r.t. dist. from the edge
• For each triangle surface, similarly model as a Gaussian w.r.t. dist. from the surface
• No need to “soften” any of the occlusion/disocclusion operations
• Each surface can be regarded as a “Gaussian” centered at the triangle

Idea 1: Apply with advanced approaches from 3DGS

• Traditional idea needs “differentiable sorting” to render transparency
• But 3DGS does not need to do this
• Split the screen into tiles, radix-sort Gaussians in each tile w/o gradients
• Directly apply volume rendering formula in this order
• Apply similar methods here
• Achieve transparency in a similar way as Gaussian

Idea 1: Apply with advanced approaches from 3DGS

• 3DGS natively supports anti-aliasing via the Gaussian distribution
• Also apply 3DGS’ culling and creation to optimize vertices and surfaces
• Remove surfaces with too high transparency
• If one edge is too long, create a vertex at middle point
• If one edge is too short, combine their side vertices

Idea 2: Geometry manipulation via Tetrahedra

• NVDiffRast does not support topology change
• Cannot add surfaces into the mesh
• Cannot create or fill-up holes along with training
• Even with 3DGS augmentation (surface deletion only)
• DMTet natively support this

Idea 2: Geometry manipulation via Tetrahedra

• With the initial Mesh, initialize a DMTet network
• Simulate marching tetrahedra to convert the current mesh into tetrahedra
• Initially, all meshes will be a surface of a tetrahedra
• Render with DMTet modeling
• Predict meshes from SDF, so that each coordinate has a gradient w.r.t. SDF network
• Can also combine Idea 1 for more differentiable operations
• DMTet’s modeling can natively support topological changes
• Only need to adjust the SDF and maintain the intersecting tetrahedra set
• Supports: new holes, fill up holes, object creation, object deletion, etc., through training
• Also enables multi-scale shape modeling

​

Idea for Application: Progressive Scene editing

• The method in Idea 2 makes it possible for some gradually shape editing
• If directly apply aggressive shape editing, may not receive correct gradient signals
• However, if the shape editing is slight, it is likely to get success through some minor adjustments
• Apply the progressive editing in “ProgressEditor”, we can achieve high-efficiency scene editing progressively, even animating

Works that demonstrates a similar idea

• 2DGS: 2D Gaussian splatting (instead of 3D) for shape reconstruction
• Also uses Gaussian + 2D surface representation
• “Mesh-Guided Neural Implicit Field Editing”
• Uses tetrahedra representation for human-guided scene editing
• Applies OCTree data structure for speeding up

Industrial Practitioner

​

Mei Han

Smart XR Glasses: iGlasses

​

Glasses

• More Wearable, More Efficient, More Realistic
• Smarter Assistant
• AI+Companion
• Vivid Memory Engrams

More Wearable, More Efficient, More Realistic

​

• Two-layer diffraction optical waveguide
• Only rendering each eye can see
• Less differential rendering task
• Immersive depth perception

Left Right

iGlasses your personal life assistant

​

• Be your Nutritionist🥗Coach, Cookbooks
• Be your Eyes👀 (helping blind or low vision people)
• Be your Translator (even pet translator)

Interactive 3D Avatar

​

• AI Companion: AI companion with clear ethical boundaries.
• Lifelike Real-Time Interactions: VLMs + Nvdiffrast, context-based facial expressions
• Emotion Support: Provides 24/7 emotional support for psychological therapy

Virtual Friend

EVE

(Nature Select)

Otome Game

Love and Deepspace

(Papergames)

Real-time Dialogue

DAN Mode

(ChatGPT)

Electronic Pet

RoVR

（Ridgeline Labs）

Interactive 3D Avatar

​

• Lifelike audio-driven talking faces generation
• More nuanced facial expression rendering

VASA-1

Xu, S., Chen, G., Guo, Y. X., Yang, J., Li, C., Zang, Z., ... & Guo, B. (2024). Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404.10667

Guo, J., Zhang, D., Liu, X., Zhong, Z., Zhang, Y., Wan, P., & Zhang, D. (2024). Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.03168..

LivePortrait

Vivid Memory Engrams

​

Baby’s first step

Key points of knowledge

• Cache momentary spatial snapshots and preserve the memories you need
• Digital contact book generation for new acquaintance

Unforgettable moments

Differentiable Rendering 4Science

​

• Biological Physics
• Material Science

[1] Ichbiah, S., Delbary, F., & Turlier, H. (2023). Differentiable rendering for 3d fluorescence microscopy. arXiv preprint arXiv:2303.10440.

[2] Sego, T. J., Sluka, J. P., Sauro, H. M., & Glazier, J. A. (2023). Tissue forge: interactive biological and biophysics simulation environment. PLOS Computational Biology, 19(10), e1010768.

[3] Shi, L., Li, B., Hašan, M., Sunkavalli, K., Boubekeur, T., Mech, R., & Matusik, W. (2020). Match: Differentiable material graphs for procedural material capture. ACM Transactions on Graphics (TOG), 39(6), 1-15.

MATch

convert photographs of material samples into procedural material model

Tissue Forge

Interactive biological and biophysics simulation environment

3D fluorescence microscopy

Hacker 1: Bundle Adjustment using Pytorch3d

Yufeng Liu

Bundle Adjustment by Differentiable Rendering

In this project I implemented bundle adjustment in Pytorch3D

​

Objective:

• Given multi-view images, solve camera poses, object mesh, and object texture
• Fit a deform offset to a sphere

​

​

Experiments:

• Reconstruct Simple mesh
• Reconstruct Complex mesh
• Reconstruct Real world object

Simple Mesh

Set up:

• Given a known mesh and a set of cameras
• All cameras point towards the origin for simplicity
• Render a set of rgb and silhouette images as ground truth
• Perturb camera positions
• Start with a sphere mesh with uniform texture

​

​

​

Implementation Overview:

​

Deformation is applied to input mesh

The predicted cameras render a set of images

Calculate average loss across all rendered images

​

Loss consists of the following parts:

• RGB: L2 loss
• Silhouette: L2 loss
• Mesh edge: prevents mesh size shrinking or expanding too much
• Mesh normals: normals of neighboring faces should be close
• Laplacian: mesh smoothness

​

I used Adam optimizer to do back propagation

Update deformation and cameras

​

Views of predicted cameras

Views of groundtruth

cameras

Complex Mesh

Views of predicted cameras

Views of groundtruth

cameras

Real Images?

Need to do more

​

Hacker 2: Minimal Implementation of Soft Rasterizer

Christopher Conway

• Soft rasterizer is “truly differentiable” as it:
• directly renders color mesh with differentiable functions
• back-propogrates supervision to mesh vertices/attributes
• The paper demonstrates rendering of a triangle primitive

Liu S. et al, “Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning”, CVPR 2019

Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning

• For this project I did a minimal implementation of a soft backward function for rendering a rectangle on an image
• I implemented a soft geometry rectangle to probabilistically map the forward rendering function
• The backward gradient is used to go from pixel to primitive
• Results are shown for estimation of a mask image of a rectangle
• The key equation from the paper (for triangles) is:

​

​

Liu S. et al, “Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning”, CVPR 2019

Minimal Implementation for Rectangular Primitive

• The primitive function renders a rectangle with center, width, and height and generates the probability map based on distance to the edges (previous D function)
• Cross entropy is used to compute loss between render and mask
• Adam optimizer is implemented in pytorch for quick tensor broadcasting

​

​

Initial Primitive

Target Mask

Rendered Rectangle

Implementation Overview:

• The animation below shows the progression over 1000 iterations, runtime was just 1-2 s.
• I experimented with sigma and adam optimizer LR values to output successfully
• Increasing sigma softens the render and impacts center (2.0 best) vs height/width (5.0 best)
• Further optimization could improve performance and decrease runtime

​

​

Rendering Animation: