Differentiable Meshing and Rendering

CMSC 498Z · DIFFERENTIABLE PROGRAMMING

DMesh (NeurIPS 2024) and its successor, DMesh++

Richard Chen · Tuesday, October 27, 2026

The plan for this lecture

TODAY

01

The problem

Why a mesh’s connectivity blocks gradients, and the old workarounds.

02

DMesh

Turning “does this face exist?” into a smooth probability.

03

DMesh++

A cheaper test that scales to hundreds of thousands of points.

04

Analysis

What the meshes still can’t do, then discussion.

DMesh & DMesh++ · Richard Chen · 2

Why meshes resist gradients

PART 1 · THE PROBLEM

A mesh is two things: where the vertices are, and which triangles connect them.

Positions are continuous. Nudge a vertex and the loss changes smoothly.

Connectivity is discrete. A face is in or out, so it has no gradient.

And it explodes. Possible connectivities grow exponentially with the number of points.

Same four points, different faces. No small nudge gets from the left mesh to the right one: it’s a jump.

DMesh & DMesh++ · Richard Chen · 3

Three earlier workarounds

PART 1 · THE PROBLEM

DMesh’s approach: make the connectivity itself differentiable.

DMesh & DMesh++ · Richard Chen · 4

A quick primer: Delaunay and power diagrams

PART 2 · BACKGROUND

Delaunay rule: the circle through any triangle’s corners (dashed) contains no other point.

Delaunay triangulation

Connect points so every triangle’s circumcircle (circumsphere in 3D) is empty. Result: well-shaped triangles, no crossings.

Weighted version

Give each point a weight w and measure with the power distance below. A heavy point claims more space; a light one can vanish.

π(p, q) = |p − q|² − w_p − w_q

Power diagram (the dual)

Each point owns the region closest to it in power distance. Two points are joined exactly when their regions touch.

DMesh & DMesh++ · Richard Chen · 5

Every face gets a probability

PART 2 · DMESH

Λ(F) = Λ_wdt(F) × Λ_real(F)

Each point = 5 numbers

Position (x, y, z), a weight w, and a “real” value ψ between 0 and 1.

Λ_wdt: a candidate?

Does F appear in the weighted Delaunay triangulation of the points?

Λ_real: on the surface?

Are F’s corners marked real? The final mesh keeps faces where both tests pass 0.5.

Only per-point numbers are optimized, so gradients reach positions, weights and ψ, and the topology changes as the points move.

DMesh & DMesh++ · Richard Chen · 6

The real part and the imaginary part

PART 2 · DMESH

2D version: teal edges are the shape (real); thin gray edges only hold the tessellation together (imaginary). Teal dots have ψ ≈ 1.

Fill the space. The weighted Delaunay triangulation splits the whole convex domain into tetrahedra.

Pick a subset. The mesh is just the faces marked real; the rest is scaffolding that gets thrown away.

Free guarantee. Triangulation faces never cross, so the mesh can’t self-intersect.

Any shape. Open, closed, even non-orientable surfaces like a Möbius strip.

DMesh & DMesh++ · Richard Chen · 7

Making both tests smooth

PART 2 · DMESH

Λ_wdt: how close is F to being in?

Using the power diagram (the triangulation’s dual), measure a signed distance τ: positive if F is in, negative if not.

Λ_wdt = mean σ(α_wdt · τ)

About O(N) instead of O(N²) for the earlier differentiable triangulation (Rakotosaona et al. 2021).

Λ_real: are the corners on the surface?

Λ_real = min(ψ₁, ψ₂, ψ₃)

A differentiable min. Lowering one point’s ψ switches off every face around it without moving anything.

The temperature α sets how sharp the soft decision is: the same continuous-relaxation trade-off from lecture.

DMesh & DMesh++ · Richard Chen · 8

Faster and more reliable than the earlier method

PART 2 · DMESH

Times are for triangles in 2D; missed edges are existing edges scored below 0.5, the case like surfaces in 3D. Previous = Rakotosaona et al. 2021, both in PyTorch (DMesh, Appendix Table 4).

Scales flat: ~110–125 ms from 1K to 30K points. The old method ran out of memory at 100K.

Honest catch: under ~1K points the old method is faster, since DMesh must build the power diagram first.

DMesh & DMesh++ · Richard Chen · 10

One representation, three kinds of input

PART 2 · DMESH

A mesh

Push Λ up on the true faces and down on every other face. A check that the representation can express real meshes.

A point cloud

Expected Chamfer distance: sample points from faces, each weighted by its face’s probability.

Multi-view images

Render each face with Λ as its opacity and compare to the photos with an L1 loss.

PLUS THREE REGULARIZERS

Weight: simplify the triangulation, fewer vertices

Real: neighbors agree on ψ, which fills holes

Quality: penalize thin, sliver triangles

DMesh & DMesh++ · Richard Chen · 11

From images: two phases, two renderers

PART 2 · DMESH

Phase 1 · move everything

Positions, weights and ψ all update

Renderer A sorts see-through faces in screen tiles, like Gaussian splatting, averaged 50/50 with nvdiffrast for edge gradients. Its depth order is approximate, which leaves false inner faces.

Phase 2 · pick faces

Freeze the triangulation, update ψ only

With positions and weights fixed, Λ_wdt is exactly 0 or 1. Renderer B walks each ray tetrahedron to tetrahedron: exact depth order, low memory.

Start: a 20×20×20 grid of points for images; for point clouds, samples plus Voronoi vertices, which sit near the medial axis.

Coarse to fine: 4 epochs, resampling 1K → 3K → 10K points from the current mesh.

Clean-up rule: a face survives only if a point on one side has ψ = 0, which deletes hidden inner faces.

DMesh & DMesh++ · Richard Chen · 12

What the regularizers actually do

PART 2 · DMESH

Weight regularization

A true trade-off: 60% fewer faces costs 45% more error, plus holes and bumps at the high end.

Quality regularization

No trade-off in error: nicer triangles and lower CD, but more faces and more holes.

Point-cloud reconstruction, each regularizer tested with the other switched off (DMesh, Appendix E.3, Tables 5 and 6).

DMesh & DMesh++ · Richard Chen · 13

Results: compact and accurate, but slow

PART 2 · DMESH

>99%

of faces recovered when converting the Stanford bunny, dragon and Buddha

<1%

false faces added in the same test

Chamfer distance ×10⁻⁵, averaged over 11 closed, open and mixed models (DMesh, Table 2).

DMesh & DMesh++ · Richard Chen · 14

Where DMesh falls short

PART 2 · DMESH

Slow

The triangulation runs on the CPU (CGAL), step by step. Over a second for 100K points, and it can’t be parallelized.

No manifold guarantee

Faces are switched on independently, so stray non-manifold edges appear.

Patchwork renderer

Two partially differentiable renderers. The first leaves false inner faces that a post-processing rule deletes.

Low resolution

A few thousand vertices per shape; fine detail on closed surfaces is lost.

DMesh++ goes after the first problem, and that also fixes the last one.

DMesh & DMesh++ · Richard Chen · 15

Replace the triangulation with a Minimum-Ball test

PART 3 · DMESH++

Dashed gray: rejected, a point sits inside its smallest ball. Solid teal: accepted, its ball is empty.

Drop the weight. Each point is now (x, y, z, ψ).

New rule. A face counts if the smallest ball through its corners holds no other point.

d = dist(center, nearest other point) − radius

Λ_min = σ(α_min · d)

Same guarantees. Every such face is a Delaunay face, so still no self-intersections and few thin triangles. Λ = Λ_min × Λ_real.

DMesh & DMesh++ · Richard Chen · 16

Why it’s so much faster

PART 3 · DMESH++

32×

faster face probabilities in 3D (16× in 2D)

75%

less GPU memory in 3D (96% less in 2D)

168 ms

for 200K points in 3D (117 ms in 2D)

Local instead of global

Each face only needs a nearest-neighbor search, which runs in parallel on the GPU. The triangulation was sequential. Roughly O(N) → O(log N) per face.

Cache the neighbors

Neighbors barely change between steps, so they’re cached and refreshed every 50 steps during reconstruction.

DMesh & DMesh++ · Richard Chen · 18

The reconstruction loop

PART 3 · DMESH++

Step 1

Initialize

A regular grid, or the input point cloud if there is one.

Step 2

Move points

Optimize positions with ψ fixed, using a Chamfer or image loss.

Step 3

Pick faces

Optimize ψ with positions fixed; drop faces no camera sees.

Step 4

Subdivide

Insert edge midpoints for detail, then repeat, coarse to fine.

A better renderer

Anti-aliased triangle edges give visibility gradients that DMesh’s renderer lacked, so shapes can actually slide into place.

Extra per-point features

Points can carry color, blended across each face, to reconstruct textured shapes.

DMesh & DMesh++ · Richard Chen · 19

Results: point clouds, 2D and 3D

PART 3 · DMESH++

3D, 500 random Thingi10K models, half closed and half open; 200K input points; CD ×10⁻⁴ (DMesh++, Table 3).

Others get worse on open surfaces; DMesh++’s error stays the same.

2.6×

faster than DMesh on 2D fonts (11.3 s vs 30.4 s), with lower error (1.82 vs 1.97 ×10⁻⁶)

~1M

edges in complex 2D drawings; DMesh ran out of GPU memory on them

DMesh & DMesh++ · Richard Chen · 20

Results: multi-view reconstruction

PART 3 · DMESH++

Average over 50 models from Thingi10K and Objaverse, 64 rendered views each (DMesh++, Table 4).

½

DMesh’s Chamfer error

4.7×

more faces, so finer detail

3.8×

faster than DMesh

DMesh & DMesh++ · Richard Chen · 21

Reinforce-Ball: removing points with REINFORCE

PART 3 · DMESH++

The trap

Point C sits between A and B and isn’t needed. Lower C’s probability to 0.8: faces AC and BC drop to 0.8, AB rises to 0.2. Alpha blending treats them as independent, so the ray sees 0.2 + 0.8 × 0.8 = 0.84 opacity instead of 1. The loss rises and the optimizer puts C back.

The fix

Keep each point with probability φ. Sample whole point sets (B = 1024), score each with reconstruction loss + ε × point count, and use the log-derivative trick:

∇E[L] ≈ (1/B) Σ ∇log P(Pᵢ | φ) · Lᵢ

2,793 → 152

edges on 2D fonts (ε = 10⁻⁵), CD 1.82 → 2.77

78% fewer

edges than DMesh’s weights at similar error

2D only

and slow: 200 s vs 11 s without it

DMesh & DMesh++ · Richard Chen · 22

What DMesh++ gave up

PART 3 · DMESH++

More non-manifold vertices

0.036 vs DMesh’s 0.002 in multi-view. The authors suspect a trade-off between expressiveness and topological errors.

No knob for mesh size

Without weights, nothing simplifies the mesh. Their fix, Reinforce-Ball (a REINFORCE-style trick), is experimental and 2D only.

Still costly on point clouds

282 s versus 4 s for Poisson reconstruction: more accurate, but far more compute.

Synthetic images only

Assumes known lighting and simple vertex colors. On real DTU photos the mesh fills with floaters.

DMesh & DMesh++ · Richard Chen · 23

DMesh vs DMesh++ side by side

RECAP

Same core idea, a probability per face; DMesh++ trades the global triangulation for a local test.

DMesh & DMesh++ · Richard Chen · 24

Where this line of work is heading

PART 4 · ANALYSIS

Real photos

MeshSplatting (CVPR 2026, with Sanghyun Son) optimizes opaque, connected triangle meshes from real images, built to drop straight into game engines.

Simulation-ready meshes

HoloTetSphere (2026) found surface methods, DMesh included, often fail or fragment when turned into volume meshes for simulation.

Open problems they name

Manifold and watertight guarantees, handling self-occlusion, and generative models that learn mesh connectivity.

Natural next step: add differentiable checks for “is it a valid, physical solid?” to the same loss.

DMesh & DMesh++ · Richard Chen · 26

Papers and code

REFERENCES

Son, Gadelha, Zhou, Xu, Lin, Zhou. DMesh: A Differentiable Mesh Representation. NeurIPS 2024. arxiv.org/abs/2404.13445

Son, Gadelha, Zhou, Fisher, Xu, Qiao, Lin, Zhou. DMesh++: An Efficient Differentiable Mesh for Complex Shapes. arxiv.org/abs/2412.16776

Rakotosaona, Aigerman, Mitra, Ovsjanikov, Guerrero. Differentiable Surface Triangulation. ACM TOG 2021.

Held et al. MeshSplatting: Differentiable Rendering with Opaque Meshes. CVPR 2026. arxiv.org/abs/2512.06818

HoloTetSphere: Unified TetSphere Mesh Reconstruction for Physical Simulations. 2026. arxiv.org/abs/2607.08398

Code: github.com/SonSang/dmesh · github.com/SonSang/dmesh2

DMesh & DMesh++ · Richard Chen · 28