1 of 24

Joint UV Optimization & Texture Baking

Julian Knodt, Zherong Pan, Kui Wu, Xifeng Gao

Low-Poly

3K Tris

High-Poly

300K tris

Lightspeed Studios

[Pierre Antoine, Sketchfab]

2 of 24

Outline:

[LuddePudde, Sketchfab]

Motivation & Problem Outline
Prior Work
Geometric Discrepancy

Parallax Mapping

Texture Resolution Mismatch

Texture Baking & UV Optimization

Conclusion

3 of 24

Motivation:

[Model from maxpanysh, Sketchfab]

Artists creates concept-art high-poly mesh

But above rendering budget,

especially for mobile games

4 of 24

Motivation

[Model from maxpanysh, Sketchfab]

[Simplygon Remeshing]

Texture baking:

transfer texture from

high-poly to low-poly

?

34K Tris

8K Tris

5 of 24

Industry Texture Baking

[Virtual Museums of Małopolska, Sketchfab]

High Poly

Low Poly

❌: Requires Per-Mesh Offset Tuning to avoid artifacts

❌: Projection errors due to differences between low-poly and high-poly meshes

✅: Fast

✅: Arbitrary Textures

6 of 24

Industry Texture Baking

[Virtual Museums of Małopolska, Sketchfab]

High Poly

Low Poly

❌: Requires Per-Mesh Offset Tuning to avoid artifacts

❌: Projection errors due to differences between low-poly and high-poly meshes

✅: Fast

✅: Arbitrary Textures

7 of 24

Differentiable Rendering

[CG Channel’s on Mitsuba 3]

Recover parameters from images:

Mesh Textures/Materials
Vertex Coordinates
Lighting

8 of 24

Off-the-Shelf Differentiable Rendering

Blurry, Unusable

[NVDiffmodeling, 2021]

9 of 24

Why doesn’t Differentiable Rendering work?

[NVDiffmodeling, 2021]

10 of 24

Why doesn’t Differentiable Rendering work?

Non-Bijective correspondence between high-poly texels and low-poly texels due to difference in geometry

11 of 24

Differentiable Parallax Mapping

Function that maps viewing direction to offset in UV space

Bijective between high-poly and low-poly texels
Separate Geometry & Texel Correlation

To get around the problem that many low-poly texels can correspond to many high-poly texels, we add differentiable parallax mapping,

disentangling geometric differences and texel view correspondence.

This is a function that changes the UV coordinate used for texturing on the low-poly mesh based on viewing direction, so even with geometric differences

between the high-poly and low-poly we can have a bijective correspondence between texels.

In the figure, two cameras are viewing the same point on the low-poly grey approximation, but at different viewing angles.

We change the UV coordinate of the low-poly approximation based on view angle, allowing each view to correspond to different points on the high-poly leaf.

UV shifting based on viewing directions is known as parallax mapping and there are a number of prior works which develop tools for forward-rendering parallax.

12 of 24

View-Parallax Prior Work

Detailed Shape Representation with Parallax Mapping, Kaneko et al, 2001
Steep Parallax Mapping, McGuire & McGuire 2005

[OpenGL tutorial & Steep Parallax Mapping (McGuire, 2005)]

Starting from Kaneko et al in 2001, which uses a simple planar approximation for parallax mapping,

to steep parallax mapping, from McGuire in 2005, which performs raymarching in a heightfield to find a UV offset, there are a plethora of approaches to parallax mapping.

These approaches are used in games today, often to add detail to planar surfaces such as floors and walls with little cost.

For example, Final Fantasy 14 uses parallax mapping for detailed rock terrain, and there are many mods for adding parallax to games such as Skyrim.

On the right, we have a brick wall with parallax mapping, which gives a much stronger perception of depth than the normal mapping on the left.

Below, we have a comparison of the original parallax mapping with steep parallax mapping on a complex surface, giving an even stronger perception of depth.

13 of 24

Issues with Prior Work:

Prior parallax use ground-truth depth:

Kaneko et al. assume simple geometry approximation
Steep Parallax Mapping is expensive, requires steps of raymarching and may have errors

May be difficult to optimize depth map for distinct meshes

Despite the numerous prior work on parallax mapping, they have all focused on a forward-rendering, where ground truth depth is known for flat surfaces such as walls and floors.

For our case of differentiable rendering, we find that these approaches have some shortcomings.

For Kaneko et al’s parallax mapping, there is an assumption that geometry is planar, and thus cannot accurately recover details on complex models,

and for McGuire’s Steep Parallax Mapping, we find it to be expensive, requiring many texture fetches.

Steep parallax mapping is also not completely differentiable as it performs discrete raymarching steps.

Furthermore, depth maps on arbitrary surfaces can be poorly defined and difficult to recover.

As an example of the shortcomings, on the right is a cube textured with an deep depth map with visual artifacts due to discretization of raymarching in steep parallax mapping.

This can be seen by the streaks between each raymarching step which were not accurately represented.

14 of 24

Spherical Harmonic View Parallax:

Original UV + Sph. Harmonics = New UV

Shift UV coordinate based on view direction.
Leads to 1-2 higher PSNR, and much sharper visually.
Compared to optimizing other parallax, ours performs consistently better.

To work nicely with Differentiable Rendering, we create a differentiable and expressive spherical harmonic view parallax mapping.

We shift UV coordinates based on the output of the Spherical Harmonic function applied to the camera’s view direction.

This allows for recovering high-frequency details without any prior on surface structure.

�As an extreme example, on the right we visualize our friend the companion cube, with 7.8K vertices, and perform texture baking to an impostor with 8 vertices.

Even with the large visual difference, our approach is able to recover convincing texture details as compared to off-the-shelf differentiable rendering.

We find that view parallax often improves PSNR by around 1-3 PSNR, but visually it is much more appealing than standard texture baking.

In addition, we ablate our approach compared to other parallax mappings and find that ours performs consistently better.

15 of 24

Is this enough for practical use?

Parallax mapping allows for capturing sharp features from original

But can output texture accurately capture high-frequency details?

16 of 24

Low-Poly Texture Quality

High-Poly: 2048 x 2048

Low-Poly: 512 x 512

We find that pixelation occurs when the target texture size is significantly smaller than the input texture’s size,

again leading to many low-poly texels corresponding to high-poly texels.

On the left, we have the original 2048 by 2048 texture, and on the right we have the target 512 by 512 UV, accurately scaled with respect to the texture on the left.

Clearly, we won’t be able to squish all the details from the mesh on the left in such a small space on the right, but we want to give more pixels to areas with more details.

This may be possible, as due to color redundancy, we can perform texture-aware UV optimization to capture more features of the high-poly mesh.

This was also shown by the previous presentation, which uses a much more aggressive approach to redundancy.

Unlike the prior work though, our approach does not seek to overlap triangles for optimization, but simply expand or reduce their size.

17 of 24

Texture-Aware UV Optimization

Signal Specialized Parameterization, Sander et al. 2002

Used in https://github.com/Microsoft/UVAtlas/

Measures reconstruction error per triangle

Assumption that each triangle is piecewise constant, requiring subdivision for high quality optimization
Essentially relies on texture gradient as a measure of importance

There is relatively little work in texture aware optimization, due to the difficulty in capturing texture information.

The main prior work we find is signal specialized parametrization from Sander et al, which is used in the open-sourced UVAtlas.

�Signal-Specialized parameterization approximates texture per triangle as piecewise constant, and requires subdivision for better optimization.

During UV optimization, it relies on texture gradient as a signal of importance per triangle.

On the right, we can see that signal-specialized UV parameterization can dramatically reduce blurring of the final model, as compared to geometric only parameterization on the left.

This gives pretty convincing evidence that optimizing UV to minimize texture difference is worthwhile.

We take an entirely different approach to Signal Specialized Parameterization, as we believe that frequency alone may not be a good measure of importance.

18 of 24

Joint UV Optimization Visualization

[Cleveland Museum of Art, Sketchfab]

Original

Texture Resolution

2048x2048

Low-Poly

Texture Resolution

512x512

Init. UV

& Texture

Final UV

On the left, is the original mesh with a texture resolution of 2048 by 2048, and on the right we have the low-poly mesh with a target texture resolution of 512 by 512.

Arrows in the image represent the optimization direction of UV vertices and their magnitude.

Initially, we bake with a fixed UV for a small number of iterations to get a baseline texture before optimizing the UV.

<CLICK>

After initial baking, we begin to optimize the UV.

As can be seen in the middle, because we are reducing the number of pixels by a factor of 16, the UV expands rapidly. Between each UV optimization step, there are 10 differentiable rendering optimization steps, introducing a motion blurring effect.

<CLICK>

After a fixed number of iterations, we fix the UV, and continue to optimize the texture to remove any effects of optimization.

19 of 24

Joint UV Optimization Visualization

[Cleveland Museum of Art, Sketchfab]

Original

Texture Resolution

2048x2048

Low-Poly

Texture Resolution

512x512

Init. UV

& Texture

Final UV

On the left, is the original mesh with a texture resolution of 2048 by 2048, and on the right we have the low-poly mesh with a target texture resolution of 512 by 512.

Arrows in the image represent the optimization direction of UV vertices and their magnitude.

Initially, we bake with a fixed UV for a small number of iterations to get a baseline texture before optimizing the UV.

<CLICK>

After initial baking, we begin to optimize the UV.

As can be seen in the middle, because we are reducing the number of pixels by a factor of 16, the UV expands rapidly. Between each UV optimization step, there are 10 differentiable rendering optimization steps, introducing a motion blurring effect.

<CLICK>

After a fixed number of iterations, we fix the UV, and continue to optimize the texture to remove any effects of optimization.

20 of 24

Joint UV Optimization Visualization

[Cleveland Museum of Art, Sketchfab]

Original

Texture Resolution

2048x2048

Low-Poly

Texture Resolution

512x512

Init. UV

& Texture

Final UV

On the left, is the original mesh with a texture resolution of 2048 by 2048, and on the right we have the low-poly mesh with a target texture resolution of 512 by 512.

Arrows in the image represent the optimization direction of UV vertices and their magnitude.

Initially, we bake with a fixed UV for a small number of iterations to get a baseline texture before optimizing the UV.

<CLICK>

After initial baking, we begin to optimize the UV.

As can be seen in the middle, because we are reducing the number of pixels by a factor of 16, the UV expands rapidly. Between each UV optimization step, there are 10 differentiable rendering optimization steps, introducing a motion blurring effect.

<CLICK>

After a fixed number of iterations, we fix the UV, and continue to optimize the texture to remove any effects of optimization.

21 of 24

Combine Image Loss & UV Optimization

Image Loss per Δ

Sym. Dirichlet Energy

Bijectivity Constraints

×

+

Perform UV optimization of Symmetric Dirichlet energy weighted by image loss per triangle
Perform line search to maintain global bijectivity
For 2D cases, on average 1-3 dB higher PSNR

Low δ:

High δ:

Our approach’s UV optimization optimizes the Symmetric Dirichlet energy weighted by the sum of each triangle’s texels rendering loss. To maintain global bijectivity, we perform line search at each UV optimization step with distance checks, preventing inversion or overlaps. This effectively makes the UV optimization procedure texture aware.

Intuitively, our approach can thought of as using image loss per triangle as a weight to the symmetric dirichlet energy, subject to bijectivity constraints

As an example, on the left in the green circle, there is a triangle in a region with near-constant color. It’s UV remains unchanged since its loss is low.

In the red circle, due to pixelation there is a higher loss, so those UV triangles are expanded.

This leads to regions which cannot sufficiently represent detail to be expanded, allowing our approach to even further improve the quality of the output.

We test this on multiple 2D images projected onto planar grids, and find that by optimizing the UV we find there is an average increase of 1-3 dB PSNR, and in some rare cases we find an improvement up to 10 dB.

22 of 24

Results

Our approach consistently leads to improvements for texture baking &

Consistent improvements in 2D texture reduction

See paper for complete results

Optimized UV

GT

NVDiff

Ours

2D Texture Reduction

3D Texture Reduction

We find our approach to be incredibly robust. Across all meshes, it shows improvement as compared to off-the-shelf NVDiffmodeling, and in all cases the results are visually much more appealing. This is due to a combination of both UV optimization and parallax mapping.

For our 2D tests, we find that the optimized UV closely correlates with highly-salient features on the image, such as in the Great Wave off Kanagawa shown in the top-right. The UV is expanded on edges of the wave, where there are high-frequency changes.

On 3D texturing examples, our results are much more clear than NVDiffmodeling, as can be seen on this easter egg. Our approach, even with a 4x texture size reduction maintains sharp features as compared to NVDiffmodeling without any artifacts.

See our paper for complete results, we perform ablations of each component of our approach, and some additional features not covered in this talk.

23 of 24

Drawbacks:

Our output

[Loarziel, Sketchfab]

Takes about 10 minutes with UV optimization
Spherical Harmonics may crack if extreme mesh differences
Difficult to preserve space between charts

24 of 24

Summary:

[Loarziel, Sketchfab]

Our output

Captures high-frequency details despite geometric differences
Texture-Aware optimized UV has much higher efficiency
Higher visual similarity than raycasting or standard differentiable rendering