1 of 13

Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation

Ruihan Gao

Kangle Deng

Gengshan Yang

Wenzhen Yuan

Jun-Yan Zhu

Carnegie Mellon University

University of Illinois Urbana-Champaign

2 of 13

Prior works

``A statue of angel, 3d asset’’

DreamFusion, Magic3D, Point-E, MVDream, DreamCraft3D, Dreambooth3D, ProlificDreamer, RichDreamer, etc

``A corgi taking a selfie’’

Text-to-3D

Image-to-3D

Zero-1-to-3, Wonder3D, Magic123, DreamGaussian, LRM, Make-It-3d, One-2-3-45, TripoSR, SV3D, etc

3 of 13

Touch provides fine geometric details

4 of 13

Tactile Input

Text to 3D

Text Input

Ours

``An avocado with avocado texture’’

We incorporate touch for 3D generation

``An avocado’’

Text Input

5 of 13

Solution: visual-tactile texture grid

Trainable 3D Texture Field

=(albedo, tactile)

Render

Albedo UV map

Mesh

Base

Mesh

Generation

Input Image

Tactile Normal UV map

Input Tactile

Image Quilting

Rendered Albedo

Add Noise

Refined Color

ControlNet

“a Miffy bunny with woven texture”

Render

Denoise

Diffusion

UNet

Reference Albedo

Add Noise

Denoise

Reference Normal

Texture DreamBooth

“V* normal map”

Rendered Normal

Rendered Color

Rendered Normal

Refined Normal

Render

Input Text

“a Miffy bunny”

OR

Tactile Matching

Tactile Guidance

Visual Guidance

Visual Matching

Render

How can we equip 3D assets with this additional touch modality?

Given a text or image input, / we generate a base mesh with albedo map / using off-the-shelf methods like Wonder3D /.

To introduce additional normal texture, we synthesize / a 2D tactile normal map from an input patch.

However, the synthesized normal map may not align with the albedo map. Thus, to ensure cross-modal alginment, we learn a 3D texture field / that maps each 3D point to a paired albedo and tactile normal texture and jointly optimize them.

To train the texture field, we can query the mesh from any random viewpoint and minimize the rendering loss.

However, vision and touch have very different range of receptive fields. To match the scale discrepancy, we sample cameras P_V / from a distance to render the visual image / and sample cameras P_T / up-close to the surface to render tactile patches /.

We use the albedo map / from the base mesh / to reconstruct the albedo rendering and use the synthesized tactile normal map / to reconstruct the tactile rendering.

To tackle the alignment challenge, we jointly refine them with multi-step denoising process.

Specifically, we use a normal conditioned-ControlNet / to refine full-color rendering and use a customized Texture DreamBooth model / to refine the tactile patch rendering.

Compared to prior works the at mainly focus on vision, / (highlight the top part)

we synthesize a normal map from tactile data and refine it on patch level. / (highlight the bottom part)

And use the high-res normal rendering / to condition visual generation to improve the cross-modal alignment. / (highlight normal rendering)

6 of 13

Ablating guidance loss

Tactile Input

strawberry

Ours

Ours w/o tactile guidance

Ours w/o visual guidance

Image Input

Rendered Full Color

Generated Normal

Generated Albedo

7 of 13

Tactile data acquisition

2D Texture Synthesis

[Efros, 2001]

GelSight Mini

Sensor Output

High-pass Filtering

Depth Estimation

Height Map

Normal Map

8 of 13

Normal Map

Object

Normal Map

Object

Normal Map

Object

woven

avocado

chopping board

orange

strawberry

table tennis handle

cantaloupe

canvas bag

table cloth

potato

striped steel

metal grid

football

corn

cork

glove

rubber

cleaning cloth

18 materials, diverse textures

9 of 13

Diverse textures with the same object

canvas bag

heat resistant rubber with heart shape

cantaloupe

stripe sculpture steel

strawberry

``A coffee cup with … texture’’

10 of 13

3D generation with single texture

``An American football’’

``A corn’’

Input

Rendered Full Color

Generated Normal

11 of 13

Multi-part Texture Generation

``a lamp with white white lampshade and wooden base, lampshade with canvas bag’s texture, base with metal grid texture’’

Input

Output

Generated Normal

Generated Albedo

Rendered Full Color

Predicted Labels

12 of 13

Thank you!

``a cactus in a pot, cactus with orange texture, pot with heat resistant rubber with heart-shape pattern texture’’

a cactus in a pot

``a goat sculpture with gold goat and black base, goat with fine striped steel texture, base with smooth wood texture’’

https://ruihangao.github.io/TactileDreamFusion/

13 of 13

3D printing to showcase the textures

1. Normal -> displacement -> deform the mesh -> 1 day turnover for printing (adjust the scale)