Photo Inspired Model Driven 3D Object Modeling

Kai Xu, Hanlin Zheng, Hao (Richard) Zhang, Daniel Cohen-Or, Ligang Liu, Yueshan Xiong

National University of Defense Technology, Simon Fraser University, Zhejian University, Tel-Aviv University

SIGGRAPH 2011

Presented by: Maham Tanveer

CMPT 985- Spring 2022

3D creation with inspiration

• A 3D model inspired from the 2D input
• Can be “as close as possible” or “a stylistic variation”
• Useable – editing and modification

2

Use what you have, create what you need

• Generating variations
• Using a pre-analyzed candidate model set

​

3

Related Concepts

Use of Controllers

4

Component-wise controllers

[Zheng et al. 2011]

Modeling by example

[Funkhouser et al. 2004]

[Xu et al. 2010]

Composition

Part Morphing

Overview

5

1. Labeled segmentation of input image
2. Choose candidate 3D model, either a close match or random
3. Deform the candidate to fit the silhouettes while preserving structure

​

​

Pre-Analysis of candidate set

• Part Correspondence and normalize the part scales [Xu et al. 2010]

​

6

• Refined fitting with set of controller with geometric interrelations stored [Zheng et al. 2011]
• Symmetry
• Proximity

​

7

Step 1 Model-driven image-space object analysis

​

Step 1: Model driven image analysis

8

2D Shape descriptors:

Fourier (contour)

Zernike (region)

Adjust rotation

And bounding box’s

Graph cut segmentation

• Pixel either a “part” or background
• Delong et all [2010] : data term, label term and smoothness term.

Step 1: Model driven image analysis

9

10

Step 2 Candidate Model Retrieval

​

Light Field Shape descriptor

• Shape descriptor: A concise way to represent a shape
• The Light Field Descriptor (LFD) of a 3D shape is a set of 2D images (silhouettes) of it, taken from a 2D array of cameras

11

https://www.cse.iitb.ac.in/~cs749/spr2016/lecs/17_features.pdf

Step 2: Model retrieval: Whole Shape

12

Query

Step 2: Model retrieval: Part Level

13

Query

OR JUST GO RANDOM !

14

Step 3 Silhouette Driven Deformation

​

15

2D image with segmentation

3D Candidate with controllers and pre-analysis

• Extract Silhouettes
• 2D Image
• 2D projection of controllers
• Point Correspondence between silhouettes using graph matching
• Over segmentation
• Segment correspondence

​

16

17

2D TO 3D

ILL POSED PROBLEM

18

2D-3D Reflectional Symmetry

• Plane : best fit for the current segment in 3D
• Project the target segment
• Cuboid
• perform refitting using mesh less deformation method [Muller et all]
• Rigid + Non uniform scaling transformation
• General Cylinder
• Create a new GC interpolate target
• Re compute medial axis and radius

​

​

​

19

External Controller Reconstruction

20

• Iterative Optimization
• Symmetric Controllers
• Closest to viewpoint
• Apply to all
• Proximity Constraints
• Update feature curves w.r.t other deformed controllers
• Restore the shape via refitting
• Influence: local frame encoding [Gal et al.]

​

21

Symmetrize Individual

Structure Symmetry

Proximity Constraints

22

Symmetrize Individual

Structure Symmetry

Proximity Constraints

23

Skeleton Driven Deformation [Kavan et all]

Transfer same transformation

24

Results

25

Using Different Candidates

26

Using Different Candidates

27

Using Different Candidates

28

Using Different Candidates

29

Google Chair Challenge

30

Asymmetric Target: conflict in constraints

31

Conclusion and Limitations

• Photos are a rich resource to utilize for inspiration
• Silhouette based deformation
• Retains structural integrity
• Re-usability
• Limitations
• Morphing an existing structure, not generating new models
• Silhouette matching is a limited constraint
• Limited by available 3D structures
• Only reflectional symmetry employed, not rotational
• Constraints can be conflicting

​

32

Future Work

• Fine Detailed Features
• Better structure deformation techniques
• Use hand drawn sketches as input
• Create new 3D structures
• Consider if mesh representation is a good choice for these tasks

​

33

Questions / Comments

34

References

• Xu, k., Li, h., Zhang, h., Cohen-or, d., Xiong, y., And cheng, z. 2010. Style-content separation by anisotropic part scales. ACM trans. On graphics (proc. Siggraph asia)
• Zheng, y., Fu, h., Cohen-or, d., Au, o. K.-C., And tai, c.-L. 2011. Component-wise controllers for structure preserving shape manipulation. Computer graphics forum (proc. Eurographics) 30, 2, 563–572
• Delong, a., Osokin, a., Isack, h., And boykov, y. 2010. Fast approximate energy minimization with label costs. In proc. IEEE conf. On CVPR, 2173–2180.
• Muller ¨ , m., Heidelberger, b., Teschner, m., And gross, m. 2005. Meshless deformations based on shape matching. ACM trans. On graphics (proc. Siggraph) 24, 3,�471–478
• KAVAN, L., Collins, s., Zˇ ara ´ , j., And o’sullivan, c. 2007.�Skinning with dual quaternions. In proc. Of symposium on interactive 3D graphics and games, 39–46 �����

35