Neural Shape Compiler:�A Unified Framework for Transforming between Text, Point Cloud, and Program

Low-Level

Point Cloud

Mesh

Voxel

Implicit Function

Multi-View

Mid-Level

Low-Level

Point Cloud

Mesh

Voxel

Implicit Function

Multi-View

draw('Top', 'Square', P=(-3,-2,0),

G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

Shape Program

CSG Tree

n-ary Hierarchy

Symmetry Hierarchy

Mid-Level

Low-Level

High-Level

Point Cloud

Mesh

Voxel

Implicit Function

Multi-View

draw('Top', 'Square', P=(-3,-2,0),

G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

Shape Program

CSG Tree

n-ary Hierarchy

Symmetry Hierarchy

Language: Semantics, Compositionality, Functionality

looks like a barbers chair, square bottom

the one with one single leg to the floor

the chair with armrests and a circular back, it's the one that's square on the bottom

resembles a hair salon chair with a solid square base

Mid-Level

Low-Level

Point Cloud

Mesh

Voxel

Implicit Function

Multi-View

draw('Top', 'Square', P=(-3,-2,0),

G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

Shape Program

CSG Tree

n-ary Hierarchy

Symmetry Hierarchy

High-Level

looks like a barbers chair, square bottom

the one with one single leg to the floor

the chair with armrests and a circular back, it's the one that's square on the bottom

resembles a hair salon chair with a solid square base

Language: Semantics, Compositionality, Functionality

Motivation

Reconstruct

Ground-truth

[Zhang et al., NeurIPS’18]

Motivation

Reconstruct

the office chair with wheels, the one with rollers

the target has an adjustment lever on the top with no holes. the armrests stick out

the one that has a pole connecting to the seat

wheels, computer chair

Ground-truth

[Zhang et al., NeurIPS’18]

Motivation

Reconstruct

the office chair with wheels, the one with rollers

the target has an adjustment lever on the top with no holes. the armrests stick out

the one that has a pole connecting to the seat

wheels, computer chair

draw('Top', 'Circle', P=(-1,0,0), G=(4,7))

​

draw('Sup', 'Cylinder', P=(-11,1,0), G=(13,1))

​

for(i<5, 'Rot', theta=72°, axis=(-12,1,0)

draw('Base', 'Line', P1=(-12,1,0), P2=(-12,-7,-5), theta*i, axis)

​

draw('Back_sup', 'Cub', P=(2,4,-1), G=(4,2,2))

​

draw('Back', 'Cub', P=(6,1,-5), G=(13,2,10), theta=0°)

​

for(i<2, 'Trans', u=(0,0,8))

draw('Chair_Beam', 'Cub', P=(2,-5,-5)+i*u, G=(5,1,2))

​

for(i<2, 'Trans', u=(0,0,8))

draw('Hori_Bar', 'Cub', P=(7,-5,-5)+i*u, G=(1,9,2))

Ground-truth

[Zhang et al., NeurIPS’18]

Motivation

Reconstruct

the office chair with wheels, the one with rollers

the target has an adjustment lever on the top with no holes. the armrests stick out

the one that has a pole connecting to the seat

wheels, computer chair

draw('Top', 'Circle', P=(-1,0,0), G=(4,7))

​

draw('Sup', 'Cylinder', P=(-11,1,0), G=(13,1))

​

for(i<5, 'Rot', theta=72°, axis=(-12,1,0)

draw('Base', 'Line', P1=(-12,1,0), P2=(-12,-7,-5), theta*i, axis)

​

draw('Back_sup', 'Cub', P=(2,4,-1), G=(4,2,2))

​

draw('Back', 'Cub', P=(6,1,-5), G=(13,2,10), theta=0°)

​

for(i<2, 'Trans', u=(0,0,8))

draw('Chair_Beam', 'Cub', P=(2,-5,-5)+i*u, G=(5,1,2))

​

for(i<2, 'Trans', u=(0,0,8))

draw('Hori_Bar', 'Cub', P=(7,-5,-5)+i*u, G=(1,9,2))

Ground-truth

Shape Compiler

Shape Compiler

[Zhang et al., NeurIPS’18]

Motivation

Reconstruct

the office chair with wheels, the one with rollers

the target has an adjustment lever on the top with no holes. the armrests stick out

the one that has a pole connecting to the seat

wheels, computer chair

draw('Top', 'Circle', P=(-1,0,0), G=(4,7))

​

draw('Sup', 'Cylinder', P=(-11,1,0), G=(13,1))

​

for(i<5, 'Rot', theta=72°, axis=(-12,1,0)

draw('Base', 'Line', P1=(-12,1,0), P2=(-12,-7,-5), theta*i, axis)

​

draw('Back_sup', 'Cub', P=(2,4,-1), G=(4,2,2))

​

draw('Back', 'Cub', P=(6,1,-5), G=(13,2,10), theta=0°)

​

for(i<2, 'Trans', u=(0,0,8))

draw('Chair_Beam', 'Cub', P=(2,-5,-5)+i*u, G=(5,1,2))

​

for(i<2, 'Trans', u=(0,0,8))

draw('Hori_Bar', 'Cub', P=(7,-5,-5)+i*u, G=(1,9,2))

Completion

Ground-truth

Shape Compiler

Shape Compiler

Ground-truth

[Zhang et al., NeurIPS’18]

Motivation

Reconstruct

the office chair with wheels, the one with rollers

the target has an adjustment lever on the top with no holes. the armrests stick out

the one that has a pole connecting to the seat

wheels, computer chair

draw('Top', 'Circle', P=(-1,0,0), G=(4,7))

​

draw('Sup', 'Cylinder', P=(-11,1,0), G=(13,1))

​

for(i<5, 'Rot', theta=72°, axis=(-12,1,0)

draw('Base', 'Line', P1=(-12,1,0), P2=(-12,-7,-5), theta*i, axis)

​

draw('Back_sup', 'Cub', P=(2,4,-1), G=(4,2,2))

​

draw('Back', 'Cub', P=(6,1,-5), G=(13,2,10), theta=0°)

​

for(i<2, 'Trans', u=(0,0,8))

draw('Chair_Beam', 'Cub', P=(2,-5,-5)+i*u, G=(5,1,2))

​

for(i<2, 'Trans', u=(0,0,8))

draw('Hori_Bar', 'Cub', P=(7,-5,-5)+i*u, G=(1,9,2))

tall chair with thin legs

it has no slats on the back

side stool with side curved brass edges

a tall, skinny, long high legs

modern kitchen counter bar stool

with no space between back and the legs

Completion

square back with hole in seat

is tall, bar stool with a square top?

the legs have pointy legs

it has a circle back and seat

Ground-truth

Shape Compiler

Shape Compiler

Ground-truth

[Zhang et al., NeurIPS’18]

Motivation

Reconstruct

the office chair with wheels, the one with rollers

the target has an adjustment lever on the top with no holes. the armrests stick out

the one that has a pole connecting to the seat

wheels, computer chair

draw('Top', 'Circle', P=(-1,0,0), G=(4,7))

​

draw('Sup', 'Cylinder', P=(-11,1,0), G=(13,1))

​

for(i<5, 'Rot', theta=72°, axis=(-12,1,0)

draw('Base', 'Line', P1=(-12,1,0), P2=(-12,-7,-5), theta*i, axis)

​

draw('Back_sup', 'Cub', P=(2,4,-1), G=(4,2,2))

​

draw('Back', 'Cub', P=(6,1,-5), G=(13,2,10), theta=0°)

​

for(i<2, 'Trans', u=(0,0,8))

draw('Chair_Beam', 'Cub', P=(2,-5,-5)+i*u, G=(5,1,2))

​

for(i<2, 'Trans', u=(0,0,8))

draw('Hori_Bar', 'Cub', P=(7,-5,-5)+i*u, G=(1,9,2))

tall chair with thin legs

it has no slats on the back

side stool with side curved brass edges

a tall, skinny, long high legs

modern kitchen counter bar stool

with no space between back and the legs

Completion

square back with hole in seat

is tall, bar stool with a square top?

the legs have pointy legs

it has a circle back and seat

Ground-truth

Shape Compiler

Shape Compiler

Shape Compiler

Shape Compiler

Shape Compiler

Ground-truth

[Zhang et al., NeurIPS’18]

Motivation

One model training over various tasks across various representations

Motivation

One model training over various tasks across various representations

Motivation

Emergent Abilities of Large Language Models, Wei et al. TMLR22

One model training over various tasks across various representations

Approach

Codes

ShapeCode Transformer

Codes

Text Encoder

PointCloud Decoder

[ ]

chair with armrests

​

​

Approach

Codes

ShapeCode Transformer

Codes

Probabilistic

Text Encoder

PointCloud Decoder

[ ]

chair with armrests

​

​

Approach

Codes

ShapeCode Transformer

Codes

Probabilistic

PointCloud Encoder

Text Encoder

PointCloud Decoder

[ ]

chair with armrests

​

​

Approach

Codes

ShapeCode Transformer

Codes

Probabilistic

PointCloud Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

draw('Top', 'Square', P=(-1,0,0), G=(2,6))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Leg', 'Cub', P=(-12,-6,-6)+i*u, G=(13,1,1))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Hori_Bar', 'Cub', P=(-12,-6,-6)+i*u, G=(2,11,1))

​

draw('Hori_Bar', 'Cub', P=(-12,5,-6), G=(2,1,12))

​

draw('Back', 'Cub', P=(1,6,-6), G=(11,1,12), theta=0°)

[ ]

chair with armrests

​

​

Approach

Codes

ShapeCode Transformer

Codes

Probabilistic

PointCloud Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

draw('Top', 'Square', P=(-1,0,0), G=(2,6))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Leg', 'Cub', P=(-12,-6,-6)+i*u, G=(13,1,1))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Hori_Bar', 'Cub', P=(-12,-6,-6)+i*u, G=(2,11,1))

​

draw('Hori_Bar', 'Cub', P=(-12,5,-6), G=(2,1,12))

​

draw('Back', 'Cub', P=(1,6,-6), G=(11,1,12), theta=0°)

[ ]

chair with armrests

​

​

Deterministic

Approach

Codes

ShapeCode Transformer

Codes

draw('Top', 'Square', P=(-3,-2,0),

G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

Probabilistic

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

draw('Top', 'Square', P=(-1,0,0), G=(2,6))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Leg', 'Cub', P=(-12,-6,-6)+i*u, G=(13,1,1))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Hori_Bar', 'Cub', P=(-12,-6,-6)+i*u, G=(2,11,1))

​

draw('Hori_Bar', 'Cub', P=(-12,5,-6), G=(2,1,12))

​

draw('Back', 'Cub', P=(1,6,-6), G=(11,1,12), theta=0°)

[ ]

chair with armrests

​

​

Deterministic

Approach

Codes

ShapeCode Transformer

Codes

draw('Top', 'Square', P=(-3,-2,0),

G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

[ ]

two wide legs

reclining chair back

a chair with no armrest

Probabilistic

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

draw('Top', 'Square', P=(-1,0,0), G=(2,6))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Leg', 'Cub', P=(-12,-6,-6)+i*u, G=(13,1,1))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Hori_Bar', 'Cub', P=(-12,-6,-6)+i*u, G=(2,11,1))

​

draw('Hori_Bar', 'Cub', P=(-12,5,-6), G=(2,1,12))

​

draw('Back', 'Cub', P=(1,6,-6), G=(11,1,12), theta=0°)

[ ]

chair with armrests

​

​

Deterministic

Approach

Codes

ShapeCode Transformer

Codes

draw('Top', 'Square', P=(-3,-2,0),

G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

[ ]

two wide legs

reclining chair back

a chair with no armrest

Probabilistic

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

draw('Top', 'Square', P=(-1,0,0), G=(2,6))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Leg', 'Cub', P=(-12,-6,-6)+i*u, G=(13,1,1))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Hori_Bar', 'Cub', P=(-12,-6,-6)+i*u, G=(2,11,1))

​

draw('Hori_Bar', 'Cub', P=(-12,5,-6), G=(2,1,12))

​

draw('Back', 'Cub', P=(1,6,-6), G=(11,1,12), theta=0°)

Deterministic

[ ]

chair with armrests

​

​

Deterministic

Approach

Codes

ShapeCode Transformer

Codes

draw('Top', 'Square', P=(-3,-2,0),

G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

[ ]

two wide legs

reclining chair back

a chair with no armrest

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

draw('Top', 'Square', P=(-1,0,0), G=(2,6))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Leg', 'Cub', P=(-12,-6,-6)+i*u, G=(13,1,1))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Hori_Bar', 'Cub', P=(-12,-6,-6)+i*u, G=(2,11,1))

​

draw('Hori_Bar', 'Cub', P=(-12,5,-6), G=(2,1,12))

​

draw('Back', 'Cub', P=(1,6,-6), G=(11,1,12), theta=0°)

Deterministic

[ ]

chair with armrests

​

​

Deterministic

LLVM

Codes

ShapeCode Transformer

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

[Zakai, 11]

LLVM

[Lattner et al., ISCGO’04]

Codes

ShapeCode Transformer

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

[Zakai, 11]

LLVM

[Lattner et al., ISCGO’04]

Codes

ShapeCode Transformer

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

[Zakai, 11]

[Zakai, 11]

Point-VQVAE

PointCloud Encoder

PointCloud Decoder

CodeBook

Encoder

3x

1x1, 512

1x1, 2048

1x1, 512

+

MaxPooling

512, 1024

1024, 2048x3

Point-VQVAE

PointCloud Encoder

PointCloud Decoder

CodeBook

Encoder

3x

1x1, 512

1x1, 2048

1x1, 512

+

MaxPooling

512, 1024

1024, 2048x3

Point-VQVAE

PointCloud Encoder

PointCloud Decoder

CodeBook

Encoder

3x

1x1, 512

1x1, 2048

1x1, 512

+

MaxPooling

512, 1024

1024, 2048x3

Text & Program – Encoders & Decoders

BPE Encoding & Decoding Texts

Byte Pair Encoding

Text & Program – Encoders & Decoders

aaabdaaabac

​

​

​

BPE Encoding & Decoding Texts

Byte Pair Encoding

Text & Program – Encoders & Decoders

aaabdaaabac

​

ZabdZabac

Z=aa

​

​

BPE Encoding & Decoding Texts

Byte Pair Encoding

Text & Program – Encoders & Decoders

aaabdaaabac

​

ZabdZabac

Z=aa

​

ZYdZYac

Y=ab

Z=aa

​

BPE Encoding & Decoding Texts

Byte Pair Encoding

Text & Program – Encoders & Decoders

aaabdaaabac

​

ZabdZabac

Z=aa

​

ZYdZYac

Y=ab

Z=aa

​

XdXac

X=ZY

Y=ab

Z=aa

BPE Encoding & Decoding Texts

Byte Pair Encoding

Text & Program – Encoders & Decoders

aaabdaaabac

​

ZabdZabac

Z=aa

​

ZYdZYac

Y=ab

Z=aa

​

XdXac

X=ZY

Y=ab

Z=aa

X: 0, Y: 1, Z: 2

a: 3, c: 4, d: 5

​

BPE Encoding & Decoding Texts

Byte Pair Encoding

Text & Program – Encoders & Decoders

aaabdaaabac

​

ZabdZabac

Z=aa

​

ZYdZYac

Y=ab

Z=aa

​

XdXac

X=ZY

Y=ab

Z=aa

X: 0, Y: 1, Z: 2

a: 3, c: 4, d: 5

​

aaabdaaabac

[0, 5, 0, 3, 4]

BPE Encoding & Decoding Texts

Byte Pair Encoding

Text & Program – Encoders & Decoders

aaabdaaabac

​

ZabdZabac

Z=aa

​

ZYdZYac

Y=ab

Z=aa

​

XdXac

X=ZY

Y=ab

Z=aa

draw('Top', 'Square', P=(-3,-2,0), G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

X: 0, Y: 1, Z: 2

a: 3, c: 4, d: 5

​

aaabdaaabac

[0, 5, 0, 3, 4]

BPE Encoding & Decoding Texts

Byte Pair Encoding

Text & Program – Encoders & Decoders

aaabdaaabac

​

ZabdZabac

Z=aa

​

ZYdZYac

Y=ab

Z=aa

​

XdXac

X=ZY

Y=ab

Z=aa

draw('Top', 'Square', P=(-3,-2,0), G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

X: 0, Y: 1, Z: 2

a: 3, c: 4, d: 5

​

aaabdaaabac

[0, 5, 0, 3, 4]

BPE Encoding & Decoding Texts

Byte Pair Encoding

draw('Top', 'Square', P=(-3,-2,0), G=(2,7))

​

[3, -3, -2, 0, 2, 7, 0, 0]

​

Text & Program – Encoders & Decoders

aaabdaaabac

​

ZabdZabac

Z=aa

​

ZYdZYac

Y=ab

Z=aa

​

XdXac

X=ZY

Y=ab

Z=aa

draw('Top', 'Square', P=(-3,-2,0), G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

X: 0, Y: 1, Z: 2

a: 3, c: 4, d: 5

​

aaabdaaabac

[0, 5, 0, 3, 4]

BPE Encoding & Decoding Texts

Byte Pair Encoding

draw('Top', 'Square', P=(-3,-2,0), G=(2,7))

​

[3, -3, -2, 0, 2, 7, 0, 0] + min value = [3, 21, 22, 24, 26, 31, 24, 24]

​

​

​

Text & Program – Encoders & Decoders

aaabdaaabac

​

ZabdZabac

Z=aa

​

ZYdZYac

Y=ab

Z=aa

​

XdXac

X=ZY

Y=ab

Z=aa

draw('Top', 'Square', P=(-3,-2,0), G=(2,7))

​

for(i<2, 'Trans', u=(0,0,11))

draw('Sideboard', 'Cub', P=(-11,-2,-7)+i*u, G=(10,6,3))

​

draw('Back_sup', 'Cub', P=(-1,2,-2), G=(3,1,4))

​

draw('Back', 'Cub', P=(2,3,-6), G=(10,3,12), theta=16°)

​

X: 0, Y: 1, Z: 2

a: 3, c: 4, d: 5

​

aaabdaaabac

[0, 5, 0, 3, 4]

BPE Encoding & Decoding Texts

Byte Pair Encoding

draw('Top', 'Square', P=(-3,-2,0), G=(2,7))

​

[3, -3, -2, 0, 2, 7, 0, 0] + min value = [3, 21, 22, 24, 26, 31, 24, 24]

​

​

​

CodeBook

Encoder

Decoder

Program

VQVAE

ShapeCode Translator

Codes

ShapeCode Translator

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

ShapeCode Translator

Codes

ShapeCode Translator

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

(Text, PointCloud) Pair

ShapeCode Translator

Codes

ShapeCode Translator

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

(Text, PointCloud) Pair

ShapeCode Translator

Codes

ShapeCode Translator

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

(Text, PointCloud) Pair

ShapeCode Translator

Codes

ShapeCode Translator

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

(Text, PointCloud) Pair

ShapeCode Translator

Codes

ShapeCode Translator

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

(Text, PointCloud) Pair

ShapeCode Translator

Codes

ShapeCode Translator

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

(Text, PointCloud) Pair

ShapeCode Translator

Codes

ShapeCode Translator

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

[Text, PointCloud] Pair

[PointCloud, Text]

[PointCloud, Program]

ShapeCode Translator

Codes

ShapeCode Translator

Codes

Probabilistic

Frontend

Backend

Text Decoder

PointCloud Encoder

ShapeProgram

Encoder

Text Encoder

PointCloud Decoder

ShapeProgram

Decoder

[Text, PointCloud] Pair

[PointCloud, Text]

[PointCloud, Program]

Data

3D Assets: A total of 140,419 shapes of 144 categories from ShapeNet, ABO, Shape Program datasets.

​

Shape-Text Pairs: A total of 107,371 (Shape, Structure-Related Text) pairs from 20,355 shapes with 9.47 words per description on average and 26,776 unique words.

​

By adjusting Shape-Text pairs from Text2Shape, ABO, and ShapeGlot datasets: (1) removing artificial colors, textures, and materials descriptions; (2) removing brand name, dimensions, and other geometry-non-related words; (3) make sure descriptions are informative.

​

​

Data

3D Assets: A total of 140,419 shapes of 144 categories from ShapeNet, ABO, Shape Program datasets.

​

Shape-Text Pairs: A total of 107,371 (Shape, Structure-Related Text) pairs from 20,355 shapes with 9.47 words per description on average and 26,776 unique words.

​

By adjusting Shape-Text pairs from Text2Shape, ABO, and ShapeGlot datasets: (1) removing artificial colors, textures, and materials descriptions; (2) removing brand name, dimensions, and other geometry-non-related words; (3) make sure descriptions are informative.

​

Shape-Program Pairs: A total of 80,000 (Chair, Program) pairs and 40,000 (Table, Program) pairs.

​

PartialShape-CompleteShape Pairs: A total of 20 x 11,028 (Partial Point Cloud, Complete Point Cloud) pairs from 11,028 shapes.

Experiments – Text2PointCloud

Closer Look into 2D Pretrained Model

Closer Look into 2D Pretrained Model

Round or Circular -> Rectangle or Rectangular.

Closer Look into 2D Pretrained Model

Compute cosine similarity over 1,358 shapes with a total of 6,697 texts.

(1) 0.2925 ± 0.02173 and (2) 0.2917 ± 0.02211

Round or Circular -> Rectangle or Rectangular.

Experiments – PointCloud Captioning

Experiments – PointCloud2Program

Experiments – PointCloud Completion

Thanks