1 of 12

Learning, Perception, and Abstraction for Long-Horizon Planning CoRL 2022

PlAnning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation

Xingyu Lin*, Carl Qi*, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, David Held

16x

2 of 12

Sequential Deformable Object Manipulation with tools

  • Challenges:
    • Long-horizon reasoning over use of different tools
    • High-dimension state of deformable objects

Carl Qi - CoRL 2022

2

12/12/22

3 of 12

A planning approach with generalization

Carl Qi - CoRL 2022

3

12/12/22

1. Generalization to more entities

2. Generalization to longer-horizon

Cut, Rearrange, Spread (CRS)

CRS-Twice

Cut

Spread

Push

Training

Testing

observation

4 of 12

Step 1: Learn short-horizon skills

Carl Qi - CoRL 2022

4

12/12/22

Demonstration trajectories for each skill

Goal-conditioned BC

skill policy

PointNet++

One policy for each skill

Cut

Spread

Push

5 of 12

Carl Qi - CoRL 2022

5

12/12/22

point cloud

scene

decomposition

(DBSCAN)

3D encoder

3D encoder

Latent Set

Step 2: Neural Spatial Temporal Abstraction

6 of 12

Carl Qi - CoRL 2022

6

12/12/22

goal

point cloud

Latent Set

Step 2: Neural Spatial Temporal Abstraction

observation

point cloud

demonstration

cut skill

Latent Set

Distance to the goal

Can I reach the goal from here?

[0, 1]

feasibility

cost

set feasibility predictor

set cost predictor

Hungarian

matching

MLP

MLP

Maxpool

MLP2

7 of 12

Carl Qi - CoRL 2022

7

12/12/22

goal

observation

unknown subgoal

backprop

known latent vectors

cut feasibility

push feasibility

spread feasibility

set reward predictor

trajectory score

cut -> push -> spread

Step 3: Latent Space Planning with Skill Abstraction

Enumerate all the skill skeletons and structures and pick one with the highest score

8 of 12

High Level Idea

1. Learn short-horizon skills

​

Spread

Gather

Push

Cut

Lift

3. Latent space planning with skill abstractions

Initial point cloud

Goal

sub-goal 1

sub-goal 2

sub-goal 3

2. Neural spatial temporal abstraction

latent set representation

feasibility of reaching one state from another

distance to goal

9 of 12

Quantitative results

Carl Qi - CoRL 2022

9

12/12/22

Spatial abstraction enables effective generalization to more entities and longer horizon

Generalization Tasks

10 of 12

Real world results

Carl Qi - CoRL 2022

10

12/12/22

CRS-Twice, Robot execution

16X

11 of 12

Conclusion

  • We propose a 3-Step solution to long-horizon deformable object manipulation
    1. Learn Short-horizon Skills
    2. Neural Spatial Temporal Abstraction
    3. Latent Space Planning with Skill Abstraction
  • We can generalize to tasks with
    • More entities
    • Longer-horizon

Carl Qi - CoRL 2022

11

12/12/22

16x

12 of 12

Collaborators & Sponsors

Carl Qi - CoRL 2022

12

12/12/22

David Held

Chuang Gan

Zhiao Huang

Yunzhu Li

Yunchu Zhang

Katerina Fragkiadaki

Xingyu Lin

16x