1 of 1

DexPoint: Generalizable Point Cloud Reinforcement Learning

for Sim-to-Real Dexterous Manipulation

Yuzhe Qin*, Binghao Huang*, Zhao-Heng Yin, Hao Su, Xiaolong Wang

Task Description:

​

https://yzqin.github.io/dexpoint

Results: Observed Cloud + Imagined Cloud -> Manipulation Video

• Relocate: pick up an object on the table and lift it to the target location.
• Door Opening: first rotate the lever of the door and then pull it to open the door.

Challenges:

• Heavy occlusion: the hand and object occlude each other during interaction.
• Low sample efficiency:
• Point Cloud: high-dim observation space
• Dexterous hand: high-dim action space

Imagined Hand Point Cloud

Objective:

Policy

Point Cloud Observation

Action

Env

Observed from Camera

Imagined from

Proprioception

RGB Image

Enable the robot see its “body”

1. Read joint position
2. Compute forward kinematics
3. Sample points from computed geometry of robot links

-> better handle occlusion

Reward Design Regulated by Oracle Contact

Philosophy: observation should be accessible in real-world, but reward can be anything oracle to accelerate training.

Use oracle contact info to control reward -> higher sample efficiency

Failure Case

Stability Test

Architecture

Comparision: Point Cloud Observation between Real and Sim