Diffusion Policy:�Visuomotor Policy Learning via Action Diffusion

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu,

Eric Cousineau, Benjamin Burchfel, Shuran Song

Presenter: Mingyo Seo

1. Multi-modal distribution�2. Sequential correlation�3. High precision

Policy learning from demostration

Policy representations

- Multi-modal distribution�- High-dimensional output�- Stable training

- Unstable to train

(need negative loss)

- Direct regression:

multi-modality X

- Categorical:

sensitive to hyperparameter

Conditional denoising diffusion process for actions of receding horizons

Key Ideas

Conditional denoising diffusion process for actions of receding horizons

Key Ideas

Closed-loop action sequence

Visual conditioning

Stable training

Denoising Diffusion Probabilistic Models (DDPM)

Problem formulation

Visual observation conditioning

• Approximate , not

​

​

​

​

​

• Accelerate inference speed
• End-to-end training of the visual encoder

• Training

No output

Data with noise

De-noised samples

Implementation: Architecture

• CNN
• Time series transformer

Implementation: Visual encoder

• ResNet-18 without pre-training
• End-to-end training
• Using pre-trained model (ImageNet) showed less promising results
• Modifications
• Global average pooling => Spatial softmax pooling
• Batch norm => Group norm

Implementation: Real-time control (DDIM)

• Denoising Diffusion Implicit Models (DDIM):

Non-Markovian process for fast computation (inference)

• DDIM shares the same objectives

Markovian

DDIM

Properties

• Multi-modal action distributions (short/long-horizon)

​

​

​

​

​

​

Properties

• Action sequence prediction
• Temporal action consistency
• Robustness to idle actions
• Reaction time (horizon length)

​

​

​

​

​

​

Properties

• Works well with position control
• Training stability

​

​

​

​

Evaluation in Simulation

Evaluation with a real robot

Evaluation with a real robot

Discussion