Presenter: Hanwen Jiang

2023.02.10

Task

Synthesize task-dependent and physically plausible actions

• Scene interaction tasks: carrying, sitting and lying down

Motivation

• Synthesizing human motion is a fundamental CG task
• Motions are driven by the need of interactions
• Actions are restricted by the environment
• Actions are afforded by the environment

Related work

• Kinematic methods
• Physics-based methods

Related work

Kinematic methods: Use deep networks to regress human motions

Cons:

• Require large amounts of high-quality 3D data
• Networks are not good at long-horizon auto-regression tasks
• Need hand-crafted physical constraints
• Poor generalization ability

​

Related work

Physics-based methods: use RL for control & simulation for physically plausibility

Pros:

• Physical plausibility
• Not data-hungry

​

Cons:

• Synthesized motions may not be human-like (can be solved by imitation learning & motion tracking)
• Hard to do imitation learning with scene-level interactions

Method

Idea: Physics-based method with adversarial learning

Two-component reward function:

Method

Adversarial training details

• Objective function of training discriminator:

​

​

​

• Specification of style reward:

State and Action Representation

State representation: assume the access of accurate object scene information

​

​

​

​

​

Action representation: joint target rotations (use PD controller) represented by exponential map

Dataset

SAMP dataset

• Captures MoCap clips of sitting and lying down behaviours
• With object states and CAD models (only 7 objects)
• 100 minutes in total

​

A carry task dataset:

• Captures 15 MoCap clips of a subject carrying a single box
• Box are initialized with random location, and its motion is also tracked

Training

Sample diverse objects from ShapeNet

• 350 unique instances to improve generalizability
• Randomize the scale of objects

​

Episode initialization

• Randomly initialize the character and objects by sampling from the dataset
• Initialize close to the completion state of the task

Training

A combination of GAIL and PPO

​

Other tricks:

• Early termination
• Replay buffer to stabilize the training of discriminator

Results

1. Measure overall success rate
1. Random initialization with 21 unseen object, 4096 trials in total

Results

2. Measure success rate under physical perturbations

• Pelt the character with 20 projectiles of weight 1.2 kg at random time step
• Randomly move the object during execution

Results

3. Comparison with previous art

• NSM, SAMP, Learning to sit (both are kinematic data-driven methods)

Conclusion

The paper propose a physics-based method for synthesizing human-scene interactions using adversarial training

​

Cons:

• Metrics that measure how the learned actions are human-like are missed
• Assume accurate object state knowledge & current object state limits fine-grained interactions
• The method is tested on one-object scenes
• The method can be hard to scale up & synthesize general-purpose motions