Lab 5: How to Train Your Robot Dog!

Jaden Clark

CS 123

Let’s make Pupper walk!

CS 123

Lab Overview

1. Software installs, train your first policy
2. Reward tuning
1. what auxiliary rewards are important?
3. Deploy on Pupper
4. Domain randomization
• Make Pupper walk better in the real world!
5. Domain randomization + terrain + reward tuning

3

Software installs

• Copy the Colab
• Purchase Colab pro
• Save receipt
• Setup a wandb account
• Run the installs

4

Rewards in RL

​

Goal: optimize the reward

reward

state

action

5

What’s a good reward…

​

Well… the reward is a function of the of the state and action at time t

​

state: 12-dof motor positions + velocities, roll/pitch/yaw, base velocities / orientation, foot contact forces/position, etc

​

action: 12-dof motor position commands

6

We want Pupper to follow a velocity command

​

Is a velocity following command sufficient?

7

We want Pupper to follow a velocity command

​

Is a velocity following command sufficient?

​

​

​

​

​

​

​

In practice… NO

​

​

8

We want Pupper to follow a velocity command

​

Is a velocity following command sufficient?

​

​

​

​

​

​

​

In practice… NO

​

Why???

9

Teaching Pupper to walk is like teaching a human

​

​

​

10

Teaching Pupper to walk is like teaching a human

​

Can we just teach a baby to walk by giving candy when it goes forward??

​

​

11

Teaching Pupper to walk is like teaching a human

​

Can we just teach a baby to walk by giving candy when it goes forward??

​

​

Need to give it auxiliary tasks…

• first standing up
• walking while holding human to learn correct gait
• etc

​

​

12

Teaching Pupper to walk is like teaching a human

​

Shooting a basketball

• Learn the correct form

​

13

Auxiliary rewards

​

Guiding gradient to correct optimum

​

14

Pupper needs auxiliary rewards too

​

How to encourage Pupper to walk with the correct gait?

​

Linear combination of differentiable rewards:

​

• Efficiency
• Torque penalties
• DOF acceleration penalties
• Stability
• orientation penalty
• Unnecessary contacts
• knees / body hitting the ground
• Robustness to different terrain
• foot clearance

​

15

Pupper needs auxiliary rewards too

​

How to encourage Pupper to walk with the correct gait?

​

• Efficiency
• Torque penalties
• DOF acceleration penalties
• Stability
• orientation penalty
• Unnecessary contacts
• knees / body hitting the ground
• Robustness to different terrain
• foot clearance

​

reward definitions: https://github.com/Nate711/pupperv3-mjx/blob/main/pupperv3_mjx/rewards.py

​

​

16

RL Workflow

​

• Update reward weights

​

​

• Train policy

​

​

• Visualize policy + review curves in wandb

​

17

Domain randomization

​

System identification is never perfect…

​

Which terms to randomize?

​

18

units is meters

Policy deployment

​

• see the deploy file with: cat~/utils/robot.sh

​

download saved file from colab

​

Copy the .sh file on your local machine

​

make it executable

• chmod +x deploy_policy.sh

​

run the file with ./deploy_policy.sh /path/to/your/policy.json

​

ctrl-b + d to exit tmux session

​

​

19

RL Workflow

​

• Improve policy in sim
• Update reward weights
• Train policy
• Visualize policy + review curves in wandb
• Domain randomization
• Deploy on Pupper

​

​

20

Challenge: Train the most agile policy

1. Check out the linked articles for inspiration on good rewards
2. Experiment with heightfield configs

​

extra credit for best policies…

21

Quick Tips

Safety

• Do not deploy policies on the table
• Do not deploy policies that are highly unstable in sim
• Be prepared to cancel potentially harmful policies

General

• Everyone should tune their own policies, and get practice deploying on the Pupper
• Work together to decide which rewards to try out

22

General safety

• Do not try and jam the HDMI cables into the pupper - a lot have been breaking. The cable should should fit easily into the port without force
• Make sure Pupper is off power when modifying the hardware
• battery is out
• any time you are adjusting screws or manipulating hardware in any way

23

24

25

26

27