1 of 11

Reinforcement Learning Journey

2 of 11

RL Environments

OpenAI Gym

Custom Env

Gazebo

Carla

3 of 11

RL Algorithms

01

02

03

04

05

Uses Q-Table to map state action pair with expected reward.

DQN replaces Q-Table with a deep network.

DDPG uses Actor Critic for continuous action space.

Fast training

On policy and has a higher stability.

Q-Learning

DQN

DDPG

TD3

PPO

4 of 11

Our Workflow

Select an Environment

Set up Action & Observation space

Select an Algorithm

Design Reward Function

01

03

04

02

Train & Result Analysis

05

5 of 11

Q-Learning

OpenAI Gym Env

Custom Env

Gazebo

Action Space:

Direction: Up, Down, Left, Right

Observation Space:

Distance with food & enemy

Action Space:

Direction: Left, Right, NOP

Observation Space:

Position: [-1.2 to 0.6] (Discrete)

Velocity: [-0.07 to 0.07] (Discrete)

Action Space:

Direction: Left, Right, Forward

Observation Space:

Goal Left, Goal Right, Goal Center, No lane detected

6 of 11

Deep Q Network

OpenAI Gym Env

Custom Env

Carla

Action Space:

Direction: Up, Down, Left, Right

Observation Space:

NxN shape image

Action Space:

Steering angle: [-1.0 to 1.0] (Discrete)

Observation Space: Image

Action Space:

Hip & Knee (Torque/vel)

Observation Space:

Speed, Angle…

7 of 11

Deep Deterministic Policy Gradient (DDPG)

Gazebo

Carla

Action Space:

Throttle: [0.0 to 1.0]

Steering angle: [-1.0 to 1.0]

Observation Space: Image

Action Space:

Steering angle: [-1.0 to 1.0]

Observation Space:

Angular direction: [-1.0 to 1.0]

8 of 11

Twin Delayed DDPG (TD3)

Gazebo

Lidar sensor (Obstacle detection)

Camera sensor (Lane detection)

Action Space:

Steering angle: [-60.0 to 60.0 degree]

Observation Space: Obstacle Distance

Action Space:

Steering angle: [-1.0 to 1.0]

Observation Space:

Angular direction: [-1.0 to 1.0]

9 of 11

Proximal Policy Optimization (PPO)

Gazebo

Carla

Action Space:

Throttle: [0.0 to 1.0]

Steering angle: [-1.0 to 1.0]

Observation Space: Image

Action Space:

Speed: [0.2 to 0.7]

Turn: [-0.3 to 0.3]

Observation Space: Lane Existence

10 of 11

Algorithm Comparison

Algorithms	Policy	Action Space	Pros	Cons
Q-Learning	Off Policy	Discrete	More expressive since specifying the value functions or policies is often more intuitive than specifying the model for the environment	Only works in environments with discrete and finite states and action spaces Requires a very large memory and a very long training time
DQN	Off Policy	Discrete	Q-table is replaced by neural network	Can’t handle continuous action space
DDPG	Off Policy	Continuous	Accommodates continuous action space	Creates risk by overestimating the Q values in the critic (value) network State vs reward graph fluctuates excessively
TD3	Off Policy	Continuous	Mitigates the overestimation bias problem of DDPG Trains much faster than DDPG	State vs reward graph does not converge
PPO	On Policy	Continuous	Regularizes the policy update and enables the reuse of training data	Hypersensitive to hyperparameter tuning such as choice of step size, learning rate, etc., along with their poor sample efficiency

11 of 11

More of our work

https://www.dsinnovators.com/ros