Towards Continuous Control for Mobile Robot Navigation:

A Reinforcement Learning and SLAM Based Approach

Nicolò Botteghi, Khaled Mustafa, Beril Sirmaçek, Mannes Poel, Stefano Stramigioli

​

Enschede, 12/6/2019

Structure of the presentation

1. Introduction

​

• Theoretical Background

​

• Motion Planner Design and Implementation

​

• Evaluation

​

• Conclusions

1. Introduction

• Autonomous navigation is a complex task

​

• Commonly used techniques fail when the

environment is unknown (i.e. potential fields)

​

• The challenge of navigation can be formulated

as a reinforcement learning (RL) problem

1.1 Reinforcement Learning

Agent

Environment

action

At

reward

Rt

state

St

Rt+1

St+1

Actor

Environment

action

reward

state

2. Actor-Critic algorithm

Critic

TD error

2.1. Deep Deterministic Policy Gradient (DDPG)

• Model free algorithm

​

• Continuous actions and state space

​

• Off-policy

2.1. Deep Deterministic Policy Gradient (DDPG)

Policy evaluation:

​

• Estimate the action-value function by

Temporal Difference learning (TD-learning)

​

​

​

​

TD-error

2.1. Deep Deterministic Policy Gradient (DDPG)

Policy improvement:

​

• Adjust the parameter of the policy with respect to the

estimated

2.1. Deep Deterministic Policy Gradient (DDPG)

Tricks for stabilising the learning:

​

1. Experience replay buffer

​

• Actor and critic are updated by sampling mini batches from the replay buffer
• Breaks temporal correlations between the samples

​

• Target networks

​

• Time-delayed copies of the actor and the critic that track the learned networks
• Update of the the target actor and target critic network has different rate than

the update of the actor and the critic

​

​

​

​

​

2.2. Simultaneous Localization and Mapping (SLAM)

• Estimate the pose of the robot

2.2. Simultaneous Localization and Mapping (SLAM)

• Estimate the pose of the robot
• Create a map of the environment

2.2. Simultaneous Localization and Mapping (SLAM)

• Estimate the pose of the robot
• Create a map of the environment
• Simultaneously!

Figure reproduced from: https://www.google.com/search?q=chicken+egg+problem&source

=lnms&tbm=isch&sa=X&ved=0ahUKEwjdn-aO_NbiAhXILFAKHSDpAtYQ_AUIECgB&biw=1440&bih=

761#imgrc=gWJhw3zqkpUP3M

2.2. Simultaneous Localization and Mapping (SLAM)

2.2. Simultaneous Localization and Mapping (SLAM)

​

The setup:

​

• Grid-based SLAM with Rao-Blackwellized particle filter

​

• Skid-Steering mobile robot —> velocity commands (linear and angular)

​

• 180 degrees LiDAR and wheels encoders

2.2. Simultaneous Localization and Mapping (SLAM)

Rao-Blackwellized Particle Filter

​

2.2. Simultaneous Localization and Mapping (SLAM)

Rao-Blackwellized Particle Filter

​

Mapping with

known poses

2.2. Simultaneous Localization and Mapping (SLAM)

Rao-Blackwellized Particle Filter

​

Mapping with

known poses

Particle filter

for state

estimation

2.2. Simultaneous Localization and Mapping (SLAM)

Rao-Blackwellized Particle Filter

​

Mapping with

known poses

Particle filter

for state

estimation

Occupancy grid-map

3. Motion Planner Design and Implementation

State

​

Laser

​

Velocity

​

Target

Action

​

Velocity

Neural Nets

3.1. Neural Networks’ Architecture

Actor network

Input

ReLU

​

512

units

ReLU

​

512

units

ReLU

​

512

units

Output

Sigmoid

Tanh

• Exploration noise on the actions:

3.1. Neural Networks’ Architecture

Actor network

Input

ReLU

​

512

units

ReLU

​

512

units

ReLU

​

512

units

Output

Sigmoid

Tanh

• Exploration noise on the actions:

Correlated OU-noise

3.1. Neural Networks’ Architecture

Critic network

Input

ReLU

​

512

units

ReLU

​

512

units

ReLU

​

512

units

Output

Linear

Input

3.2. Reward Function

“Reward function is the most important aspect in reinforcement

learning problem since the actions are selected in such a way that

the cumulative reward is maximized”

• For navigation tasks a commonly used reward function is:

where is the Euclidean distance

from the current position of the robot

to the target location

3.2. Reward Function

• Our approach aims at exploiting the knowledge about the map provided

by the SLAM algorithm in order to define a new reward function:

3.2. Reward Function

• Our approach aims at exploiting the knowledge about the map provided

by the SLAM algorithm in order to define a new reward function:

3.2. Reward Function

• Our approach aims at exploiting the knowledge about the map provided

by the SLAM algorithm in order to define a new reward function:

Map’s posterior

3.2. Reward Function

• Our approach aims at exploiting the knowledge about the map provided

by the SLAM algorithm in order to define a new reward function:

Map’s posterior

: occupied cells in

the neighborhood

​

: distance to the robot and the occupied cell

3.2. Reward Function

4. Evaluation

We compared the two rewards functions for different goal locations in

term of:

​

• Convergence speed

​

• Success ratio

​

• Number of collisions with obstacles

4. Evaluation

Experiment #1

Experiment #2

4. Evaluation

Experiment #1

Experiment #2

4. Evaluation

4. Evaluation

4. Evaluation

5. Conclusions

Wrap-up:

​

• Deep RL path planner for autonomous navigation in unknown

environment

​

• Integration of map knowledge in to the reward function

​

• Reward shaping based on the knowledge provides improvements

of the performance with respect to standard reward function

​

5. Conclusions

Current and future work:

​

• Generalization to different target locations

​

• Generalization to unseen environments

​

• Transfer policy learned in simulation to real robot

Thank you for your attention!!!