Comparative Analysis of Deep Reinforcement Learning for Trajectory Estimation of Autonomous Vehicles In Highways

Prepared by:

Md Adnan Faisal Hossain

ID: 1606063

​

Supervised by:

Dr. S.M. Mahbubur Rahman

EEE, BUET

Motivation for Autonomous Driving

2

• Reduction in road traffic accidents
• Independent transportation for people with disabilities
• Increased human productivity
• Reduced traffic congestion
• Reduced travel time
• Improved fuel economy
• Positive effect on the environment

​

​

Highway Autonomous Driving

A Review of Motion Planning for Highway Autonomous Driving - Laurène et al

3

• A long-term goal of reaching destination
• Predicting the behavior of other road-users
• Trajectory estimation for performing short-term maneuvers
• Object detection and collision avoidance
• Vehicle speed control
• Vehicle lane-keeping and lane-changing
• Merging rows of vehicles and entering/exiting highway

Why use Reinforcement Learning?

4

• Difficult to model the non-linear dynamics of vehicles using traditional approaches
• All driving scenarios cannot be ascertained under a pre-defined set of rules
• Complex non-linear control systems need to be used
• Reinforcement Learning provides an alternative to physical modelling of the vehicle for learning the optimal control strategy
• The vehicle (agent) learns to drive from “experience” generating its own set of rules
• Datasets are not required to train a model

Reinforcement Learning at a glance

5

• Agent is the autonomous ego vehicle that interacts with the environment
• States are the parameters of the agent with respect to the environment (position, velocity etc.)
• The policy π is the set of rules followed by the RL algorithm to determine the probabilities of different actions at a given state
• The reward is a positive incentive for taking “good” actions
• The goal of the RL algorithm is to obtain the optimal policy π^* using an iterative process

​

​

​

Mathematical Foundations of Reinforcement Learning

6

Reinforcement Learning is a method to obtain an optimal policy that generates an optimal state-action-value function from a given Markov Decision Process (MDP)

​

MDP – (s, A, {P_sa}, γ, R)

S : set of states

A : set of actions

P_sa(s`) : state transition probability

γ: Discount factor

R_t : Reward from current action in current state

G_t : Total discounted reward (payoff)

V^π(s) : Value function

V^*(s) : Optimal Value function

Q^π(s, a) : Action-value (Q) function

π(s) : RL Policy

π^*(s) : Optimal RL Policy

​

Expected Future Reward

Basic Actor-Critic Reinforcement Learning Algorithm

7

​

• Initialize replay memory to capacity D
• Define 4 Deep Neural Networks and initialize them randomly-
• Actor Policy Network, Actor Target Network
• Critic Policy Network, Critic Target Network

For each episode:

• Select initial states s₀

Repeat for each time step:

• Generate random number between [0, 1]
• If n < 𝜀 , take random (off-policy) action
• If n > 𝜀 , take greedy (on-policy) action using Actor Policy Net
• Use P_sa(s^’) for each state s to determine s^’(next state)
• Calculate the reward R_t for taking action a in state s
• Store the experience tuple (s, a, R_t, s^’) in replay memory buffer
• Take samples from the replay memory to:
• Update Actor Policy Net:
• Use Actor Policy Net to get actions a for states s
• Plug s and a into Critic Policy Net to get Q(s, a)
• Use gradient ascent to update Actor Policy Net by maximizing Q(s, a)
• Update Critic Policy Net:
• Use Actor Target Net to get actions a^’ from next states s^’
• Plug a^’ and s^’ into Target Critic Net to get Q^’(s^’, a^’)
• Plug s and a into Critic Policy Net to get Q(s, a)
• Define loss function: δ = R_t + Q^’(s^’, a^’) – Q(s, a)
• Minimize δ using gradient descent to update Critic Policy Net
• Every N steps copy Policy Network parameters into Target Network

​

​

​

Actor

Policy

Network

Critic

Policy

Network

Current State (s)

Action values (a)

Q(s, a)

Actor

Target

Network

Critic

Target

Network

Next State (s^’)

Action values (a^’)

Q^’(s, a)

Description of the work

8

Contribution to the literature:

• Quantitative comparison of four distinct RL algorithms on the highway driving task

​

Techniques incorporated to improve performance:

• Incorporation of a safety net around RL model to avoid rear-end collisions
• Using a set of continuous-time polynomial trajectories as the action space instead of discrete throttle and steering actions
• Use of Frenet Space co-ordinates for trajectory generation

​

​

Reinforcement Learning Algorithms

“Continuous control with deep reinforcement learning” - Timothy et al

“Proximal Policy Optimization Algorithms” – John et al

“Trust Region Policy Optimization” – John et al

“Asynchronous Methods for Deep Reinforcement Learning” – Volodymyr et al

9

• Deep Deterministic Policy Gradients (DDPG)
• Trust Region Policy Optimization (TRPO)
• Proximal Policy Optimization (PPO)
• Advantage Actor Critic (A2C)

All the algorithms are implemented in an actor-critic style DRL model

Deep Reinforcement Learning model setup

10

​

Typical end-to-end Deep RL model setup for autonomous driving –

• State space: Raw sensory measurements such as image pixels from a camera sensor
• Action space: Discretized values of steering angle (θ), and throttle (T)

​

Deep RL model setup in the proposed literature –

• State space: Co-ordinates in the Frenet-space (s,d) of the ego vehicle and N surrounding vehicles (obstacles) normalized between [-1,1]
• Action space: Set of continuous-time polynomial trajectories characterized by the tuple A = {t_f , v_f , d_f } normalized between [-1, 1]

​

​

Deep Reinforcement Learning model setup

"Optimal trajectories for time-critical street scenarios using discretized terminal manifolds" - Werling et al

11

Action space

T: Set of spatiotemporal trajectories called lattice

T = {τ₁ , τ₂ ……. τ_m}

Trajectory τ = [ s(t) , d(t) ] , τ ϵ R²

​

• τ is generated from terminal manifolds: speed v_f , lateral position d_{f ,} and arrival time t_f
• Terminal manifolds are continuous-time outputs of a policy network

​

Policy Network

12

(15, 2, 30)

Number of filters

Actor

(1,2)

Ego

(14,2) obstacles

(128, 128)

Front

Image

(128, 128)

Rear

Image

(128, 128, 3, 30)

(128, 128, 3, 30)

(𝛼, 𝛽)

State from past 30 time steps

Conv 2D

(32, 4, 2)

Conv 2D

(64, 4, 1)

Conv 2D

(64, 3, 1)

Global Average pooling

Conv 3D

(64, 4, 2)

Conv 3D

(32, 4, 1)

Global Average pooling + reshape

Conv 3D

(64, 4, 2)

Conv 3D

(32, 4, 1)

Global Average pooling + reshape

Critic

TD error

Output

(3,1)

t_f

d_f

v_f

Filter size

Stride

(1, 1, 128)

Q(s, a)

Input States

Continuous Actions

Reward Function

"Integrating Deep Reinforcement Learning with Model-based Path Planners for Automated Driving" - Ekim et al

13

• Vehicle is penalized for collision or going off-road
• Vehicle is penalized for driving below the minimum speed (subject to traffic conditions)
• Lane changes resulting in speed gain above threshold are rewarded (otherwise lane changes are penalized)
• Vehicle penalized if the relative distance to preceding car decreases below safe threshold

, if collision or off-road

, otherwise

, speed gain > 5 km/h

, otherwise

R_t : Reward obtained for performing action a in state s

r_c : Penalty for collision

r_v : reward for driving above minimum speed

r_lc : lane change reward/penalty

r_d : reward for safety

​

​

Training Process

"Congested Traffic States in Empirical Observations and Microscopic Simulations" - Martin et al

14

Training steps:

• The ego vehicle and obstacles are spawned at random points on the highway at the start of each episode
• Episode is terminated if ego drives for 500 m, goes off-road or makes a collision
• Ego vehicle uses its sensors to get the value of input states
• Ego vehicle executes actions based on RL policy and receives reward
• RL policy is updated on each time-step based on total future reward

Safety Net:

• Safe action substitutes RL policy action when collision is imminent
• Used to manually avoid rear-ending obstacles
• Safety Net is applied during both training and testing

​

​

T_C : Calculated time to collision

T_HB : Threshold for hard-braking

T_B : Threshold for braking

a_s: Safety action

Simulation Environment

"CARLA: An Open Urban Driving Simulator" - Alexey et al

15

• CARLA v0.9.9.2 simulator is used as the high-fidelity urban driving platform
• The simulator enables the modeling of vehicle dynamics, environment stochasticity and sensory measurements
• Tesla Model 3 was selected as ego vehicle
• The highway platforms of “Town 3” of the simulator were used for simulation purposes
• GEFORCE RTX 3060 GPU was used for training the DRL agent

CARLA Environment setup:

• Five-lane highways with medium to steep curves are used for training
• Anywhere between 1 to N vehicles are placed within 150 m of the ego vehicle
• 14 discretized regions are defined around the ego vehicle –

Sample Simulations

16

17

Quantitative Analysis

"Quantitative Evaluation of Autonomous Driving in CARLA" - Shang et al

18

Performance Metrics:

Quantitative Analysis

19

Results:

• Monte Carla Simulation was performed for testing over 50 iterations with each iteration consisting of 100 episodes
• The results obtained for 4 different DRL algorithms over the performance metrics were compared
• The mean reward curves of the 4 DRL algorithm training processes were also compared

​

Mean Reward Curve (Training Curve):

20

Conclusion and Future Work

21

Conclusion:

• PPO2 algorithm failed to converge to a meaningful optimal policy
• TRPO model converged to a policy with the maximum mean reward
• TRPO model converged to a driving policy with a focus on speed
• DDPG model converged to a driving policy with a focus on safety and comfort without penalizing speed too much
• A2C model provides a trade-off between speed and comfort
• The driving performance indicators for PPO2 model do not provide any reasonable conclusions
• Converging to the highest mean reward does not guarantee optimal driving behavior

​

Future work:

• Implement a reward function that promotes rider safety and comfort exclusively
• Take into account scenarios of entering and exiting highways into our highway driving model
• Create a model for urban driving and form an ensemble of the two models
• Test the performance of other state-of-the art DRL algorithms