1 of 42

How to train your dragon ATCo

A BlueSky-Gym reinforcement learning workshop

PRIMARY COLORS

SECONDARY COLORS

2 of 42

Overview of the workshop

2

Title only

PRIMARY COLORS

SECONDARY COLORS

3 of 42

Today you will learn how to reproduce the following video:

3

During this workshop, you will:

​

  1. Train your own ATC agents
  2. Learn how to adapt environments to your own needs
  3. Lower the barrier to entry
  4. Explore potential next steps

​

Title only

PRIMARY COLORS

SECONDARY COLORS

4 of 42

Overview of the workshop

4

Introduction & Theory

Q&A

The Next Steps:

Customizing Environments

Hands-On Session

10 min

20 min

30 min

5 min

15 min

Title only

PRIMARY COLORS

SECONDARY COLORS

5 of 42

Overview of the workshop

5

Introduction & Theory

Q&A

The Next Steps:

Customizing Environments

Hands-On Session

Title only

PRIMARY COLORS

SECONDARY COLORS

6 of 42

Reinforcement Learning applications in ATC are growing

6

Increasing research interest from the community into Reinforcement Learning

​

This leads to

  • Fast progress (+)
  • Novel & creative concepts (+)
  • Large areas of overlap (-)
  • Diverging benchmarks & SOTA (-)

​

  • Hype (+-)

​

​

Title only

PRIMARY COLORS

SECONDARY COLORS

7 of 42

But what is Reinforcement Learning actually?

7

Environment

Agent

Observation

Reward

Action

Markov Decision Process (MDP):

Machine learning methods for MDPs

The goal is to:

  • Map observations to actions through a ”policy” which:
  • Maximizes the sum of rewards.

Often problems arise because of:

  • Exploration vs Exploitation
  • Low sample efficiencies
  • Instabilities
  • Lack of consistent convergence

Which is why many different algorithms exist

Recommended reading:

Reinforcement Learning: An introduction – Barto & Sutton

​

Title only

PRIMARY COLORS

SECONDARY COLORS

8 of 42

Various libraries help standardize RL research

8

Gymnasium: standard environments and API

Stable Baselines3: algorithms and training

Environment

Agent

Observation

Reward

Action

Title only

PRIMARY COLORS

SECONDARY COLORS

9 of 42

Various libraries help standardize RL research

9

Gymnasium: standard environments and API

Stable Baselines3: algorithms and training

MuJoCo

Classic Control

Atari, 3rd party and more

Standardized MDP formulation and code structure:

Contains a large collection of different DRL algorithms capable of handling:

  • Vectors, dictionaries and images for the observation (combinations also work)
  • Discrete and continuous action spaces
  • Parallel running of multiple environments

​

Perfect for proof of concept or simple implementations, but customization / control is lacking for more advanced features.

Title only

PRIMARY COLORS

SECONDARY COLORS

10 of 42

BlueSky helps standardize and simplify ATC research

10

Build-in conflict detection capabilities

Aircraft performance models based on OpenAP or BADA

Large database of airports and waypoints

Fast-time simulation within Python

Highly customizable through plugins

Title only

PRIMARY COLORS

SECONDARY COLORS

11 of 42

BlueSky-Gym = BlueSky + Gymnasium

11

Build on 2 widely used packages

Diverse set of environments

Random generation of the scenarios

Fully open-source

Combining BlueSky with Gymnasium allows for easy adoption and extensive examples on both sides of the implementation

Providing a large set of examples to lower the barrier to entry for specific usecases

Random generation of the initial conditions for each episode increases generalization of the trained models

Open-source hopefully helps drive community-wide development and easier modification to personal needs

Title only

PRIMARY COLORS

SECONDARY COLORS

12 of 42

Overview of the workshop

12

Introduction & Theory

Q&A

The Next Steps:

Customizing Environments

Hands-On Session

Title only

PRIMARY COLORS

SECONDARY COLORS

13 of 42

Overview of the workshop

13

Introduction & Theory

Hands-On Session

Q&A

The Next Steps:

Customizing Environments

Title only

PRIMARY COLORS

SECONDARY COLORS

14 of 42

Navigating to BlueSky-Gym

14

https://github.com/TUDelft-CNS-ATM/bluesky-gym

Title only

PRIMARY COLORS

SECONDARY COLORS

15 of 42

Training your first agent

15

Installing packages and dependencies

Setting up the folder structure

Writing some code

Running the training script

Title only

PRIMARY COLORS

SECONDARY COLORS

16 of 42

Installing BlueSky-Gym and Stable Baselines3

16

Using conda

Using your own

Using virtualenv

If anaconda is your main environment manager:

conda create –n bsg python

conda activate bsg

pip install bluesky-gym

​

If virtualenv is your main environment manager:

ensure python => 3.11 is installed

python3.11 -m venv bgs

activate:

(mac) source bsg/bin/activate

(win) bsg\Scripts\activate

or manually

pip install bluesky-gym

​

​

​

​

If you have any other preferred package manager:

create environment with python=>3.11

pip install bluesky-gym inside env

​

​

​

Title only

PRIMARY COLORS

SECONDARY COLORS

17 of 42

Training your first agent

17

Installing packages and dependencies

Setting up the folder structure

Writing some code

Running the training script

Title only

PRIMARY COLORS

SECONDARY COLORS

18 of 42

Setting up the folder structure

18

Create a new folder for this workshop

Inside this folder, create:

logs/

models/

train.py

Title only

PRIMARY COLORS

SECONDARY COLORS

19 of 42

Training your first agent

19

Installing packages and dependencies

Setting up the folder structure

Writing some code

Running the training script

Title only

PRIMARY COLORS

SECONDARY COLORS

20 of 42

Writing some code

20

Import packages

From SB3, import the algorithm you want to use:

​

Continuous actions:

  • SAC, DDPG, PPO, TD3, A2C, …*

Discrete actions (currently not in BSG):

  • PPO, DQN, A2C, TRPO, …*

​

​

*see: https://stable-baselines3.readthedocs.io/en/master/guide/algos.html for more info.

Title only

PRIMARY COLORS

SECONDARY COLORS

21 of 42

Writing some code

21

Select the environment you wish to train your agent on.

​

Currently available*:

  • DescentEnv-v0
  • PlanWaypointEnv-v0
  • HorizontalCREnv-v0
  • VerticalCREnv-v0
  • SectorCREnv-v0
  • StaticObstacleEnv-v0
  • MergeEnv-v0
  • All Gymnasium environments

​

​

​

*see: https://github.com/TUDelft-CNS-ATM/bluesky-gym/tree/main/bluesky_gym/envs

Select env

Import packages

Title only

PRIMARY COLORS

SECONDARY COLORS

22 of 42

Writing some code

22

Here we create the base model with the algorithm of your choice.

To change the hyper-parameters, use*:

[ALG](“MultiInputPolicy”,

env,

learning_rate = …,

gamma = …,

tau = …,

policy_kwargs = …,

etc..)

​

​

*see: https://stable-baselines3.readthedocs.io/

Train the model

change total_timesteps for today to something like 10e4

Import packages

Select env

Title only

PRIMARY COLORS

SECONDARY COLORS

23 of 42

Writing some code

23

Once the model is done training we can visualize the model by loading it, and re-making the environment with:

render_mode = ”human”

​

I recommend adding

input()

before loading the model to ensure you are ready to watch the policy.

​

Loading and

visualization

Import packages

Select env

Train the model

Title only

PRIMARY COLORS

SECONDARY COLORS

24 of 42

Training your first agent

24

Installing packages and dependencies

Setting up the folder structure

Writing some code

Running the training script

Title only

PRIMARY COLORS

SECONDARY COLORS

25 of 42

Running the training script

25

Because we put verbose=1, we can observe the progress in the terminal:

Title only

PRIMARY COLORS

SECONDARY COLORS

26 of 42

Training your first agent

26

Installing packages and dependencies

Setting up the folder structure

Writing some code

Running the training script

Title only

PRIMARY COLORS

SECONDARY COLORS

27 of 42

Overview of the workshop

27

Introduction & Theory

Hands-On Session

Q&A

The Next Steps:

Customizing Environments

Title only

PRIMARY COLORS

SECONDARY COLORS

28 of 42

Overview of the workshop

28

Introduction & Theory

Hands-On Session

Customizing Environments

Q&A

The Next Steps:

Title only

PRIMARY COLORS

SECONDARY COLORS

29 of 42

Creating your own clone of BlueSky-Gym

29

Title only

PRIMARY COLORS

SECONDARY COLORS

30 of 42

The structure of BlueSky-Gym (environments)…

30

Environments

Wrappers for customizing

Example training scripts

Main training loop

Title only

PRIMARY COLORS

SECONDARY COLORS

31 of 42

The structure of BlueSky-Gym (environments)…

31

Title only

PRIMARY COLORS

SECONDARY COLORS

32 of 42

… and how to change it for your needs

32

Subclassing

  • Allows you to change some code, without having to copy everything
  • Useful for just making changes to the MDP without changing core functionality

​

Using wrappers

  • Not covered in this workshop
  • Allows you to add functionality on top of existing functions
  • Check bluesky_gym/wrappers for examples

​

​

Title only

PRIMARY COLORS

SECONDARY COLORS

33 of 42

Putting it to practice (live coding session)

33

​

​

​

Creating a subclass to change some core functionality of an existing environment.

Think of:

  • Adding something to the observation
  • Changing the reward function
  • Altering the mapping of the actions
  • Add some extra logging through _get_info()
  • …

​

Make sure that the subclass has access to the parent class, for example by adding the subclass in the same file.

​

Title only

PRIMARY COLORS

SECONDARY COLORS

34 of 42

Registering your new environment

34

Bluesky_gym/envs/__init__.py

Bluesky_gym/__init__.py

Title only

PRIMARY COLORS

SECONDARY COLORS

35 of 42

Running your new environment

35

‘JansEnv-v0’

Title only

PRIMARY COLORS

SECONDARY COLORS

36 of 42

Overview of the workshop

36

Introduction & Theory

Hands-On Session

Customizing Environments

Q&A

The Next Steps:

Title only

PRIMARY COLORS

SECONDARY COLORS

37 of 42

Overview of the workshop

37

Introduction & Theory

Hands-On Session

Customizing Environments

The Next Steps:

Q&A

Title only

PRIMARY COLORS

SECONDARY COLORS

38 of 42

Multi-Agent RL through pettingzoo

38

​

​

​

PettingZoo* is a gymnasium alternative for multi-agent applications.

  • Environments can easily be converted to the PettingZoo format by maintaining a list of agents and looping through them.
  • Each agent will then act according to its own or a shared policy.
  • See my fork for some examples: https://github.com/jangroter/bluesky-gym/tree/petting_zoo/bluesky_zoo

​

​

​

​

* https://pettingzoo.farama.org/index.html

Title only

PRIMARY COLORS

SECONDARY COLORS

39 of 42

Creating your own environments from scratch

39

Take inspiration from existing environments:

​

You can also create your own environments.

For this I recommend reading up on BlueSky and Gymnasium documentation

Don’t forget to register your new environment (see slide 33)

​

+

Title only

PRIMARY COLORS

SECONDARY COLORS

40 of 42

Deploying learned models in BlueSky

40

Because the models are trained in BlueSky environments, with little additional code, trained models can be deployed in BlueSky through plugins

​

The plugin requires:

  • An __init__() function that loads your model
  • _get_obs()
  • _get_action()
  • A timed update() function that calls your model

​

Title only

PRIMARY COLORS

SECONDARY COLORS

41 of 42

Overview of the workshop

41

Introduction & Theory

Hands-On Session

Customizing Environments

The Next Steps:

Q&A

Title only

PRIMARY COLORS

SECONDARY COLORS

42 of 42

Reinforcement Learning applications in ATC are growing

42

Increasing research interest from the community into Reinforcement Learning

​

This leads to

  • Fast progress (+)
  • Novel & creative concepts (+)
  • Large areas of overlap (-)
  • Diverging benchmarks & SOTA (-)

​

  • Hype (+-)

​

​

Title only

PRIMARY COLORS

SECONDARY COLORS