3 of 94

Acknowledgements

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

RL theory: Sutton and Barto book “Reinforcement Learning: an introduction”, Prof. David Silver lectures, Prof. Marios Kountouris (EURECOM) notes, Felix Wagner.

RL use cases at CERN: M. Schenk, J. Wulff, N. Bruchon, B. Goddard, S. Hirlander, V. Kain, N. Madysa, G. Valentino, F. Velotti, CERN Openlab, and the ML Community Forum.

If you find some of your materials without the proper credits, let me know and I will update the slides accordingly. Send me an email to matteo.bunino@cern.ch

4 of 94

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Image credits

5 of 94

Use cases motivating reinforcement learning (RL)

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Examples from Sutton and Barto book:

A master chess player makes a move. The choice is informed both by planning – anticipating possible replies and counterreplies - and by immediate, intuitive judgments of the desirability of particular positions and moves.

An adaptive controller adjusts parameters of a petroleum refinery operation in real time. The controller optimizes the yield/cost/quality trade-off on the basis of specified marginal costs without sticking strictly to the set points originally suggested by engineers.

A gazelle calf struggles to its feet minutes after being born. Half an hour later it is running at 20 miles per hour.

A mobile robot decides whether it should enter a new room in search of more trash to collect or start trying to find its way back to its battery recharging station. It makes its decision based on the current charge level of its battery and how quickly and easily it has been able to find the recharger in the past.

6 of 94

RL in games

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

“AlphaStar” wins Starcraft against

99.85% of human players (2019).

“AlphaGo” winning against

the Go world champion (2016).

https://www.nature.com/articles/s41586-019-1724-z

https://www.nature.com/articles/nature16961

Look for “AlphaGo’s move 37” on the web...

7 of 94

Reinforcement learning concepts

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

In a nutshell: learn a policy which maximizes the total expected reward over time.

Multistage decision-making process: the learner is not told which actions to take - it discovers which actions yield the most reward by trying them.

Not supervised learning: the agent learns from its own experience, not from representative examples.

Not unsupervised learning: maximize a reward signal instead of trying to find hidden structure in data.

Reinforcement learning (RL) peculiarities:

Learn by trial-and-error search
Delayed reward

…similar to human learning

8 of 94

Reinforcement learning concepts

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Goal: learn the optimal policy which maximizes the total expected reward over time.

Environment: can be accessible only partially. Some dynamics may remain obscure, and we get only what we can observe.

Interpreter: that’s defined by us. Sort of pre-processing. It builds the state based on the history of previous observations and interactions. It also implements the reward function.

State: describes the environment. It belongs to the states space

Reward (scalar number) is the only feedback the agent receives, which describes the “goodness” of the trajectory so far.

Action: sampled by the agent from the actions space

Interaction defines trajectories:

Observation

9 of 94

Reinforcement learning concepts

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Goal: learn the optimal policy which maximizes the total expected reward over time.

Policy is a mapping from perceived states of the environment to actions to be taken:

State value function: value of a state = total amount of reward an agent can expect to accumulate over the future, starting from that state (specifies what is good in the long run). �It estimates how good is for an agent to be in a given state.

Observation

10 of 94

Reinforcement learning concepts - example

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Goal: learn the optimal policy which maximizes the total expected reward over time.

State: tuple (ball_x, ball_y, cursor_h, opponent_cursor_h), for each t.

Action: up or down of 1cm. {‘up’, ‘down’}

Reward: e.g., scored points, or +1 if agent scored, -1 if opponent scored, 0 otherwise. The design of the reward function is often tricky and shall be tuned.

Optimal policy: find the best mapping between the state and the action to take. E.g., go up when the ball is coming top right.

State value function (informal): how many points am I expecting to score given that now I am in state ?

Atari’s “Pong”

11 of 94

Reinforcement learning challenges

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Often a long sequence of actions before we discover consequences of the actions.

e.g., win or lose game only after moves are complete.

Never see the result of actions not taken.
Never told what the best action was.
The outcome of our actions may be uncertain.
We may not be able to perfectly sense the state of the world.
The reward may be stochastic or delayed.
We may have no clue (model) about how the world responds to our actions.
We may have no clue (model) of how rewards are being paid off.
World may change while you try to learn it: dynamic environment.
How much time for exploration (of an uncharted territory) before exploitation of what we have learned?

Foster innovative solutions, e.g., “AlphaGo’s move 37”

12 of 94

Markov Decision Process

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

13 of 94

Maths prerequisites

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

14 of 94

Markov Decision Process (MDP)

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

The environment can be modeled as an MDP when the following are known:

Action space:
State space:
Reward space:
MDP dynamics:

The MDP/env dynamics fully describes the MDP under analysis, allowing for analytical solutions.

MDP is finite if are finite sets.

Under this formulation, we say that the agent interacts with the environment by performing some action , transitioning to a new state and receiving a scalar feedback called reward .

This results in a trajectory: �(where is the terminal state).

Markov property:

Random variables

15 of 94

Environment dynamics

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

When the environment dynamics function is known, we can compute everything else one may want to know about the environment:

State-transition probabilities

Expected rewards for state-action pairs

Expected rewards for state-action-next-state triples

16 of 94

Episodic and continuing tasks

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Continuing tasks:

The agent-environment interaction could go on forever.
There is no terminal state.
Trajectories can reach infinite length.
Examples: thermostat keeping the room temperature stable, steering the beam in the LHC.

Episodic tasks:

The agent-environment interaction is limited in time.
At some point a terminal state is reached.
Examples: board games, video games, robotic arm manipulating objects.

17 of 94

Return

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

The goal of the agent is to find the optimal policy: “what is the best action I should take in state S_t?”

To assess the “goodness” of a state (an action), the agent tries to estimate the cumulative future reward of a trajectory starting from that state (and taking that action). More formally, we call this property return, and we define it as the cumulative future reward:

is the discount factor. For continuing tasks , thus the discount factor has to be for the sum to converge.

18 of 94

Policy

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

A policy describes the behavior of the RL agent, mapping from state to probabilities of selecting each possible action.

Policy

Example:

The first step to find the optimal policy is to assess how good is the current one…

	p(a₁)	p(a₂)
State s₁	0.7	0.3
State s₂	0.1	0.9

S₁

S₂

a₁

a₂

a₁

a₂

0.7

0.3

0.1

0.9

19 of 94

Hands-on: get familiar with MDP and return

In this first exercise we will:

Learn the basics of Gymnasium, a fork of OpenAI Gym.
Play with MDP dynamics
Compute the return

20 of 94

Value Functions

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Allow to assess the “goodness” of some policy .

The state value function is the expected return when starting in state and following thereafter:

How to generalize this?

21 of 94

Value Functions

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Allow to assess the “goodness” of some policy .

The state value function is the expected return when starting in state and following thereafter:

Still not so useful… What do we do with ?

22 of 94

Value Functions

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Allow to assess the “goodness” of some policy .

The state value function is the expected return when starting in state and following thereafter:

Interesting recursive relationship to “remove” the return… thus the explicit dependency on the future.

23 of 94

Value Functions

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Allow to assess the “goodness” of some policy .

The state value function is the expected return when starting in state and following thereafter:

24 of 94

Value Functions

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Allow to assess the “goodness” of some policy .

The state value function is the expected return when starting in state and following thereafter:

When is finite, we can solve it directly as a linear system in unknowns:

Where R is the immediate expected reward and P is the state transition matrix.

However, this is expensive also for small MDPs!

25 of 94

Value Functions

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Allow to assess the “goodness” of some policy .

The state value function is the expected return when starting in state and following thereafter:

The state-action value function is the expected return when starting in state , taking action , and following thereafter:

Very similar to the state value function… but not recursive

26 of 94

Value Functions

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Allow to assess the “goodness” of some policy .

The state value function is the expected return when starting in state and following thereafter:

The state-action value function is the expected return when starting in state , taking action , and following thereafter:

27 of 94

Value Functions - dynamic programming

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Dynamic programming (DP) allows to iteratively solve Bellman equations to estimate the value function under some policy .

Reference: RL book - chapter 4

28 of 94

Bellman optimality equations

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

So far, we “evaluated” some policy by computing its associated value functions and .

How can we compute directly the optimal policy ?

The optimal policy is the one which maximizes the expected cumulative future reward in each state

The goal of the RL agent is to find the optimal policy which maximizes the�value functions.

a.k.a. value function

29 of 94

Bellman optimality equations

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

State value functions:

State-action value functions:

In both cases, we replace the expectation with a max over the action space

30 of 94

Optimal policies

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

The optimal policy is the policy that assigns non-zero probabilities only to the actions that maximize the the value function in some state, for all states.

	a1	a2	a3
s1	20	30	30.1
s2	15	15	3

Optimal state-action value function

	Pr(a1)	Pr(a2)	Pr(a3)
s1	0	0	1.0
s2	0.5	0.5	0

Optimal policy #1

	Pr(a1)	Pr(a2)	Pr(a3)
s1	0	0	1.0
s2	0.99	0.01	0

Optimal policy #2

	Pr(a1)	Pr(a2)	Pr(a3)
s1	0	0	1.0
s2	1.0	0	0

Optimal policy #3

	Pr(a1)	Pr(a2)	Pr(a3)
s1	0	0	1.0
s2	0	1.0	0

Optimal policy #4

Greedy policies

31 of 94

Optimal policies - dynamic programming

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Dynamic programming (DP) allows to iteratively solve Bellman optimality equations to estimate the value function under the optimal policy.

Reference: RL book - chapter 4

32 of 94

Hands-on: dynamic programming

In this exercise we will learn how to use dynamic programming for:

Policy evaluation, also called prediction. The DP algorithm is called “policy evaluation”.
Policy improvement, also called control. The DP algorithm is called “value iteration”.

33 of 94

MDP summary

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

In some situations, we can assume that the environment can be modeled as a Markov decision process.

The environment dynamics are fully known as a common function. In the discrete case, you can imagine p as a 4-dimensional lookup table for probabilities.

This allows us to easily compute Bellman equations and Bellman optimality equations:

These recursive equations can be solved

As system of (non)linear equations
Iteratively by means of dynamic programming (e.g., policy iteration, value iteration algorithms).

From the Bellman optimality equations, it is easy to obtain the optimal policy which maximizes future rewards.

34 of 94

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

35 of 94

Sample-based methods

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

36 of 94

Sample-based methods

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

When the environment dynamics are not known, we can simulate them by interacting directly with the environment.

Again we can have episodic and continuing tasks.

In this case, episodes are characterized by trajectories of finite length, terminated by some terminal state :

The longer we interact with the environment,
the more data we collect,
the better our estimate of the underlying dynamics will be precise…

…at the cost of taking very long time.

Sample efficiency (informally): how many interactions with the environment do we need before being able to exploit the gained knowledge for our goals?

Observation

37 of 94

Monte Carlo methods

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Base on experience: sample sequences of states, actions and rewards from actual or simulated interactions with the environment:

“Monte Carlo” replace expectation on the return with average:

is known only at the end of an episode, thus we can only apply Monte Carlo methods to episodic tasks, which terminate at some point (reach some terminal state ).

Therefore, value functions and policies are updated only at the end of each episode. Monte carlo is incremental in an episode-by-episode sense (off-line), but not in a step-by-step sense (on-line):

38 of 94

Monte Carlo prediction

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Prediction = estimating the value function

39 of 94

Monte Carlo prediction - example

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Observation

Interact with the env…

	p(a₁)	p(a₂)
State s₁	0.7	0.3
State s₂	1.0	0

S₁

S₂

a₁

a₂

a₁

a₂

0.7

0.3

1.0

…according to some policy.

Q(s,a)	a₁	a₂
State s₁	???	???
State s₂	???	???

Initial Q table:

Q(s,a)	a₁	a₂
State s₁	23.1	12.09
State s₂	2.57	???

Resulting Q table:

Upon convergence

We never visited (s₂, a₂)!

40 of 94

Exploration vs. exploitation

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

When the env dynamics are not known, we need to sample from the environment, at the risk of incurring in bias.

Exploration requires devoting some interactions budget to low-rewarding interactions, however in the long run it can result in better rewards.

Exploiting too early can lead to suboptimal policies, which are too shortsighted. They prefer small immediate rewards versus big delayed rewards.

Too few exploration in favour of exploitation may bias the agent, with the risk of locking him sub-optimal policies forever!

To find the best policy, the agent may have to explore a lot before, at a greater computational cost. Trade-off!

41 of 94

Maintaining exploration

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

In practice, a popular way to maintain exploration is resorting to -greedy policies.

Is usually small (e.g., 0.1) and < 1 .

Example, given 3 actions a₁, a₂, a₃ where A^*=a₃:

(Greedy action)

1/3

a₁

a₂

a₃

Pr(a₁) = /3

Pr(a₂) = /3

Pr(a₃) = /3 + (1 - )

Let’s visualize it with the help of a probability tree…

42 of 94

Monte Carlo control

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Control = improving the current policy

Novelty

43 of 94

Monte Carlo control - example

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Observation

Interact with the env…

	p(a₁)	p(a₂)
State s₁	/2 + 1 -	/2
State s₂	/2 + 1 -	/2

S₁

S₂

a₁

a₂

a₁

a₂

…according to some policy.

Q(s,a)	a₁	a₂
State s₁	???	???
State s₂	???	???

Initial Q table:

Estimate Q table

Loading…

44 of 94

Machine learning

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum commodo diam nec ex dignissim mollis. Nunc posuere risus eu orci ultrices, sed cursus orci malesuada. Maecenas lectus nnc, scelerisque et pharetra at, condimentum sed purus. Fusce ac massa et orci euismod auctor. Proin et hendrerit justo. Nulla facilisi. Quisque eros augue, vehicula ac cursus a, rhoncus vitae leo. In eu tortor id urna pretium feugiat.

Quisque et convallis mauris, aliquet iaculis diam. Morbi eu iaculis ipsum. Aenean justo massa, aliquam eget metus non, sodales pharetra elit. Curabitur sed varius leo. Donec volutpat purus vel molestie congue. Morbi congue commodo massa in viverra. Curabitur et hendrerit ipsum. Cras condimentum iaculis libero nec imperdiet. Integer elit nulla, mollis eget porta accumsan, semper at tellus. Duis egestas, ligula a gravida elementum, enim magna bibendum turpis, ut pretium arcu purus et urna. Pellentesque facilisis sapien eu tellus ultricies aliquet.

MC has high variance + off-line -> slow learning

45 of 94

Monte Carlo control - example

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Q(s,a)	a₁	a₂
State s₁	???	???
State s₂	???	???

Initial Q table:

Q(s,a)	a₁	a₂
State s₁	23.1	12.09
State s₂	2.57	42.5

Resulting Q table:

Upon convergence

Red: max action value

	p(a₁)	p(a₂)
State s₁	/2 + 1 -	/2
State s₂	/2 + 1 -	/2

Initial policy:

	p(a₁)	p(a₂)
State s₁	/2 + 1 -	/2
State s₂	/2	/2 + 1 -

Final policy:

Control

Red background: preferred action

46 of 94

Monte Carlo (MC) - summary

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

The return is known only at the end of an episode, thus we can only apply Monte Carlo methods to episodic tasks, which terminate at some point (reach some terminal state ).

Therefore, value functions and policies are updated only at the end of each episode. Monte carlo is incremental in an episode-by-episode sense (off-line), but not in a step-by-step sense (on-line):

Drawbacks:

Off-line method: policies and value functions can be updated only at the end of an episode -> Low sample efficiency: need many interactions with the environment to converge.
Not applicable to continuing tasks.
it is subject to relatively high variance, since it estimates the expected return as the (weighted) sum of the rewards (random variables):

End of first episode. The second starts.

47 of 94

TD(0) methods

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

How to improve sample efficiency of MC methods?

Idea: turn off-line into on-line!

Recall that the expression of the return can be rewritten recursively:

Now, at each step (interaction) estimate the return by means of bootstrapping:

The incremental update is possible as soon as are available.

Don’t need to wait for the end of the episode to update value functions and policies.

“If one had to identify one idea as central and novel to reinforcement learning, it would undoubtedly be temporal-difference (TD) learning.”

– Barto-Sutton RL book.

48 of 94

TD(0) methods

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Don’t need to wait for the end of the episode to update value functions and policies.

The on-line update rule becomes, for some small learning rate 0< < 1 :

In both cases, the new estimate of the value function is a linear combination of the previous estimate and the “TD error”.

49 of 94

TD(0) control - SARSA

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

It estimates the expected return for state-action pairs assuming the current policy continues to be followed: on-policy update for Q. �

50 of 94

TD(0) control - Q-learning

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

It estimates the expected return for state-action pairs assuming a greedy policy were followed despite the fact that it's not necessarily following a greedy policy: off-policy update for Q. �

More generally, off-policy means that the return is computed using a different policy from the one used to choose the next action (i.e., the policy through which we “explore” the environment).

51 of 94

Hands-on: TD(0)

In this exercise we will learn how to implement:

SARSA
Q-learning

for policy control.

52 of 94

TD(0) summary

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Pros (on-line method):

Improved sample efficiency: converges faster than MC.
Applicable to continuing tasks.

Cons (due to bootstrapping):

Biased estimate of the return.
Difficult to propagate sparse rewards through bootstrapped returns.
More susceptible to the violation of Markov property. Harder to reconstruct the whole interactions “storyline”.

SARSA vs. Q-learning:

53 of 94

Model-based methods

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

The interaction with the environment may be expensive:

Slow to respond (e.g., human)
Costs of operating the environment (e.g., LHC)
Env can be hardly reachable (e.g., Mars)

…how can we train our agent well, without interacting too much?

Keep a model of the environment and sample (also) from it!

General idea:

Interact with the real env:
Update Q(s,a) based on real env
Update the model
For N times:

Sample from model:
Update Q(s,a) based on model

54 of 94

Tabular Dyna-Q

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Model: store previous interactions with the environment. Can only sample state-action pairs visited previously.

55 of 94

Tabular Dyna-Q

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Model: store previous interactions with the environment. Can only sample state-action pairs visited previously.

56 of 94

Model-based methods - summary

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Pros:

Improve sample efficiency: reduce the number of costly interactions with the environment.

Cons:

The model may be wrong or outdated (non-stationary environment). How often update the model?

57 of 94

Function approximation

58 of 94

Limitations of tabular methods

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Tabular methods encountered: Monte Carlo, SARSA, Q-learning, tabular Dyna-Q.

State space and action space must be discrete: define rows and columns in the Q-table.
Q-table values are learned independently: requires many interactions and cannot infer the value from a “similar” state: there is no generalization across similar states (or state-action pair).
Don’t scale to problems with a large number (millions) of states.

How do we represent this in a table?

One entry for each unique combination of the colors of all pixels?

. . . . . . . . . . .

. . . . . . .

. . .

. . . . . . . . . . .

Can we find a better way?

Atari’s “Pong”

59 of 94

Function approximation - value function

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Function approximation shifts the task of learning the values for each state or state-action pair to learning a parameterized version of the value functions that minimizes a given objective.

The parametric state value function is with parameters .

The objective to minimize is the Mean Squared Error in the approximation of by .

where is the proportion of times state s was visited.

The objective above can be minimized by means of Stochastic Gradient Descent (SGD), obtaining the update rule:

A similar reasoning holds for the parametric state-action value function:

60 of 94

Function approximation - value function

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Since the value function is unknown, we substitute it with an unbiased estimator of it: .

Monte Carlo:
TD(0): . Is bootstrap legit???

Depends on the trainable parameters!

61 of 94

Function approximation - value function

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Since the value function is unknown, we substitute it with an unbiased estimator of it: .

Monte Carlo:
TD(0): performs a semi-gradient update (do not use the full gradient information).

Nice but… how to put everything together?

Gradient update
RL interactions
Policy update

62 of 94

Function approximation - value function

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Example of on-policy control with function approximation: SARSA.

63 of 94

Function approximation - DQN

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Train RL agent to play Atari games. Feature extractor: convolutional neural network (CNN). Reward: +1 if scored a point, -1 otherwise.

Example of off-policy control with function approximation. The Q function is approximated with a neural network

Image credits.

Experience replay

DQN

Image credits.

Gameplay

Off-policy: DQN is updated using “old” transitions sampled from the replay buffer

64 of 94

Function approximation - DQN

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Continuous state space: embedding vector produced by the feature extractor (CNN, part of DQN).

Discrete actions space: depends on the specific game.

Experience replay:

Similar idea as Dyna-Q for model-based RL.
Improves sample efficiency: DNNs are data hungry.
Decorrelates samples in the training batch. Good for convergence properties and training stability.

Q(s, a₁)

Q(s, a₂)

Q(s, a₃)

Q(s, a₄)

State features

Image credits.

65 of 94

Function approximation - DQN

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Update rule:

Sample a batch of B transitions from the replay buffer (hereafter #B=1).
Compute the loss:

Gradient update:

“Target” network

“Policy” network

66 of 94

Function approximation - DQN

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Target and policy networks visualized:

Image credits.

Copy parameters every K steps

Policy network

67 of 94

Hands-on: DQN

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

In this exercise we will learn how to implement Q-learning with function approximation.

Image credits.

68 of 94

Function approximation - policy gradient

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

So far we obtained the optimal greedy policy from the (approx) optimal value function

However, in some cases it would be more convenient to learn directly the policy!

State space may be “complex”, whereas the policy could be “easy” (e.g., always go left)
-greedy is generally suboptimal: with probability take random action.
Continuous action space (e.g., robotic arm control).
Greedy policy may be suboptimal (e.g., “rock, paper, scissor”).

A greedy policy is easily exploited by the opponent.

Example:

Always draw paper

69 of 94

Function approximation - policy gradient

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Write the policy as a parametrized function:

For instance:

Define an objective, for instance:

And the corresponding param. update rule:

Is called actions preferences. Can by a neural network.

70 of 94

Function approximation - policy gradient

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Episodic case

71 of 94

Function approximation - policy gradient

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

REINFORCE is based on MC -> high variance.

Adding a “baseline” leaves the expected value of the update unchanged, but it can have a large effect on its variance.

Image from Barto-Sutton book

For instance:

72 of 94

Function approximation - policy gradient

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

REINFORCE with baseline

Image from Barto-Sutton book

Kind of “loss”, depends on G_t

73 of 94

Function approximation - actor-critic

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

REINFORCE based on MC -> off-line.

Solution: substitute MC return with its bootstrapped version. Becomes on-line.

Keep using value function as a baseline:

Update rules become:

Value function params (critic)

Policy function params (actor)

Note: they may use different alphas.

74 of 94

Function approximation - actor-critic

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Episodic case because for the continuing case we need to change the objective function

(out of the context of this lecture)

75 of 94

RL for particle accelerators

76 of 94

AWAKE - background

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Problem: difficult to accelerate electrons in the LHC. Due to curvature, they emit radiation and lose energy. Since the circumference of the LHC is finite, it can provide a finite amount of energy to the particles. This results in a dynamic equilibrium.

A possible solution could be using linear/larger accelerators (e.g., FCC), or improving how we accelerate particles.

Wakefield Acceleration Experiment: experimental beamline exploring innovative acceleration techniques.

Use two beams in the same line:

e^-e^-e^-e^-e^-p⁺p⁺p⁺p⁺p⁺

Go through a plasma cell:

The protons ionise the plasma, creating a wakefield:�this corresponds to a very strong electric field.
The electrons “surf” the wakefield like a wave, receiving �a very strong acceleration.

Wakefield acceleration could replace current RF cavities in�the far future.

e^-

q⁺

wakefield

77 of 94

AWAKE - beam steering

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

RL problem: keep the electrons beam within the beamline and steer them into the plasma to best exploit the wakefield.

States: readouts from 10 sensors (BPMs)

Actions: 10 correctors

Simulation of the beamline available: train the agent in simulation and fine-tune it on the real machine.

…remember the expensive agent-environment interactions mentioned before?

Beam-time is expensive: the agent has to learn fast! Sample efficiency is critical.

78 of 94

AWAKE - beam steering

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Illustration of Q-learning: 1D beam steering

79 of 94

AWAKE - Q-learning with NAF

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Standard DQN only applicable to discrete action tasks
Our control tasks: typically continuous state-action spaces
Various ways to extend Q-learning to continuous action tasks

Most successful: actor-critic algorithms, e.g. DDPG and extensions
If convex problem, can use trick: assume Q-function belongs to function class that is easy to optimise, e.g., NAF (Normalized Advantage Function)

80 of 94

AWAKE - train Q-learning on simulation

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

81 of 94

AWAKE - train Q-learning on simulation

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Further experiments improved sample efficiency by using model-based RL, inspired to Dyna-Q

82 of 94

Bunch splitting

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

83 of 94

Bunch splitting - background

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

RF cavities: used to accelerated particles in the LHC

84 of 94

Bunch splitting - background

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

RF cavities: used to accelerated particles… but also to split bunches!

The split is done in PS to prepare beams for LHC. �We gradually change the intensity (voltage) of high harmonics in the RF cavities: h₇, h₁₄, h₂₁

Voltage and phase has to be dynamically adjusted for each harmonic:

Compensate for voltage and phase errors
Synchronize phase with beam

Done manually: not always reproducible.�Task: RL to optimise splittings to produce uniform bunches. Good for science!

85 of 94

Bunch splitting - background

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Animation of a bunch (triple) splitting. Note the variation of intensity per harmonics.

86 of 94

Bunch splitting - RL problem

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Automating bunch splitting is good for reproducibility. RL-based splitting is “in production” at PS.

Challenge: reward function design. For both phase and voltage, compare bunch profiles (Means Squared Error).

Phase profiles:

87 of 94

Bunch splitting - RL problem

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Automating bunch splitting is good for reproducibility. RL-based splitting is “in production” at PS.

Challenge: reward function design. For both phase and voltage, compare bunch profiles (Means Squared Error).

Similarly, for voltage profiles:

88 of 94

Bunch splitting - RL problem

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Optimization of voltage profiles is based on the assumption that the phase is already optimal.

Train two RL agents:

SAC-Phase-Sim2Real: Trained using the phase MSE loss criteria (used to define a step-wise reward) to optimise the phase only.
SAC-Volt-Sim2Real: Trained using the overall MSE loss (used to define a step-wise reward) to optimise the voltage only, assuming phase is already optimised.

89 of 94

Bunch splitting - RL problem

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Optimization of voltage profiles is based on the assumption that the phase is already optimal.

90 of 94

Bunch splitting - RL problem

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

For stability reasons, it is convenient to “bias” the agent with a prediction from a supervised CNN.

The CNN is trained on simulated data.

CNN feature extractor

Regression head

(phases)

Guess initial phase values

91 of 94

Bunch splitting - the big picture

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

92 of 94

Bunch splitting - conclusions

11 Jul 2024

Matteo Bunino | An introduction to RL and its applications at CERN

Trained on simulation and applied on machine without re-training.

Consistent good performance for:

varying intensities (1.3e11-2.6e11)
different beam types (72b, BCMS)

Consistently rivals experienced operators in optimisation steps: averaging ~8.5 steps per optimisation (depending on initial conditions).