1 of 61

Reinforcement Learning�

Prof. André E. Lazzaretti

lazzaretti@utfpr.edu.br

2 of 61

Introduction (Markov Decision Process)

3 of 61

Introduction

4 of 61

Topics

Definitions: action, state, return, discount factor, policy
Goal of reinforcement learning
State-action value function (Q-function)

RL -> learn the Q-function

Bellman equation
Lunar Lander
Deep Reinforcement Learning
ε-greedy policy
Code example

5 of 61

6 of 61

7 of 61

8 of 61

9 of 61

10 of 61

11 of 61

12 of 61

13 of 61

Topics

Definitions: action, state, return, discount factor, policy
Goal of reinforcement learning
State-action value function (Q-function)

RL -> learn the Q-function

Bellman equation
Lunar Lander
Deep Reinforcement Learning
ε-greedy policy
Code example

14 of 61

15 of 61

Topics

Definitions: action, state, return, discount factor, policy
Goal of reinforcement learning
State-action value function (Q-function)

RL -> learn the Q-function

Bellman equation
Lunar Lander
Deep Reinforcement Learning
ε-greedy policy
Code example

16 of 61

17 of 61

18 of 61

19 of 61

20 of 61

21 of 61

22 of 61

23 of 61

24 of 61

Topics

Definitions: action, state, return, discount factor, policy
Goal of reinforcement learning
State-action value function (Q-function)

RL -> learn the Q-function

Bellman equation
Lunar Lander
Deep Reinforcement Learning
ε-greedy policy
Code example

25 of 61

26 of 61

27 of 61

28 of 61

29 of 61

30 of 61

Topics

Definitions: action, state, return, discount factor, policy
Goal of reinforcement learning
State-action value function (Q-function)

RL -> learn the Q-function

Bellman equation
Lunar Lander
Deep Reinforcement Learning
ε-greedy policy
Code example

31 of 61

32 of 61

33 of 61

34 of 61

35 of 61

Topics

Definitions: action, state, return, discount factor, policy
Goal of reinforcement learning
State-action value function (Q-function)

RL -> learn the Q-function

Bellman equation
Lunar Lander
Deep Reinforcement Learning
ε-greedy policy
Code example

36 of 61

→ during test stage

37 of 61

→ during test stage

38 of 61

39 of 61

40 of 61

41 of 61

42 of 61

43 of 61

44 of 61

45 of 61

46 of 61

Topics

Definitions: action, state, return, discount factor, policy
Goal of reinforcement learning
State-action value function (Q-function)

RL -> learn the Q-function

Bellman equation
Lunar Lander
Deep Reinforcement Learning
ε-greedy policy
Code example

47 of 61

48 of 61

Example

49 of 61

Topics

Definitions: action, state, return, discount factor, policy
Goal of reinforcement learning
State-action value function (Q-function)

RL -> learn the Q-function

Bellman equation
Lunar Lander
Deep Reinforcement Learning
ε-greedy policy
Code example

50 of 61

Reinforcement learning example

Codes.

51 of 61

Proximal policy optimization (PPO)

https://gymnasium.farama.org/environments/toy_text/cliff_walking/

slippery

Each time step incurs -1 reward, unless the player stepped into the cliff, which incurs -100 reward.

https://youtu.be/sskW45rYFg4?si=g_OJTlKn25Ze8LA0&t=5118

52 of 61

Actor-Critic Idea

53 of 61

Step 1: Init Actor Network

54 of 61

Step 2: Init Critic Network

55 of 61

Step 3: Collect trajectories

56 of 61

Step 4: Rewards to go

57 of 61

Step 5: Calculate the advantages

58 of 61

Step 6: Optimize the Critic Network

59 of 61

Step 7: Optimize the Actor Network

If the ratio (new network is a bit different from the old network), go ahead and make the slight change

Don't make change (clipping!)

60 of 61

Step 8 and 9

Step 8:

Repeat steps 6 and 7 again, for n times!

Step 9:

Repeat 3 to 8 again, for T times!

61 of 61

Proximal policy optimization (PPO)

https://www.youtube.com/watch?v=5VHLd9eCZ-w&list=WL&index=12&t=1520s&ab_channel=JohnnyCode

https://arxiv.org/pdf/1707.06347