1 of 14

Lecture 14๏ฟฝContinuous Q learning ๏ฟฝ๏ฟฝ

Sookyung Kim๏ฟฝ

1

2 of 14

RECAP: Q-learning

Transition, and aโ€™ doesnโ€™t have to be ๏ฟฝsampled from current policy:๏ฟฝoff-policy RL

2

3 of 14

Recap: Problems of online Q-learning algorithm

(1) Correlated Samples

(2) Target changes as Q changes

3

4 of 14

Recap: Correlated samples in online Q-learning - Experience replay buffer

4

5 of 14

Recap - Target Changes as Q Changes: ๏ฟฝTarget Network

​

  • Fixed Target Network
    • The error is unstable because the target function, which varies with the network parameters, is included in the error calculation.
    • To enhance the stability of the training process, the target is updated only once every 1,000 steps

Target (Bellman)

Prediction

5

6 of 14

Recap - Target Changes as Q Changes: ๏ฟฝTarget Network

6

7 of 14

Q-Learning with Continuous Actions

7

8 of 14

Whatโ€™s the problem with continuous action?

8

9 of 14

Optimization after๏ฟฝ discretization of continuous action

9

10 of 14

Use function class that is easy to optimize

10

11 of 14

Learn an approximate maximizer

11

12 of 14

Learn an approximate maximizer

~ NFQCA, TD3, SAC

12

13 of 14

Summary of DDPG

13

14 of 14

Summary of DDPG

14