1 of 14

Lecture 9.� Actor-Critic Design Decisions��

Sookyung Kim�

1

2 of 14

Taxonomy of RL algorithm

Actor-Critic

2

3 of 14

Value Function Fitting

3

4 of 14

From Evaluation to Actor Critic

4

5 of 14

Actor-critic algorithm (with discount)

5

6 of 14

Actor-critic Design Decisions

6

7 of 14

Architecture Design

7

8 of 14

Online Actor-critic in practice

8

[Batch size]�It learns one sample at a time: Too much variance�This is trained fully online: it learns one sample at a time (action 🡪 get a transition : sars 🡪 update value function & policy.

NN: One sample train will not work = Too much variance

Let’s use Batch (Parallel workers)

Instead of using just one simulator, we use multiple simulator, sample multiple transition. (but using different random seed)

Collect all data (transitin) 🡪 update V and Pi (Synchronously). And repeat…�This will give you a batch size equal to the number of worker threads. �

If we want to make it more faster, Use asynchronous parlallell actor-critic. �여러개의 worker thread 가 동시에 트렌지션 데이터를 만들고, 32개가 모이면 벨류펑션(크리틱)과 폴리시(엑터)를 업데이트 함. ��—> update가 각 다른 스피드의 폴리시에 대해 이루어짐. �

9 of 14

Can we remove the on-policy assumption entirely?

9

10 of 14

Let’s see what that looks like

10

11 of 14

Let’s see what that looks like

11

12 of 14

Fixing the policy update

Remember

12

중요:�Fix the problem of step 3. : Change V🡪 Q: In Q, there is no assumption that action a_t actually came from your policy. 🡪 Q is valid for any action!

Problem. Now I learn Q_hat, but target is still the function of V_hat?�Remember: The value function is expected value of Q function when the expectation is taken under your policy pi.

: V = 엑션이 폴리시를 따를때 Q의 기대값임.

따라서 타겟 식에서 V🡪 Q바꾸고, 대신 엑션을 리플레이 버퍼에서 샘플된 엑션이 아닌, 현제 폴리시를 따르는 엑션을 넣어주면 됨.

우리는 폴리시 넷트워크 (엑터)가 있으으로, 스테이트를 넣으면 엑션을 예측할 수 있음. �“ Trick: We have functional access to our policy, so we can ask our policy what it would have done if it had found itself in the old state S_i’, even though it actually never happened. So then we get this action a_i’ and we plug it into Q value, and that gets you the target value representing the value of the latest policy at this old state si’”

비록 우리가 리플레이 버퍼에서 샘플링한 스테이트가 지금 현제에 샘플한게 아니어도, 폴리시 펑션에 의해, 예전 스테에트가 주어졌을때 액션을 뭘 할 지를 예측할수 있음.

따라서 비록 우리가 실제로 에피소드를 플레이할때, 예전 스테이트 S’ 에서 현제폴리시에 따른 a’을 실행한 적이 없지만은,

그 엑션을 큐 펑션에 넣어가지고, 예전스테이트 S’이 현제 폴리시를 따를때의 타겟 벨류를 계산해 낼수 있음..엄청 쿨함!!

13 of 14

Fixing the policy update

13

14 of 14

Some implementation details

DDPG in next chapter

14