1 of 14

Lecture 8.� Actor-Critic Algorithm�

Sookyung Kim�

1

2 of 14

Recap: Policy Gradient

This is reward from time-step t to end �because of the causality

2

3 of 14

Taxonomy of RL algorithm

Actor-Critic

3

4 of 14

Improving the policy gradient: �Lowering Variance

t

4

5 of 14

Baseline Trick:�Lowering Variance

b

=

Advantage function

5

6 of 14

State & State-action Value Function

6

In reality, we don’t have correct value of A, but we need to estimate. (NN)

종종 Advantage를 NN으로 approximate하면 biased 될수있는데, 그리도 variance가 적어져서, bias-variance trade off를 함.

Bias VS Variance Trade off:

Before (PG): MC estimate: having one sample, summate over the trajectory and substracting with the baseline.

Cons: treat one sample by one sample🡪 high variance, �Pro: The baseline does not change the expected value of the gradient. �

AC: Q-V. averaged quantity: Lower Variance. But high bias. I will explain why later on!��Why is it biased? The bias arises from the bootstrapping process, where the value function (critic) is often an approximation. If the value function is not perfectly accurate, the gradient estimate is biased. This means that the error in the value function propagates into the policy update, leading to biased updates. Furthermore, since Q-value or advantage estimates are used to update the policy, any errors or imperfections in these estimates introduce bias.��In flowchart, we have much more elaborated greenbox, Fit wt Q,V and A.

7 of 14

Value Function Fitting

7

8 of 14

Policy Evaluation

8

9 of 14

Monte Carlo evaluation with function approximation

9

10 of 14

Can we do better?

Bootstrapping

Might incorrect

Low variance (its not from single sample, but output from NN), �high bias (NN estimation can be bias)

10

11 of 14

Policy Evaluation Example

11

12 of 14

From Evaluation to Actor Critic

12

13 of 14

Putting Discount Factors

13

14 of 14

Actor-critic algorithm (with discount)

14