OVERVIEW
Three derivation paths
From policy improvement
to practical deep RL
AWR / PPO / TD3 + SAC
01 Reweight actions with advantages
02 Limit each sample’s policy improvement
03 Improve actions through a learned Q-function
The same flow: identity → approximation → objective → update.
DEEP RL / DERIVATIONS
01 / 35
SETUP
Notation
Keep the normalization explicit
Discounted MDP; the same initial-state distribution for both policies.
μ: reference policy π: candidate policy θ: actor parameters D: replay data
Using Δ lets every performance-difference identity retain its exact scale.
DEEP RL / DERIVATIONS
02 / 35
SETUP
The two branches
Which policy owns the advantage?
The performance-difference lemma can be written in either direction.
AWR / PPO: reference-policy advantage
TD3 / SAC: candidate-policy advantage
AWR / PPO fix the advantage; the critic branch learns the value of actions.
DEEP RL / DERIVATIONS
03 / 35
01 AWR
Step 1 · Exact identity
Start with the improvement we want
Evaluate the candidate policy using the reference policy’s advantage.
An action is attractive when it improves on μ’s expected value at that state.
The advantage is fixed; the candidate still changes both states and actions.
DEEP RL / DERIVATIONS
04 / 35
01 AWR
Step 2 · Importance sampling
Move the expectation to behavior data
Factor the state–action occupancy ratio into two terms.
State ratio: which states are visited.
Action ratio: which actions are selected at those states.
This change of measure is exact when the required support is present.
DEEP RL / DERIVATIONS
05 / 35
01 AWR
Step 3 · Local approximation
Freeze state visitation, keep action choice
Assume a small policy change makes the state distributions similar.
Keep the action ratio. Under μ, the expected advantage is zero at each state.
The return difference is now represented by a local surrogate.
DEEP RL / DERIVATIONS
06 / 35
01 AWR
Step 4 · KL regularization
Optimize improvement under a trust region
Penalize moving too far from the behavior policy.
The KL penalty makes the per-state optimization solvable in closed form.
DEEP RL / DERIVATIONS
07 / 35
01 AWR
Step 5 · Solve per state
Differentiate the constrained objective
At one state, p and m are the candidate and reference policies; A is the advantage.
Normalization determines the remaining state-dependent constant.
DEEP RL / DERIVATIONS
08 / 35
01 AWR
Step 5 · Closed-form target
Tilt the behavior policy by advantage
Actions with larger advantages receive exponentially larger relative weights.
Large β: stay close to μ. Small β: concentrate on high-advantage actions.
This is an improvement target; it is not yet a fitted neural policy.
DEEP RL / DERIVATIONS
09 / 35
01 AWR
Step 6 · Project into the actor
Fit the target with weighted regression
Project the nonparametric target into the policy family.
The projection turns policy improvement into weighted maximum likelihood.
DEEP RL / DERIVATIONS
10 / 35
01 AWR
Step 6 · Practical AWR objective
Use exponential weights in the dataset
Standard AWR omits the per-state partition function in the regression weights.
Dropping Z(s) changes the relative weight assigned to different states.
The unrestricted per-state optimum is unchanged; a shared neural actor can differ.
Treat the missing Z(s) as a practical weighting choice, not a global constant.
DEEP RL / DERIVATIONS
11 / 35
01 AWR
Step 7 · Implement the update
Fit a value baseline, then regress the actor
Use estimated returns to form fixed advantage weights.
Estimate returns → fit V → compute weights → update π by supervised regression.
AWR avoids an action-value critic; it still uses a state-value baseline.
DEEP RL / DERIVATIONS
12 / 35
02 PPO
Steps 1–2 · Exact identities
Start again with the old policy
Use μ for the old policy and hold it fixed while optimizing one collected batch.
PPO uses the same reference-advantage branch as AWR.
DEEP RL / DERIVATIONS
13 / 35
02 PPO
Step 3 · Local approximation
Use the same state-distribution surrogate
Approximate the state-distribution ratio by one; preserve the action ratio.
Repeated optimization can exploit this local surrogate far from the old policy.
PPO needs a mechanism that limits the incentive for large policy changes.
DEEP RL / DERIVATIONS
14 / 35
02 PPO
Step 4 · Trust-region motivation
From a KL constraint to a simpler objective
TRPO constrains an average KL divergence; PPO proposes simpler surrogates.
PPO-Clip takes a different route: remove the reward for changing a ratio too far.
Clipping is a surrogate-design choice; it does not follow by rearranging KL.
DEEP RL / DERIVATIONS
15 / 35
02 PPO
Step 5 · Design the clipping rule
Cap useful changes in the correct direction
The sign of the advantage determines which movement would help the sample.
Keep penalizing changes in the harmful direction; only beneficial saturation is flat.
Positive advantages cap increases; negative advantages cap decreases.
DEEP RL / DERIVATIONS
16 / 35
02 PPO
Step 5 · Combine both signs
Obtain the PPO clipped surrogate
One minimum implements both cases.
This is a pessimistic version of the sampled surrogate, not a bound on true return.
The minimum preserves the penalty for harmful probability changes.
DEEP RL / DERIVATIONS
17 / 35
02 PPO
Step 6 · Understand the gradient
The update stops rewarding saturated samples
Differentiate the ratio while the advantage estimate remains fixed.
At the old policy, PPO begins with the ordinary policy-gradient direction.
DEEP RL / DERIVATIONS
18 / 35
02 PPO
Step 7 · Estimate advantages
Build advantages from a state-value critic
A common choice is generalized advantage estimation (GAE).
Use the rollout value estimates to compute fixed advantages and return targets.
PPO usually learns V(s); it does not require differentiating a Q(s,a) critic.
DEEP RL / DERIVATIONS
19 / 35
02 PPO
Step 7 · Implement the update
Optimize a batch, then collect fresh data
Minimize the actor loss together with a value loss and optional entropy bonus.
Collect with the old policy → estimate advantages → optimize for several epochs.
Keep old probabilities fixed; then refresh the old policy and collect a new batch.
Clipping controls incentives; an optional KL early stop monitors actual drift.
DEEP RL / DERIVATIONS
20 / 35
03 TD3 / SAC
Step 1 · Reverse the identity
Put behavior visitation outside the advantage
Swap the two policies in the performance-difference lemma.
The state distribution is now fixed, but the value function depends on π.
DEEP RL / DERIVATIONS
21 / 35
03 TD3 / SAC
Step 1 · Expand the value
Expose an action-improvement objective
Write the state value as the expected action-value under the candidate policy.
The first term changes the selected actions.
Both terms also change through the true Qπ.
We cannot discard the second term until the critic is held fixed.
DEEP RL / DERIVATIONS
22 / 35
03 TD3 / SAC
Step 2 · Freeze the learned critic
Separate evaluation from improvement
Evaluate the current policy with Q; hold the critic fixed during actor improvement.
The behavior-action term is now constant in θ. Replay supplies the states.
This is approximate policy improvement, not an exact arbitrary-replay gradient.
DEEP RL / DERIVATIONS
23 / 35
03 TD3
Step 3 · Deterministic actor
Move actions uphill through the critic
Let the policy select a single continuous action at each state.
The actor proposes an action; the critic supplies its local improvement direction.
The chain rule replaces a sampled-action log-likelihood gradient.
DEEP RL / DERIVATIONS
24 / 35
03 TD3
Step 4 · Critic evaluation
Train Q with the Bellman equation
Recorded transitions evaluate the current deterministic policy’s continuation.
Bootstrapped targets need stabilization when the same critic also guides the actor.
TD3 adds three mechanisms to reduce critic error and unstable policy updates.
DEEP RL / DERIVATIONS
25 / 35
03 TD3
Step 4 · Stabilized target
Smooth the next action and take the lower Q
Use target actor and critic networks, denoted by bars.
Target smoothing reduces sharp peaks; the minimum reduces overestimation.
DEEP RL / DERIVATIONS
26 / 35
03 TD3
Step 4 · Practical update
Delay the actor while the critics catch up
TD3 = twin critics + target policy smoothing + delayed policy updates.
Update both critics frequently; update the actor and slow targets less frequently.
TD3 uses the first critic for the actor and the smaller target value for bootstrapping.
DEEP RL / DERIVATIONS
27 / 35
03 SAC
Step 5 · Change the objective
Reward return and policy entropy
SAC uses a stochastic actor and a maximum-entropy return.
α controls the trade-off between reward and entropy.
DEEP RL / DERIVATIONS
28 / 35
03 SAC
Step 5 · Use the soft identity
Carry entropy through the reversed branch
The return difference now includes the entropy difference between policies.
Freezing the soft critic yields action improvement plus entropy regularization.
DEEP RL / DERIVATIONS
29 / 35
03 SAC
Step 6 · Solve the per-state problem
The optimal soft policy is Boltzmann
For a fixed critic, entropy regularization gives a normalized exponential policy.
The additive log-partition term is constant during the actor optimization.
DEEP RL / DERIVATIONS
30 / 35
03 SAC
Step 6 · Reparameterize the actor
Differentiate through a sampled action
Write the action as a differentiable transform of fixed-distribution noise.
Freeze critic weights, but retain the gradient through the action and log density.
The actor follows high Q while retaining entropy.
DEEP RL / DERIVATIONS
31 / 35
03 SAC
Step 7 · Soft Bellman target
Bootstrap with value minus log probability
Sample the next action from the current actor and evaluate target critics.
Modern SAC uses target critics and the current actor; no target actor is required.
DEEP RL / DERIVATIONS
32 / 35
03 TD3 / SAC
Step 7 · Why replay works here
Sample new actions at replay states
The actor and critic use different parts of each recorded transition.
Actor: replay states + actions from the current actor.
Critic: recorded (state, action, reward, next state) transitions.
No action importance ratio is needed for these actor objectives.
Replay still changes state weighting and coverage.
Off-policy learning does not make the replay actor gradient exact for J.
DEEP RL / DERIVATIONS
33 / 35
COMPARISON
The resulting actor updates
Three routes from the same starting point
The branch choice determines what is fixed and how the policy is improved.
AWR reweights actions; PPO limits incentives; TD3 / SAC improve through Q.
DEEP RL / DERIVATIONS
34 / 35
REFERENCES
Primary sources
Papers behind the derivations
Detailed qualifications and source URLs are included in the slide notes.
AWR — Peng et al. (2019) · arXiv:1910.00177
PPO — Schulman et al. (2017) · arXiv:1707.06347
TRPO — Schulman et al. (2015) · arXiv:1502.05477
DPG — Silver et al. (2014) · ICML / PMLR 32
TD3 — Fujimoto et al. (2018) · arXiv:1802.09477
SAC — Haarnoja et al. (2018) · arXiv:1812.05905
Notation is unified across methods to match the flow of the original keynote.
DEEP RL / DERIVATIONS
35 / 35