Part 2 - Going Beyond Behaviour Cloning with Off-Policy Reinforcement Learning

Adam Jelley

University of Edinburgh

• Going beyond Behaviour Cloning with Off-Policy RL
• Review of Off-Policy RL
• TD3+BC (A Minimalist Approach to Offline Reinforcement Learning)
• SAC-N / EDAC (Uncertainty-Based Offline RL with Diversified Q-Ensemble)

Improving on Behaviour Cloning

• What if we want to improve on the performance of the agent that collected the offline data?
• Return-conditioning (as in Decision Transformer)
• Percentage behaviour cloning (e.g. 10%-BC)
• Advantage weighting (Advantage-Weighted Regression: Simple and Scalable Off-Policy RL)

​

​

• Off-Policy Reinforcement Learning?

Review of Off-Policy RL

• Off-policy RL algorithms learn from data collected from a policy which may be different to current policy (usually stored from earlier policy in replay buffer)
• E.g. Q-learning

Source: Berkeley Deep RL Course, Lecture 8

Review of Off-Policy RL - DDPG

• Q-learning generalised to continuous actions -> Actor-Critic
• Actor learns to maximise Q function
• Critic learns Q function for actor policy
• Example of generalised policy iteration

​

• Introduced in deep RL as Deep Deterministic Policy Gradients (DDPG) in Continuous control with deep reinforcement learning (2015)

Review of Off-Policy RL - TD3

• Instability led to Twin-Delayed Deep Deterministic Policy Gradients (TD3) Addressing Function Approximation Error in Actor-Critic Methods (2018)

​

• Effectively DDPG with “tricks”:
• Clipped double Q-learning (“Twin”): Two Q networks to reduce overestimation of Q-values
• Delayed policy updates (“Delayed”): Update value functions more frequently than policy
• Target policy smoothing: Add noise to target actions to avoid local optima

Review of Off-Policy RL - SAC

• In parallel, Soft Actor-Critic (SAC) introduced as an off-policy actor-critic from maximum entropy RL framework in Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018)

​

• Adopted the TD3 “tricks” (twin networks and delayed actor updates, but no noise required as policy smoothing already captured by max entropy objective)

​

• Both SAC and TD3 are regularly used for continuous domains for their sample efficiency (due to being off-policy)

Improving on Behaviour Cloning with Off-Policy RL

• Naively applying off-policy algorithms to offline RL (treating the offline data as the replay buffer) usually leads to policy collapse
• As actor and critic optimise, go further out of distribution of training data
• No additional interactions to provide feedback on extrapolation error
• Values become noisy and policy ends up getting similar returns to random…

​

How can we handle the inevitable distribution shift when optimising return?

​

• Early approaches introduce regularisation towards given dataset:
• Policy Regularisation (regularise actions) e.g. TD3+BC
• Critic Regularisation (regularise values) e.g. CQL

A Minimalistic Approach to Offline RL - Introduction

A Minimalistic Approach to Offline RL - Theory

A Minimalistic Approach to Offline RL - Experiments

A Minimalistic Approach to Offline RL - Experiments

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Introduction

Use uncertainty not regularisation!

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Uncertainty Penalisation with Q-Ensemble

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Uncertainty Penalisation with Q-Ensemble

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Uncertainty Penalisation with Q-Ensemble

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Uncertainty Penalisation with Q-Ensemble

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Ensemble Gradient Diversification

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Ensemble Gradient Diversification

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Ensemble Gradient Diversification

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Ensemble Gradient Diversification

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Experiments

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Experiments

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Experiments

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Experiments

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Experiments

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Experiments

Uncertainty Based Offline-RL with Diversified Q-Ensemble - Conclusion

Critique and Further Research Directions

• Core idea makes a lot of sense
• Use uncertainty rather than regularisation to control distribution shift

​

• Implementation can be improved
• Building on SAC means underlying approach is well-developed
• Ensemble similarity loss term seems quite empirical - how can we get a better measure of uncertainty?
• With a more complete posterior Q distribution, how do we best utilise this uncertainty in both the actor and the critic?

​

​

• Despite shortcomings, approach works very well (stable, efficient and performant), at least on standard benchmarks

​

• Potential for this approach to be applied in the online/fine-tuning setting too, using the offline data to solve the “cold-start” problem