Replay Buffer with Local Forgetting for Adapting to Local Environment Changes in Deep MBRL

Ali Rahimi-Kalahroudi, Janarthanan Rajendran, Ida Momennejad, Harm van Seijen, Sarath Chandar

​

​

Adaptivity is a Key Feature of Model-Based Learning

• Van Seijen, Harm, et al. "The LoCA regret: a consistent metric to evaluate model-based behavior in reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 6562-6572.

The Local Change Adaptation (LoCA) Setup

• Van Seijen, Harm, et al. "The LoCA regret: a consistent metric to evaluate model-based behavior in reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 6562-6572.

The Local Change Adaptation (LoCA) Setup

• Van Seijen, Harm, et al. "The LoCA regret: a consistent metric to evaluate model-based behavior in reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 6562-6572.

The Local Change Adaptation (LoCA) Setup

​

• Van Seijen, Harm, et al. "The LoCA regret: a consistent metric to evaluate model-based behavior in reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 6562-6572.

Current Deep Model-Based RL Methods Are Not Adaptive!

• Wan, Yi, et al. "Towards evaluating adaptivity of model-based reinforcement learning methods." International Conference on Machine Learning. PMLR, 2022.

Replay Buffer Against Catastrophic Forgetting

Stale Data

Stale Data

Interference-Forgetting Dilemma

Interference-Forgetting Dilemma

Local Forgetting (LoFo) Replay Buffer

Local Forgetting (LoFo) Replay Buffer

Local Forgetting (LoFo) Replay Buffer

Local Forgetting (LoFo) Replay Buffer

LoFo Buffer

Experiments

Dreamer w/ the LoFo Buffer

Dreamer w/ the LoFo Buffer

Dreamer’s Reward Estimates

Dreamer’s Reward Estimates

Dreamer’s Reward Estimates

Dreamer’s Reward Estimates

Limitations and Future Work

• Demonstration of the Possibility of Local Adaptivity ✅

Limitations and Future Work

• Demonstration of the Possibility of Local Adaptivity ✅
• General Strategy - LoFo ✅
• Forgetting Samples that are Spatially Close, but Temporally Far

Limitations and Future Work

• Demonstration of the Possibility of Local Adaptivity ✅
• General Strategy - LoFo ✅
• Forgetting Samples that are Spatially Close, but Temporally Far
• More Complex Domains
• Initial Dataset with a Random Policy ❌

Limitations and Future Work

• Demonstration of the Possibility of Local Adaptivity ✅
• General Strategy - LoFo ✅
• Forgetting Samples that are Spatially Close, but Temporally Far
• More Complex Domains
• Initial Dataset with a Random Policy ❌
• Quadratic Time Complexity
• Simplest form of Nearest-Neighbours ❌

Thank you!

Contact Info: ali-rahimi.kalahroudi@mila.quebec