Deep Reinforcement Learning: An Overview!

By Akhil Devarashetti

Who am I?

Computer Vision Engineer at BioTrillion

Masters degree in Artificial Intelligence�from University of Cincinnati in 2020

​

​

​

​

Websites: akhil.ai, ml.gallery

What I’ll cover today

RL Concepts:

- Environments

- Policy Gradients

- Q Learning

- Curiosity

- Model-Based

For each RL concept:

- Some theory

- Code

- Demo

​

What is Reinforcement Learning?

Any algorithm that learns by trial and error

Random Moves

Learn

Good Moves

Win/Lose

Why Care?

Any business value? Practicality?

Sources:

https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40

https://venturebeat.com/2021/02/23/how-reinforcement-learning-chooses-the-ads-you-see/

https://www.analyticsvidhya.com/blog/2020/10/reinforcement-learning-stock-price-prediction/

https://openai.com/projects/five/

​

But why I love RL is because …

Sources:

https://github.com/openai/procgen

https://ad6dm.net/log/2018/11/the-gamification-ham-radio/

​

It’s fun!

Closest thing to Artificial General Intelligence

Sources and links:

https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html

https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark

https://deepmind.com/blog/article/muzero-mastering-go-chess-shogi-and-atari-without-rules

https://openai.com/about/

https://youtu.be/WXuK6gekU1Y

​

But why I love RL is because … It is challenging!

Sources:

https://wandb.ai/akhilez/SimpleGridWorld?workspace=user-akhilez

​

Let’s get the basics out of the way

Env (state)

Agent

Actions: up, down, left, right

Reward:

+10 if green box is reached

-10 if red box is reached

​

Next state

Objective: Maximize rewards

State Transitions

LEFT

DOWN

0

Reward

0

Reward

(s₁, a₁, r₁)

(s₂, a₂, r₂)

(s₃, a₃, r₃)

… (s_t,a_t,r_t)

Episode

Working with environments - gym.Env class

Takes an action

Returns next_state and reward

Working with environments - Demo run

​

Is this agent intelligent?

What does an agent do?

Agent chooses an action given a state

action = agent(state)

Can we replace this agent with a neural network?

Neural network as agent

Neural Network

Up

​

Down

​

Left

​

Right

Softmax

(sampled, not argmax)

(4, 4, 4)

(objects, height, width)

Neural Net Agent

Neural Network in PyTorch

Replaced random agent with neural net agent

Is this agent intelligent?

How do we TRAIN this network?

The objective is to maximize rewards. So that is what we’ll do.

Object/Loss function = Maximize rewards!��maximize(r₁ + r₂ + r₃ + r₄ + … + r_t)

But neural networks like to MINIMIZE the objective/loss

​

So let’s minimize(-1 * (r₁ + r₂ + r₃ + r₄ + … + r_t))

There’s no gradient attached!

Probabilities: p₁ p₂ p₃ … p_t

minimize(-1 * (r₁*p₁ + r₂*p₂ + r₃*p₃ + … + r_t*p_t))

​

Reward weighting with probabilities

Policy Gradients!

Code Sample

Sampling actions

Loss function

minimize(-1 * (r₁*p₁ + r₂*p₂ + r₃*p₃ + … + r_t*p_t))

​

Demo: https://wandb.ai/akhilez/rl_talk_gridworld/runs/p06l2doh?workspace=user-akhilez

Oh, and there’s credit assignment

Some games win in the last step

Ex: gridworld?

Steps leading up to the end are most significant.

Weights: … , 0.77, 0.81, 0.86, 0.9, 0.95, 1

​

​

λ = 0.95

Weights = reversed(λ, λ², λ³, λ⁴, …)

​

​

Some games lose in the last step

Ex: balancing a pole

Steps leading up to the end are least significant.

Weights: 1, 0.95, 0.9, 0.86, 0.81, 0.77, …

​

​

Objective function: minimize(-1 * (r₁*p₁*λ + r₂*p₂*λ² + r₃*p₃*λ³ + … + r_t*p_t*λ^t))

Problem with PG?

I can learn only after playing the whole episode :(

Let’s predict reward

For every action, there is an equal and op… reward

Ex: Stock investing.

Actions: [invest , don’t invest ]

Reward: [stock up , stock down ]

Let’s predict tomorrow’s reward i.e. reward for current action.

Reward Predictor

Neural Network

Up

​

Down

​

Right

​

Left

Predicted rewards

But there’s no immediate reward :(

We get a reward only after navigating and reaching the end position.

-1 reward for all actions.

So which action is the best?

Psst! taking left has a higher long-term reward.

Immediate and Long Term Rewards

return(state_t) = R_t = r_t + (𝞬 . r_t+1) + (𝞬² . r_t+2) + …

Prediction

R_t = r_t + 𝞬 . R_t+1

Return = reward … plus a tiny bit of future rewards

𝞬 = small number b/w 0 to 1

Predict

Label

Return Predictor

Q Learning!

Q_t

s_t

env.step

r_t

s_t+1

Q_t+1

R_t ← r_t + 𝞬 . R_t+1

Let’s look at some code

Random action 10% of the time

Next state’s Q value

r_t + 𝞬R_t+1

MSE loss

Demo: https://wandb.ai/akhilez/rl_talk_gridworld/runs/34sqqdx0?workspace=user-akhilez

Ways to improve

Q Learning:

• Experience replay
• Target network
• Rainbow DQN

Policy Gradients:

• Credit assignment
• Proximal Policy Optimization

Combining PG and DQN: Advantage Actor Critic

Q Learning Revolution

The famous Atari paper by DeepMind

Sources:

- https://www.nature.com/articles/nature14236

​

0%!!??

Sparse Reward Problem

• Lots of planning
• No guidance via rewards
• Random exploration will not help

Play online: https://www.retrogames.cz/play_124-Atari2600.php?language=EN

Curiosity!

Our agent is too reliant on external rewards.

​

No motivation to explore.

​

How is a human curious?

Curiosity!

• Reducing uncertainty about the world
• Intrinsic rewards from high loss
• Increase the reward frequency?

LEFT

Intrinsic Curiosity Module

MSE( , )

~s_t+1

s_t+1

Intrinsic reward

DQN with intrinsic reward

R_t ← (r_{t ex} + r_{t in} ) + 𝞬 . R_t+1

Demo: https://wandb.ai/akhilez/Mario/runs/16dbk5q3?workspace=user-akhilez

Noisy TV Problem

Random events in the environment.

Examples:

• Random blowing leaves, mist, etc
• Non-Player Characters interaction

How is this a problem?

• Every random event increases the intrinsic reward.
• The agent stares without exploration

How to solve noisy TV?

a_t

Other techniques

• Random Network Distillation
• Episodic Memory
• Agent 57

Sources:

• https://openreview.net/pdf?id=Sye57xStvB
• https://openai.com/blog/reinforcement-learning-with-prediction-based-rewards/
• https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark

Model Based

Can you ask a neural network that plays chess - What’s your plan?

Neural Networks lack planning.

Game Tree

Demo

Curse of Dimensionality

1. Too Broad
2. Too Deep

1. Too Broad

Narrow down the search space with probability distribution over valid actions.

Sources:

• https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf

2. Too Deep

After a certain depth is reached,

predict the value of that branch with value network.

Sources:

• https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf

What if we don’t have an emulator for planning?

Well, we can learn the dynamics - learned model

MuZero

h(s) → ~s

f(~s) → p, v

g(~s_t, a) → ~s_t+1, r

​

p = probability distribution over actions�v = value of the state

Source: https://deepmind.com/blog/article/muzero-mastering-go-chess-shogi-and-atari-without-rules

Other techniques

• Dreamer V2
• GameGAN

​

Other topics:

• Inverse RL
• Offline RL

Sources:

• https://arxiv.org/pdf/2010.02193.pdf
• https://nv-tlabs.github.io/gameGAN/

The End

Resources:

• Key papers: https://spinningup.openai.com/en/latest/spinningup/keypapers.html
• Discord: https://github.com/andyljones/reinforcement-learning-discord-wiki/wiki
• Debugging advice: https://andyljones.com/posts/rl-debugging.html
• Reddit: https://www.reddit.com/r/reinforcementlearning/