1 of 37

Curiosity-driven Exploration�by Self-supervision Prediction�Deepak Pathak, Pukit Agrawal, Alexei Efros, Trevor Darrell�ICML 2017

Resources:�https://archive.org/details/Redwood_Center_2017_10_11_Deepak_Pathak

LLNL-PRES-705839

1

2 of 37

Motivation

LLNL-PRES-705839

2

3 of 37

Not so intelligent AI

Task-specific
Not adaptive
Expensive

Labeling Training Set

Test in Testset

Cat

LLNL-PRES-705839

3

4 of 37

Life-long learning in human

Passive

Is imitation learning the route to humanoid robot?, Schaal, 1999

Active

The Scientist in a crib, Gopnik, Metzoff &Kuhl,1999

machines

LLNL-PRES-705839

4

Human learning: life long paradigm

Active interaction: Actively interacting with environment(baby play with toy)�play with toy 🡪 try to make sense of environment 🡪 make the hypothesis 🡪 verify this hypothesis by make another action in the worlds.
Passively observing others: Imitation learning, learning by watching youtube video.

Those two parts are both important, and critical part of human learning. �Active learning help human learn how we act on environment, how to map our sensory input to control.�Passive learning is important to learn diversity (we cannot exploring everything by acting interaction with environment)
The main motivation of this paper is how can we design these life-long learning system as “We” human do but in machine. And how do we go about that.

Let’s think about applying active learning in supervised setting. Then, Its not scalable for this kind of setting. Lets think that there’s task but the task is not known to this infant. Then, how to train that particular task is we label all possible actions of that task for every possible scenarios. So, that requires exponential number of labels and we cannot collect all possible number of labels.
So, we know that life long learning cannot be designed by supervised fashion.

5 of 37

Reinforcement Learning �

Reinforcement Learning:�Input: Given environment which provides numerical reward signal, and agent which act inside of that environment �Output: Let agent learn how to take actions(policy) in order to maximize reward.

Goal: Learn how to take actions in order to maximize reward
Design RL: Objective, State, Action, Reward �

Extrinsic reward�(Extrinsic of agent )

LLNL-PRES-705839

5

6 of 37

Reinforcement Learning �

Typically, RL requires very dense reward

Mnih et al, Nature 2015

Jaderberg et al, ICLR 2017

Extrinsic reward

LLNL-PRES-705839

6

Then, what do we do for design life-long learning system?

The way to get around supervised learning problem is reinforcement learning which has active setting rather than supervised setting.

It is very popular these days.

But, it has also caveat.

[explain Atari game] what makes learning really possible is reinforcing agent to learn the policy by using score of the game as reward.

If you think about this score, it is actually really dense supervision.

It tells you every time steps, what action is right and what action is wrong.

Other case, like navigation game. Navigation game, in nature, is sparse task..

In this navigation, the goal of agent is to reach to some end location.

Even in this sparse task, most model incentivize the reward function by adding this intermediate incentive, such as apple.

So, RL requires very dense reward which is really limiting.

And, this is not how real world works.

In real world, there is no apples in every time steps.

7 of 37

Rewards are sparse in real world …

Now

In real world, reward can delivered by days, months or years!

30 years later

“curiosity”

LLNL-PRES-705839

7

8 of 37

Intrinsic Motivation/Curiosity

LLNL-PRES-705839

8

9 of 37

Model

LLNL-PRES-705839

9

10 of 37

How can we design “curiosity”

Let the agent Incentives to go to area where it didn’t visit before.
Problem: No generalization

Building model of prediction
Potentially generalize to novel scenarios

Prediction Uncertainty

Visitation Counts

LLNL-PRES-705839

10

Ok. Then, how do we design curiosity? How can we build it into the machine?

One very easy way is

encourage agent to do something new

or go some area where it didn’t visit before,

by something known as “visitation counts”

The idea is let the agent incentives to go to area where it didn’t visit before.

The issue is, the agent can go and exploring new, but boring stuff.

So, there is no generalization here. So, it is basically building a table and working on the table.

So, there is no learning or no generalization going on of task here.

The other paradigm of approach is prediction uncertainty.

So, it related to the building model of prediction.

The idea is let the agent build the model how the world works.

And agent apply this model for different parts of the scenarios, and if it incurs high error,

It will get more curious of that part of the environment. [explain by your own language!]

And it is learning paradigm, so this can be potentially generalize for novel scenarios which has not seen before.

11 of 37

On ‘Mental Model’ of the prediction …

If the organism carries a small scale model of external reality and its own possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilize the knowledge of the past in dealing with present and the future, and in every way react in much fuller, safer and more competent manner to emergencies which face it. ��- Kenneth Craik, 1943 chapter 5, page 61

LLNL-PRES-705839

11

12 of 37

Design of curiosity

Action

Observation

LLNL-PRES-705839

12

13 of 37

Design of curiosity

Action

Observation

LLNL-PRES-705839

13

14 of 37

Design of curiosity

Action

Observation

LLNL-PRES-705839

14

15 of 37

Design of curiosity

Action

Observation

LLNL-PRES-705839

15

16 of 37

Design of curiosity

Action

Observation

Do nothing

LLNL-PRES-705839

16

17 of 37

Design of curiosity

LLNL-PRES-705839

17

18 of 37

Design of curiosity

LLNL-PRES-705839

18

Let’s consider one more example.

Here, the agent is trying to say what happens when we go right.

In earlier case, if we press right, it goes right.

So, it believe same thing will happen here.

But in this case, because of the piller, the agent incurs high prediction error.

Now, if you think about this, this prediction error tells to agent there’s something wrong here which agent doesn’t know about.

So, that gives the agent more incentive to come here.

This is the crux of the idea.

The idea is, let use "this prediction error" as the curiosity.

The higher the error, the more curious the agent would be.

And over time you will learn the model of agent, then this error will drop down.

So, here, the agent will incentivize to go right and stop by the piller.

This is the high level idea.

19 of 37

Overview

Train a Model

Predict consequences of the action

Bad prediction 🡪 higher curiosity

LLNL-PRES-705839

19

20 of 37

Training RL with external reward

External Reward

Policy Network

Curiosity Reward (intrinsic)�

+

LLNL-PRES-705839

20

Now, let see how we exactly formulate the curiosity.

So, to remind you how the reinforcement learning works, how it works,

The agent take the observation and it goes to some network, and the network outputs some sequence of actions, and this action takes to the next states.

Now, in reinforcement learning, you use some external reward, such as score given from outside game emulator. So, network is trained to predict action, such that it maximize expected sum of rewards.

In this work, what is proposed is the mental model of the curiosity, which called intrinsic curiosity module. (click)

What it does is, it takes triplet of your current state, action and next state as input, And it outputs some curiosity reward. Curiosity bonus.

Now, if you look at this, even if you don’t have extrinsic reward (click), then its just self learning system.

As the agent get more and more experience,

this model overtime trained to decrease curiosity reward,

But the agent always will take the action to increase the curiosity reward.

21 of 37

Intrinsic curiosity module (ICM)

Forward�Model

Curiosity in pixel-space is hard

[Schimidhuber 2001]

LLNL-PRES-705839

21

Now, let’s just unbox the intrinsic curiosity module and see what’s inside of it.

Very naïve way to implement this is,

Start from current state and action.

The way that I generate the curiosity reward is I take those two things, and I train model to predict next state in pixel space. Very simple. Very naïve way, suppose I do this.

Then you take the error between predicted pixel level next state and real observation of next state.

And the higher the error, the more the curiosity.

Because the agent suppose to be curious which is more surprising to them. Which is more un-predictable. Ok?

So, the idea is using this error between predicted next state and real observation of next state as curiosity reward to optimize the policy.

This is not quite there yet.

Now the another question arise. What if there’s some intrinsic error of the environment in nature. What if there’s white noise in environment.

So, the issue is predicting pixel level is very hard. So, the curiosity in pixel space is very hard problem. Then, how can we tackle this issue?

22 of 37

LLNL-PRES-705839

22

23 of 37

LLNL-PRES-705839

23

24 of 37

Intrinsic curiosity module (ICM)

Encode relevant parts

That the agent can affect
That can affect the agent

Forward�Model

Curiosity in pixel-space is hard

LLNL-PRES-705839

24

25 of 37

Intrinsic curiosity module (ICM)

Forward�Model

LLNL-PRES-705839

25

Then how do we learn this feature space?

The idea is, if these features encode every thing affect to agent and agent can affect,

we should able to predict action from previous feature and next feature.

” If these features encode every thing affect to agent and agent can affect,

You can say that these feature should also be useful to predict the action back”. OK?

So, these features are learned to predict the action, which means that

these encoding has no incentive what so ever to encode anything

that is not relevant to action prediction.

And that advise you the notion of ignoring random part of environment.

Because you can not really affect them and they are not really affect you.

And, this is full things and complete picture of intrinsic curiosity module, and all parts are trained simultaneously.

The goal of ICM is minimizing this error as well as this error,

And overtime as you trains, the other network- reinforcement learning network, trys to go to areas where this curiosity reward is high.

26 of 37

Experiments

LLNL-PRES-705839

26

27 of 37

Self-supervised Curiosity

Does it work when no external reward, only curiosity?
Do learned skills generalize?
Evaluate curiosity + extrinsic reward
Is it robust?

LLNL-PRES-705839

27

28 of 37

No external reward, only curiosity

Mario:�Learning to play with no reward

At the start of training, agent repeat similar actions
After some training

Emergent behavior

Jumping enemies, pipes and pits
Killing enemies
Staying alive�:Skills learned in order to become curious (unpredictable)

Learn to cross over 30% of level 1

LLNL-PRES-705839

28

Firstly, is the model working with no external reward but only curiosity?

The answer is yes, and surprisingly well.

I will show you the demo later on, but in Mario case, the agent learns how to play with no external reward at all but only with curiosity reward.

At first of training, the agent repeat the same actions like jumping around, go right and left,

But after some training it learns skills by itself from curiosity,

Such as jumping enemies, pipes and pits, killing enemies and so on.

It is because the agent train itself those skills just to stay in curious.

So, if the Mario need to be in curios, it should be staying alive and moving forward.

So, those skills beneficial to survive can learned by itself.

As a result, Mario learn to cross over 30% of level 1. the reason why its 30% is, at 38% of time, there comes deep pit, and we need to press up key consecutively 15 times, and agent just die before that if it fails to execute to press up button 15 times. And our agent just die at this limit. So, it receive no information saying that there’s curious world beyond that.

29 of 37

No external reward, only curiosity

VizDoom (3D navigation):� Coverage during exploration

Visitation of curious (ICM) > visitation of random exploration

LLNL-PRES-705839

29

30 of 37

Do these skills generalize?�: Main problem of RL🡪 not generalizable

Trained on level 1

Tested on level 2

Trained on level 1

Tested on level 3

Curriculum learning

(teacher’s forcing)

LLNL-PRES-705839

30

31 of 37

Evaluate curiosity + extrinsic reward�

VizDoom Game�Note: Agent does not have access to Map

LLNL-PRES-705839

31

32 of 37

Evaluate curiosity + extrinsic reward�

“Sparse”

Very Sparse

LLNL-PRES-705839

32

33 of 37

Is it robust?

LLNL-PRES-705839

33

34 of 37

Is it robust?

Robust to irrelevant part of the observation.

LLNL-PRES-705839

34

35 of 37

Demo video

https://youtu.be/J3FHOyhUn3A

LLNL-PRES-705839

35

36 of 37

Summary

Learn skills with no extrinsic reward (only curiosity)
Skills generalize to novel scenarios
Effective exploration when extrinsic reward is sparse
Robust to irrelevant part of the observation.

LLNL-PRES-705839

36

37 of 37

LLNL-PRES-705839

37