Self-Play in

Multi Snakes Environment

Jeewoo Kim

Mentor: Shounan An

Yum!

Intro

About Me - Jeewoo Kim

Education

1st year Computing AI (UG) at Imperial College London

Interests

Reinforcement Learning, Music Generation

​

Motivation

RL Newbie, but want to do multi agents!

Try simple multi agents environment

Apply well-known algorithms into simple env

Do lots of experiments to understand what’s � happening in the small grid world

​

​

Motivation

Requests for Research 2.0 (https://blog.openai.com/requests-for-research-2/)

Slitherin: Multi Snakes Environment

1. Set up a reasonably large field with multiple snakes
2. Solve the environment using self-play with some RL algorithms, �and overcome self-play instability.

Why Snake Game?

Simple in the beginning, but more complicated� and diverse by adding more rules to the env!

Best multi agents env to do experiments

Hands-on experiments in this camp!

What are pros and cons of self-play?

What is self-play instability?

Goals of this Project

1. Understand RL algorithms (DQN, PPO)
2. Implement them in TF
3. Solve single snake environment using DQN, PPO
4. Solve multiple snakes environment using self-play
5. Get an intuition on the behaviors of agents from analyses

Theory

Reinforcement Learning

​

​

Policy Gradients

• Compute an estimate of the gradient of policy parameters�in order to maximise the expected return using SGD
• But needs to set up right step size (1)
• Perform one gradient update per sampled trajectory and high sample complexity (2)

​

PPO - Solve both (1), (2)

• Alternates between sampling multiple trajectories from the policy and performing several epochs of SGD on the sampled dataset to optimise this surrogate objective

​

Self-Play

Use a single network to train both sides of game

(AlphaGo - One network plays both black and white)

Tends to easily overfits to its fighting opponent

​

1. Randomly samples from older version of opponents

or

2. Generate opponents from an ensemble of policies (Competitive Self-Play)

​

​

​

Development

Algorithms Implementation

DQN

Implemented DQN based on the paper and tutorials

Too SLOW to train agent in my environment! (10 steps / s)

​

OpenAI Baselines

Decided to use baselines code from OpenAI

Was much faster than my code, especially PPO

​

Algorithms Implementation

DQN - Single Snake

Didn’t converge. Reward: -1 ~ 0 after 5 million steps

Problems

• Sparse reward (20 X 20 grid, 1 fruit)
• No visual information about the boundaries

Solution

• Change environment size into 10 X 10
• Added white wall visual information on the boundaries

Self-Play Set up

Based on Competitive Self-play by OpenAI

​

Self-Play for n > 2

For n(agents) > 2, How should I update opponent networks?

​

​

Results

Single Snake using PPO

Reward over timesteps (10 X 10)

Two Snakes using Self-Play

Self-Play method (10 X 10)

​

Two Snakes using Self-Play

Self-Play method

​

Fight against past models -> generalised

Past opponents

Past opponents

Past opponents

Past opponents

Append

Past opponents

Past opponents

Past opponents

Randomly select

Single Snake using PPO

Reward over timesteps (19 X 19)

Two Snakes using Self-Play

Self-Play method (19 X 19)

​

Two Snakes using Self-Play

Intuitions from the Observation

1. Can see they try to avoid each other (focus more on surviving)

2. Surviving Longer > Eat Fruits with risk

3. Learns ‘if opponent eats fruits, �fruits will be generated somewhere�else, so change the direction’

4. Agents trained on bigger environment are more unstable (wobbling)

Comparison

​

Vanilla PPO didn’t learn because of sparse reward (there’s only 1 reward out of 361 grid space)

Can infer that self-play performs better exploration

Three Snakes using Self-Play

Used the first method

​

Extended: Adversarial Environment

Long-term reward > short-term reward

Wanted to see ‘they fight each other after growing into certain lengths’

Error in the code (Fruits keep generating in the same positions)

Conclusions

1. Competitive self-play might be a good way to make agents to explore.
2. Agents might be overfitted to certain strategies -> Experiment with agents with different seeds
3. Agent trained on bigger environment performs unstable (random) moves.

​

Future Plans

1. Generalise codes for ‘n’ snakes
2. Improve my environment codes (Change code structure)
3. Try the other update method for n > 2 snakes and compare
4. More evaluation on agent trained with self-play to observe instability
5. Try to solve self-play instability using ensembles or another network ...etc
6. More scientific analysis
7. Write blogpost

​

Acknowledgements

Thanks to my mentor Shounan An�and other mentors Sourabh and Eric, �and all the camp organisers and sponsors

​

To participants, thank you so much for�good memories! Best memory in my life

​

Thank you!

https://github.com/jdubkim/dlcampjeju2018

​

​

jeewoo1998@gmail.com

References