1 of 26

PyCon 2019

Scraping a Million Pokemon Battles

Distributed Systems By Example

Author: Duy Nguyen (pugs-rule)

2 of 26

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

“Dweeeeeeeeeeeeeee”

3 of 26

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Introduction

Currently at Google in Core Systems

Particularly care about gender equality and mentorship in tech

Previously at Ellevation Education then left to wander the world

4 of 26

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Introduction

Currently at Google in Core Systems

Particularly care about gender equality and mentorship in tech

Previously at Ellevation Education then left to wander the world

5 of 26

lvh - Distributed Systems 101 - PyCon 2015

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Objective and Scope

I love Pokemon...for the most part.

Distributed Systems “102”

We’ll journey together through this passion project of mine and draw on specific examples to better understand getting started working with distributed systems or microservice architectures in the cloud.

Scalability and 3 “Pillars”

Concurrency of Resources
Asserting for Correctness
Resilience against Failures

Correctness

Resilience

Concurrency

[3 minute(s) | running total = 0:05] Objective

democratize information

I love Pokemon and I don't mean Pokemon go in this case I'm referring to the handheld games.
a site where you can play against other people online.
the one that I don't love is that the community is not terribly welcoming towards beginners.
more experienced players will go through Great Lengths to hide their strategies.
this was not okay with me and then I discovered the battles were not authenticated. formulaic mega charizard X.
so in my Defiance I sign up for free to your account with AWS and I started scraping millions of these battles.

so we're going to journey through this passion project of mine and draw on specific examples to understand getting started working with distributed systems or microservice architecture in the cloud.
distributed system 102
exceeding technical
continue discussion tailored so that it's grounded and practical.
i want to give you frameworks and assets for breaking down this problem space
I want these takeaways to be applicable today
working on some new design oh I've seen this pattern before

so this is how I break down this problem I think about oftentimes a major reason to pursue distributed systems
scale a resource on the scale people or machines
however this comes this comes at a non trivial consequences I’m gonna call out two

in my two things I'll call out holistically the difficulty of reasoning about the system as a whole and secondly you will have to you will have to come to grips the fact that your system will fail
so with that framework in mind that I really want to we get today we're going to be thinking about in exchange for this scalability or concurrency of resources
what's scaffolding

assert the correctness of my system in other words how do I test my distributed system
in addition how do i mitigate failure or how do I impart resilience into my system

concurrency of resource; testing for correctness; and resilience against failures

6 of 26

Pokemon Showdown

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Architecture Overview

Correctness

Resilience

Concurrency

7 of 26

(4) Lambda Function watches the S3 Bucket for new objects

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Architecture Overview

Correctness

Resilience

Concurrency

(5) New objects are indexed in ElastiCache (Redis)

(3) Battle logs are stored in S3

(1) Room List Watcher pushes new URLs onto the SQS

(2) Download Bots pull new URLs from SQS and scrapes the battle logs

8 of 26

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Concurrency

“Our application cannot handle increases in traffic.”

1 urls = room_list_watcher.scrape() 2 3 for url in urls: 4 5 download_bot.scrape(url=url)

[10 minute(s) | running total = 0:18] Concurrency

unmanageable

earlier mentioned that when a structure this talk into three pillars understanding how distributed systems provide us concurrency of resources and then figuring out and then figuring out how we can compensate for testing for correctness and resilience against failures I got to play with the sexiest part of distributed systems

pseudo code for reference

claire scale maintain and throughput or people and compute

fits in my head
rate of work in terms of magnitude

concurrency analyze characteristics of work specifically magnitude/responsibilities

partition or seams

raymond on lock vs. queue

i/o-bound

but what if want to be stateless drop thread with queue

partition helps so SQS

model of computation with message passing

go and channel same design

9 of 26

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Concurrency - System Characteristics

1 urls = room_list_watcher.scrape() 2 3 for url in urls: 4 5 download_bot.scrape(url=url)

Business Logic

State

10 of 26

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Concurrency - System Characteristics

1 urls = room_list_watcher.scrape() 2 3 for url in urls: 4 5 download_bot.scrape(url=url)

Business Logic

State

t ~= 300 milliseconds

11 of 26

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Concurrency - System Characteristics

1 urls = room_list_watcher.scrape() 2 3 for url in urls: 4 5 download_bot.scrape(url=url)

Business Logic

State

t ~= 300 milliseconds

2 seconds <= t <= 45 minutes

12 of 26

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Concurrency - Partitioning

1 urls = room_list_watcher.scrape() 2 3 for url in urls: 4 5 download_bot.scrape(url=url)

Business Logic

State

t ~= 300 milliseconds

2 seconds <= t <= 45 minutes

13 of 26

Raymond Hettinger - Thinking about Concurrency - PyCon Russia 2016

“Unless you're implementing an operating system,

use higher-level primitives such as atomic message queues.”

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Concurrency - Model of Computation (Python)

1 def produce(): 2 while True: 3 urls = room_list_watcher.scrape() 4 for url in urls: 5 url_queue.put(url)

1 url_queue = Queue.Queue()

1 def consume(): 2 while True: 3 url = url_queue.get() 4 download_bot.scrape(url=url)

14 of 26

github.com/boto/boto3

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Concurrency - Model of Computation (AWS)

1 def produce(): 2 while True: 3 urls = room_list_watcher.scrape() 4 for url in urls: 5 url_queue.put(url)

1 def consume(): 2 while True: 3 url = url_queue.get() 4 download_bot.scrape(url=url)

1 url_queue = SQSQueue( 2 name=self._properties['queue']['name'])

15 of 26

Rob Pike - “Concurrency Is Not Parallelism” - Heroku

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Concurrency - Model of Computation (Go)

1 urlQueue := make(chan string)

1 go func() { 2 for { 3 urls := roomListWatcher.Scrape() 4 for _, url := range urls { 5 urlQueue <- url 6 } 7 } 8 }

1 go func() { 2 for { 3 url := <-urlQueue 4 downloadBot.Scrape(url) 5 } 6 }

“Concurrency is the composition of independently executing components.”

16 of 26

Augie Fackler and Nathaniel Manista - Stop Mocking, Start Testing - PyCon 2012

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Correctness

“Our application is more difficult to reason about.”

New Problems

Loss of determinism
Long startup times
Increased flakiness

Glossary

Term	Definition
Double	A generic term for any object that replaces/stand in for a production object during testing.
Mock	Has expectations about the calls or interactions.
Fake	Has working implementations but take some shortcut (e.g. InMemoryDatabase).

17 of 26

Google - Software Development Coding and Practices Guide - Mock Objects

Mike Krieger - Scaling Instagram - Airbnb

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Correctness - Testing External Dependencies

Implementation	Requirements	Trade-Offs

Pokemon

Google

Community

18 of 26

Google - Software Development Coding and Practices Guide - Mock Objects

Mike Krieger - Scaling Instagram - Airbnb

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Correctness - Testing External Dependencies

Implementation	Requirements	Trade-Offs
Fake	Somewhat closer to production environment	Somewhat more engineering time for development and maintenance

Pokemon

Google

Community

19 of 26

Google - Software Development Coding and Practices Guide - Mock Objects

Mike Krieger - Scaling Instagram - Airbnb

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Correctness - Testing External Dependencies

Implementation	Requirements	Trade-Offs
Fake	Somewhat closer to production environment	Somewhat more engineering time for development and maintenance
Real	Closest to production environment Least engineering time for development and maintenance	Longest test execution time Most computationally expensive

Pokemon

Google

Community

20 of 26

Google - Software Development Coding and Practices Guide - Mock Objects

Mike Krieger - Scaling Instagram - Airbnb

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Correctness - Testing External Dependencies

Implementation	Requirements	Trade-Offs
Fake	Somewhat closer to production environment	Somewhat more engineering time for development and maintenance
Real	Closest to production environment Least engineering time for development and maintenance	Longest test execution time Most computationally expensive
Mock	The original Instagram engineering philosophy is “Do the simple thing first.”	Furthest from production environment Most susceptible to false positives (interface drift)

Pokemon

Google

Community

21 of 26

github.com/App-vNext/Polly

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Resilience

“Our application will inevitably fail.”

Policy	Premise	AKA
Timeout	Beyond a certain wait, a success result is unlikely.	"Don't wait forever"
Retry	Many faults are transient and may self-correct after a short delay.	"Maybe it's just a blip"
Fallback	Things will still fail - plan what you will do when that happens.	"Degrade gracefully"

22 of 26

github.com/SeleniumHQ/selenium

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Resilience - Timeout

1 def find_button(self, locator): 2 condition = expected_conditions.element_to_be_clickable( 3 locator=locator) 4 try: 5 button = self._wait_context.until(condition) 6 except selenium.common.exceptions.TimeoutException: 7 button = None 8 9 result = lookup.results.Find(value=button, zero_value=None) 10 return result

23 of 26

github.com/rholder/guava-retrying

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Resilience - Retry

1 def build_policy(...): 2 stop_strategy = retry.stop_strategies.AfterDuration( 3 maximum_duration=self._properties['policy']['stop_strategy']['maximum_duration']) 4 wait_strategy = retry.wait_strategies.Fixed( 5 wait_time=self._properties['policy']['wait_strategy']['wait_time']) 6 7 retry_policy = retry.PolicyBuilder() \ 8 .with_stop_strategy(stop_strategy) \ 9 .with_wait_strategy(wait_strategy) \ 10 .continue_on_exception(automation.exceptions.ConnectionLost) \ 11 .continue_on_exception(automation.exceptions.WebDriverError) \ 12 .continue_on_exception(exceptions.BattleNotCompleted) \ 13 .build()

24 of 26

Amandine Lee - Passing Exceptions 101: Paradigms in Error Handling - PyCon 2017

Correctness

Resilience

Concurrency

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

Resilience - Fallback

1 def scrape(self, url): 2 elements = None 3 while elements is None: 4 try: 5 elements = self._policy.execute(self._scraper.scrape, url=url) 6 except retry.exceptions.MaximumRetry as e: 7 # The expected errors have persisted. Defer to the 8 # fallback. 9 elements = list() 10 except selenium.common.exceptions.StaleElementReferenceException as e: 11 # An expected error has occurred that cannot be handled 12 # by alternative measures. Reload the existing scraper. 13 self._reload() 16 17 return elements

25 of 26

Jack Diederich - “Stop Writing Classes” - PyCon 201 2

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

“We don’t ship code; we ship features.”

“We don’t solve problems for computers; we solve problems for people.”

[2 minute(s) | running total = 0:30] Resilience

community; positive

I hope I know it's apparent than I'm reasonably approachable person so if you see me in the hallway and seme male or female if you just want to shoot me an email please encourage you to just grab me if you see me in the hallway or shoot me an email or she may need or shoot me an email or shoot me an e-mail what does that mean you want to talk about with that but you want to talk about whether that be you want to talk about getting into Google weather or do you want to talk about interviewing for Google interviewing for Google or distributed systems distribute systems word Hawaii orc to diving I ate some pie. Deliberately did not leave time for questions but I died but I do want to wrap up but I do want to wrap up with bye-bye taking a ride I do want to take advantage I do want to take advantage of this platform and in this conclusion I owe you first don't have time I unfortunately don't have time to build this up so I had to come out quite strong estate of gender equality in the state of gender equality in Tech is unacceptable unacceptable we can do so much better than this we can do so much better than this and I I I wholeheartedly believe and I wholeheartedly believe that this is that advanced disc cause Advance is the cause not just for females or female Engineers for females but for all people in pack for all people in Tech I'm I remember back to that that very famous talk by Jack Dietrich the very famous python talk by Jack Dietrich where he deposits that where you remind us that we're not shipping code not shipping code no customer cares about the customer care is that we just sent them another thousand lines of JavaScript thinking about the features we should code not features in the same way and the same way and same way I encourage you to remind I encourage you to to frame it as to frame it as this is not this is not a pipe we're not trying to we're not trying to deliver value we're not trying to write code we're not Engineers for computers or Engineers were not trying to solve problems for computers which one to solve problems for humans or humans for People for People for People principal I cannot believe that and I believe that sometimes and I believe that it can be very easy I can be very and I can be very easy and I can be very easy and it can be very easy to lose sight of that so I thought I heard you. That's why I urge you just to keep in the back of your mind just keep in the back of your mind this value is value this value that that would work that we write that way can we write code for computers go outside so I told you to keep in the back of your mind I keep in the back of your subconscious that reminder that you're solving problems not for computers we don't solve problems for computers you solve problems for people. Thanks for your time gang. I'll see you around.

26 of 26

bit.ly/pycon-poke

Duy Nguyen - Scaping a Million Pokemon Battles (bit.ly/pycon-poke)

HAH, you think there’s still time for questions?