1 of 31

Better, Faster & cheaper Spotinst

2 of 31

01

02

03

Hackathon Idea

Introduction

Architecture

04

Business Impact

06

Business Viability

07

08

Learnings

Future Work

05

Demo

2

ZEOTAP

2

3 of 31

01 // Hackathon Idea

ZEOTAP

3

ZEOTAP

3

4 of 31

Spotinst is taking away Spot inst(ances)!

4

ZEOTAP

4

5 of 31

Jobs are Failing

5

ZEOTAP

5

6 of 31

Developers are Exhausted!

6

ZEOTAP

6

7 of 31

Decreased Revenue

7

ZEOTAP

7

8 of 31

To Infinity and beyond!

8

ZEOTAP

8

9 of 31

02 // Zscaler is here!

Better, Faster & cheaper Spotinst

ZEOTAP

9

ZEOTAP

9

10 of 31

Problem Statement

EC2 instances available cheaper in Spot market than On Demand
Spotinst allows us to scale up EMR clusters when needed
Doesn’t work well when EC2 instances are scarce
Scales down the cluster even when jobs are running
Slow and cumbersome

10

ZEOTAP

10

11 of 31

03 // Architecture

Better, Faster & cheaper Spotinst

ZEOTAP

11

ZEOTAP

11

12 of 31

Architecture

Smart Rule Engine �(computes metrics from yarn)

Intelligent Instance Selector �(intelligently selects instance type for a given needed memory to minimize cost and increase availability)

Simple Cluster Scaler�(communicates with AWS APIs to scale up/down an EMR cluster)

Zscaler UI

(shows current state of the cluster along with cost information)

12

ZEOTAP

12

13 of 31

Smart Rule Engine

Desired Memory = Average Container Size * Pending Containers

Average Container Size = Memory Allocated / Container Allocated

ZEOTAP

13

ZEOTAP

13

14 of 31

Simple Cluster Scaler

Polls Smart Rule Engine for cluster state
Take Scale up/down decision
Ask Intelligent Instance Selector for instance type
Communicates with AWS spot apis
Publishes cluster state for reporting
Request timeouts
Handling Race conditions such as up-up, down-up, up-down
Uses AWS spot data feed api for reporting
Backend server and API to serve requests to UI

Simple Cluster Scaler�(communicates with AWS APIs to scale up/down an EMR cluster)

ZEOTAP

14

ZEOTAP

14

15 of 31

Intelligent Instance Selector

Goal - Find out best instance types and their number based on features and given constraints
Features are spot prices, interruption probability, etc
Constraint - total allocated memory equal to given memory
Data is acquired through AWS apis and Web scraping
Model is extensible to more features.
Weights are decided manually as of now. But can be tuned using supervised learning once feedback is available

ZEOTAP

15

ZEOTAP

15

16 of 31

03 // Business Impact

ZEOTAP

16

ZEOTAP

16

17 of 31

Faster Scale up

17

ZEOTAP

17

18 of 31

Better Interface

(Deals with Memory instead of nodes)

18

19 of 31

Say No to Job Failures

Never scales down a node that is running containers for any job!
No requests for Instance Types interrupted recently
Cancels requests that are not fulfilled within a time limit

19

ZEOTAP

19

20 of 31

Cost Savings

20

ZEOTAP

20

21 of 31

Proactive Alerts & Logs Visibility

21

22 of 31

Happy Engineers

22

23 of 31

05 // Demo

ZEOTAP

23

ZEOTAP

23

24 of 31

06 // Business Viability

ZEOTAP

24

ZEOTAP

24

25 of 31

Flexibility for configuring platform for variety of workloads
Generic - Can be used by any company using big data
Cost savings by ensuring availability and intelligent instance type provisioning
Open Source potential

Business Viability

ZEOTAP

25

ZEOTAP

25

26 of 31

07 // Our Learnings

ZEOTAP

26

ZEOTAP

26

27 of 31

ZEOTAP

27

ZEOTAP

27

28 of 31

07 // Future Work

ZEOTAP

28

ZEOTAP

28

29 of 31

THANK YOU

ZEOTAP

29

ZEOTAP

29

30 of 31

Spotinst vs Zscaler

}

72.60%

88.10%

15.5% more savings

30

ZEOTAP

30

31 of 31

Zscaler Features

Faster Scale up and Scale down with Exponential steps
Stable scale down of cluster achieving better jobs stability
Proprietary Machine Learning (ML) model for EC2 Instance selection
Generalized for any AWS EMR cluster, can be used across businesses
Open Source Potential

31

ZEOTAP

31