SpotWeb: Running Latency-sensitive Distributed Web Services on Transient Cloud Servers

Ahmed Ali-Eldin, Jonathan Westin, Bin Wang, Prateek Sharma*, Prashant Shenoy UMass Amherst, *Indiana University

Introduction

Data Center

Request Server

Provide Server

(fixed rate)

On-demand Server

Introduction

Data Center

Request Server

Provide Server

(fixed rate)

On-demand Server

Introduction

Data Center

Request Server

Provide Server

(fixed rate)

Transient Server

On-demand Server

Transient Server

• It is 70-90% cheaper than On-demand Server.
• It can be unilaterally interrupted by the cloud provider.

​

​

Latency-sensitive Applications

• Web services
• Interactive data analytics

​

• Have service level objectives (SLOs)
• Interrupt can result in downtimes and SLO violations

​

​

Spotweb’s Objective

• Run latency-sensitive distributed applications on transient server
• provide SLO guarantees

Spotweb’s Approach

SpotWeb System Architecture

• Transiency-aware Load-balancer
• SpotWeb’s system monitoring
• SpotWeb’s predictors
• MPO Optimizer

SpotWeb System Architecture

MPO Optimizer

SpotWeb System Architecture

MPO Optimizer

SpotWeb Load-Balancer

SpotWeb Load-Balancer

• Transiency-aware
• WRR algorithm

Transiency-aware Load-balancer

• The overall system utilization is low or medium, and the load can be migrated to the other running instances with no degradation in the SLOs
• The system utilization is high, and new instances can be started within the warning period, with the load migrated to the newly started instances
• the system utilization is high, and new instances can not be started within the warning period. Load will be migrated to the other running instances, or dropped until the new instances are available

SpotWeb Load-Balancer

SpotWeb System Architecture

MPO Optimizer

SpotWeb System Architecture

SpotWeb System Architecture

SpotWeb System Architecture

• Predict future information after period H

MPO Optimizer

SpotWeb System Architecture

MPO Optimizer

SpotWeb System Architecture

MPO Optimizer

SpotWeb System Architecture

MPO Optimizer

SpotWeb System Architecture

MPO Optimizer

Server Portfolio?

• Based on the financial portfolio�(comprising of various asset classes such as stocks, bonds)

Portfolio A

stock A

stock B

The Key Insight

​

Choosing an uncorrelated mix of assets can insulate the overall portfolio

​

if stock A experiences large declines,

it cannot affect stock B

Server Portfolio?

Portfolio

A x 2

1. → 25 req/s

Portfolio

A x 3

6 c/hour

Server Portfolio?

Portfolio

A x 2

2. → 115 req/s

Portfolio

A x 2

B x 1

19 c/hour

Optimization Method - Single Point portfolio Optimization (SPO)

• To select a portfolio of financial assets in order to maximize risk-adjusted returns

Optimization Method - Single Point portfolio Optimization (SPO)

Portfolio A

High Expected Returns

​

High Risk

​

Cost : 50$

Portfolio B

Low Expected Returns

​

Low Risk

​

Cost : 50$

• Not willing to take risks

​

• Optimized portfolio : B

Optimization Method - Single Point portfolio Optimization (SPO)

Portfolio A

High Expected Returns

​

High Risk

​

Cost : 50$

Portfolio B

Low Expected Returns

​

Low Risk

​

Cost : 50$

• Willing to take risks

​

• Optimized portfolio : A

Optimization Method - Multi-period Portfolio Optimization (MPO)

• Designed to avoid the main drawbacks with SPO
• Use future predictions of the market dynamics to select a portfolio

Optimization Method - Multi-period Portfolio Optimization (MPO)

t

t

• SPO

• MPO

H

Optimization Method - Multi-period Portfolio Optimization (MPO)

• Designed to avoid the main drawbacks with SPO
• Use future predictions of the market dynamics to select a portfolio

• If one asset class is overvalued and expected to decline in the future,�it is better not to have that portfolio

Optimization Method - in the SpotWeb

• Modified MPO

Optimization Method - in the SpotWeb

• Modified MPO

= (fraction of predicted workload) x (cost of service per request)

​

= (predicted cost of service per ‘total request’)

Optimization Method - in the SpotWeb

• Modified MPO

= Risk cost for penalty in case of interrupt

Optimization Method - in the SpotWeb

• Modified MPO

= Risk cost of server-to-server impact in portfolio

Experimental Evaluation

1. featureless workload
2. workload having spike

Transiency-aware load balancing Efficacy

• Experiment environment
• run 6 machine on Amazon’s US-east-1
• m4.xlarge x 2
• m4.2xlarge x 2
• m4.4xlarge x 2
• 70~95% average utilization
• average load of 600 requests/second

Transiency-aware load balancing Efficacy

• Latency starts to increase after three minute
• Latency is normalized after the new machine starts
• the cache warm-up period for one of the servers to be between 30 to 90 seconds

​

Transiency-aware load balancing Efficacy

• The average response time is wells below 200 ms during normal operation
• SpotWeb managed to bring down the 90%lie response time to less than 700ms with no request dropped

​

Intelligent over-provisioning

• sample workload

Intelligent over-provisioning

• sample workload

Benefits of Price-awareness

• r5d.24xlarge : 1920 req/s
• r5.4xlarge : 320 req/s
• r4.4xlarge : 320 req/s

Benefits of Price-awareness

Exploiting Future Workload Knowledge

Impact of Prediction Accuracy

• As the prediction accuracy decreases,�the savings decrease
• To give some perspective, SpotWeb’s predictor has a 3-5% prediction error

Scalability of SpotWeb

• SpotWeb’s algorithm is highly scalable
• SpotWeb scales sub-linearly with increasing the number of markets, or increasing the look-ahead horizon
• SpotWeb’s ability is consider large numbers of markets compared to other current approaches

Discussion

• the longer prediction horizon can improve the performance of SpotWeb
• the results hold grue for other cloud providers as SpotWeb is not dependant on anything particular to AWS

Conclusion

• Benefits of using the SpotWeb
• cost savings of up to 90% compared to conventional on-demand cloud servers
• SLO-aware server provisioning and auto scaling for web clusters

Q&A