Scalable Hyper-parameter Optimization using RAPIDS and AWS

AKSHIT ARORA

@_AkshitArora

SRISHTI YADAV

@_srishtiyadav

SPEAKER: AKSHIT ARORA

• Deep learning solutions architect at NVIDIA
• Optimize and deploy deep learning and machine learning pipelines in production
• Previously, built deep learning models for tasks in domains such as – Education, Virtual Reality, Natural Language Processing and Weather Prediction

🔗 aroraakshit.github.io

\Uhk_Shith Aurora\

2

SPEAKER: SRISHTI YADAV

• Deep learning researcher at Simon Fraser University, Canada
• Research, Create and Optimize deep learning and machine learning algorithm
• Also, building deep-learning algorithm for sensors at International Space Station (Intern: Urthecast)

🔗 srishti.dev

3

3

WHY THIS TALK?

4

WE ALL LOVE TO SPEED UP THINGS

BUT ARE WE 🏃‍♀️🏃‍♂️ THINGS FASTER ?

5

COMPANIES LOVE TO STORE AND USE DATA

BUT ARE THEY REALLY CASHING IT ?

6

WE KNOW CLOUDS ARE A THING

BUT WE DON’T KNOW IF IT’S WITHIN REACH 🤣

7

HOW ARE WE HELPING?

8

⚡🏃‍♀️🏃‍♂️ : Papermill

💰💰💰 : GPU

☁️🖥️☁️ 🖥️ : AWS

​

9

IT CAN BE AN OCEAN OF DETAILS

BUT BEFORE WE IN

10

Hyper-Parameter Optimization (HPO)

For 5 parameters, each with 4 desired values, we will have 4⁵ possible combinations

• Combinations can increase exponentially. And your training time with it. 😯
• Manually trying each of them 🤮
• Solution: Automating Hyper-parameter optimization (HPO) 🎉

Source: Hyperparameters Optimization | Towards Data Science | by Pier Paolo Ippolito

11

WHAT ARE WE GOING TO DO?

​

12

CPU Based Scalability

Learn to parameterize notebooks from a set of parameters

You don’t want to do all of them manually! 🤦

​

Automate parallel computation of parameterized notebooks.

​

Since we use CPU here, you can play with it at home too, without any cost.

​

​

​

​

13

Papermill lets you:

• Parameterize notebooks
• Execute notebooks

Helpful for running HPO tasks where the same model needs to be trained again and again with different sets of parameters.

(Docs: nteract/papermill: 📚 Parameterize and execute notebooks)

14

DEMO TIME

​

https://github.com/copperwiring/scalable-hpo-pybay/tree/master/papermill_demo

15

GPU Based Scalability

Learn to speed up the computation using GPU

​

Make efficient use of GPU using RAPIDS

​

Scale up the computation and have better metric visualization using AWS

​

Learn to do hyper parameter optimization to find best parameters from given set

You don’t want to iterate over all of them manually, trust us!

16

• Amazon SageMaker is a fully managed machine learning service
• Allows the users to build, train, analyze and deploy models in production ready environment

(Docs: https://docs.aws.amazon.com/sagemaker/)

17

• A collection of open-source libraries for end-to-end GPU Accelerated Data Science.
• Involves little to no code change

(Docs: https://rapids.ai/hpo)

18

• A collection of open-source libraries for end-to-end GPU Accelerated Data Science.
• Involves little to no code change

(Docs: https://rapids.ai/hpo)

19

DEMO TIME

​

https://github.com/rapidsai/cloud-ml-examples/tree/master/aws

20

Thank you!

Please find our code, slides and other resources at our github page:

https://github.com/copperwiring/scalable-hpo-pybay

21