1 of 15

Distributed Hyperparameter Search on Ray

Richard Liaw*, Eric Liang*, Robert Nishihara, Philipp Moritz, Joseph Gonzalez, Ion Stoica

ray.readthedocs.io/en/latest/tune.html

2 of 15

What is Tune?

Distributed System (Ray)

Distributed Training

Data Processing

Streaming

Distributed RL

Model Serving

Hyperparameter Search

Tasks

Dynamic Task Graphs

Ray Libraries

Actors

Ray API

Ray backend

3 of 15

What is Tune?

Distributed System (Ray)

Distributed Training

Data Processing

Streaming

Distributed RL

Model Serving

Hyperparameter Search

Tasks

Dynamic Task Graphs

Ray Libraries

Actors

Ray API

Ray backend

4 of 15

Hyperparameters?

4

5 of 15

5

6 of 15

Are hyperparameters actually that important?

7 of 15

Why a framework for tuning hyperparameters?

Model training is time-consuming

Resources are expensive

We want the best model

So why do we need a framework?

Without an automated technology like Tune, you need to make manual adjustments to the hyperparameters over the course of many training runs to arrive at the optimal values. Hyperparameter tuning algorithms makes the process of determining the best hyperparameter settings easier and less tedious.

On the other hand, compute resources are expensive, and model training is becoming more and more computationally intensive. Since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.5 month-doubling time.

Finally, the time it takes for a model to train to state of the art can take days if not weeks to finish.

Reasonably, we want a system that can maximize our model performance, efficiently utilize our resources with minimal time and price.

8 of 15

Tune is built with Deep Learning as a priority.

Resource Aware Scheduling

Seamless

Distributed Execution

Simple API for

new algorithms

Framework Agnostic

ray.readthedocs.io/en/latest/tune.html

Tune is one of the only hyperparameter tuning frameworks built with Deep Learning as a priority. This means that Tune has:

The ability to multiplex training on multiple GPUs across multiple nodes. For example, using Ray as a backend, you can do highly efficient training in certain situations, such as sharing one GPU across 4 models when the model isn’t large.
Seamless distributed execution. Tune works on your single machine, parallelizing the search across cores. But also, you’re given the ability to scale from a single machine to a cluster without changing a single line of your code.
Tune offers many algorithms for helping guide your search, but if not satisfied with our offering, Tune offers a simple API for adding new model search algorithms.
Best of all, Tune is compatible with any deep learning and machine learning framework.

9 of 15

Tune is simple to use.

ray.readthedocs.io/en/latest/tune.html

Tune is easy to use. Unlike many other hyperparameter search libraries, you don’t need to restructure your entire codebase to use it.

You can use one of two interfaces, a function-based API for quick prototyping or a class-based API, which enables features such as training checkpointing. In my experience, modifying a codebase to use Tune typically takes less than 10 lines of code.

Now, let’s talk about what Tune offers for hyperparameter optimization.

Tune offers “search algorithms”, which are algorithm-based techniques. These are what you would traditionally find for “hyperparameter optimization”, and include things such as HyperOpt and random search. Most hyperparameter optimization algorithms or services can be adapted as a Tune search algorithm in just a few lines of code.

In yellow, Tune offers “trial schedulers” which are execution-based techniques for hyperparameter optimization. These techniques allow Tune to be very resource efficient. For example, HyperBand is an efficient algorithm that kills low-performing trials. Ray’s actor API is a great fit for these algorithms because it enables features such as modifications to the model during training and early termination.

Now, here’s one thing that will be hard to find without Tune: you can actually combine execution-based techniques (yellow) such as HyperBand with algorithm-based techniques (blue) such as Bayesian Optimization. Recent research has shown that certain combinations can accelerate your search process.

However, often times these two families of algorithms are implemented as stand-alone libraries and take extra engineering effort to make sure both are compatible with each other. With Tune, combining algorithms is just an extra 2 lines of Python.