1 of 13

​Team 08: Grow your data science & AI skills by participating this Kaggle competition

Fast or Slow? Predict AI Model Runtime

Prof. Dr. Marius Lindauer, Dr. Alexander Tornede, Sarah Segel, Tim Ruhkopf, Difan Deng

1

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

2 of 13

Motivation

  • Runtime of deep learning models on TPUs is greatly influenced by the network graph compiler’s settings
    • Similar to optimization of code performed by compilers (e.g. Java)
  • Finding a good compiler configuration can save a lot of runtime
    • Less energy
    • Better environmental footprint
  • Google posed a corresponding Kaggle challenge that we will tackle within the next days!

2

Source: https://www.kaggle.com/competitions/predict-ai-model-runtime

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

3 of 13

Agenda For Our Challenge

  1. Day 1 (Today):
    1. Welcome (Marius)
    2. Introduction to SMAC (Marius)
    3. SMAC Notebook (Alex)
    4. SMAC Hands-On (You)
    5. Introduction to the Kaggle Challenge (Sarah)
  2. Day 2 (Tomorrow)
    • Working on the Challenge!

3

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

4 of 13

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

by Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, �Tim Ruhkopf, René Sass & Frank Hutter

​

New Team Members: Alexander Tornede, Helena Graf, � Sarah Segel, Tanja Tornede, Edward Bergmann

4

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

5 of 13

Hyperparameter Optimization

5

target algorithm A

optimizer

Goal: Find the best performing configuration:

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

6 of 13

Modular Design

6

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

7 of 13

SMAC for Black-Box Functions

7

* full facade name in SMAC3 v2: BlackBoxFacade

Image credit Derek Bingham

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

8 of 13

SMAC for CASH and Structured �Hyperparameter Optimization

8

* full facade name in SMAC3 v2: HyperparameterOptimizationFacade

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

9 of 13

SMAC for Expensive Tasks and �Automated Deep Learning

9

* full facade name in SMAC3 v2: MultiFidelityFacade

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

10 of 13

SMAC for Algorithm Configuration

10

* full facade name in SMAC3 v2: AlgorithmConfigurationFacade

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

11 of 13

Comparison to Other Packages

11

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

12 of 13

Exemplary Results

12

NetLetter (6D)

NBHPONaval (9D)

Nas1Shot1_2

Take-Aways:

  1. SMAC with a RF as black-box HPO approach “SMAC (RF)” outperforms �other approaches with TPE and GP models
  2. SMAC’s implementation of BOHB [Falkner et al. 2018] “SMAC-HB” �(also using a RF as surrogate) has a very strong any-time performance

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime

13 of 13

Find Us

13

Funded by:

Original authors:

New team members:

M. Lindauer et al.: Team 08: Fast or Slow? Predict AI Model Runtime