1 of 10

Kubeflow Lightning Talk:

Katib & Hyperparameter Tuning

Richard Liu

ricliu@google.com

​

2 of 10

Why Katib?

  • Hyperparameter optimization can significantly improve your model performance
  • … but only if done correctly, which is hard
    • Common problems in HP optimization
      • Overfitting
      • Wrong metrics
      • Too few hyperparameters
  • Introducing Katib: a fully open source, Kubernetes-native hyperparameter tuning service
    • Inspired by Google Vizier
    • Framework agnostic
    • Extensibile algorithms
    • Simple integration with other Kubeflow components

3 of 10

Define/Parameterize the Model (1)

  • User starts with an existing training model in their own editor

​

  • Ex: Ames housing value prediction example using XGBoost

​

​

​

4 of 10

Define/Parameterize the Model (2)

​

  • User selects which hyperparameters to tune:
    • Number of boost trees in the model
    • Learning rate

​

  • Parameterize these as inputs to the model executable

​

5 of 10

Define/Parameterize the Model (3)

  • User determines how to evaluate the model

​

  • Output the evaluation metric in the training code
    • Format: metric-name=metric-value

​

  • Build the model as a Docker image

Ex

6 of 10

Configure/Run a HP Tuning Experiment

User configures:

​

  • Feasible hyperparameter ranges
  • Objective metrics
  • Algorithm configurations
  • Trial templates

​

7 of 10

Visualize and Compare Model Performance

Visualize how models perform using different hyperparameters:

  • Sort trials by accuracy or hyperparameter values
  • Observe trends in the performance:
    • Learning_rate: ~0.10
    • Estimators: ~8000
  • Save the model to DB for reuse
  • Export results to CSV

​

8 of 10

What’s Next in 2019

  • Coming soon: Katib V1alpha2 APIs
    • Streamlining APIs
    • Standardizing the vocabulary
    • Enhancing accessibility and debugging experience
    • More kubernetes-native
  • Integration with metadata and artifacts store
  • Integration with notebooks
  • Neural architecture search

9 of 10

How to Contribute?

  • GitHub: https://github.com/kubeflow/katib
    • Feedback and feature requests
    • “Help Wanted” features
    • New algorithms
    • Infrastructure and testing improvements
  • Slack: https://kubeflow.slack.com/messages/CE0BURK1B/
    • Katib community meetings

​

10 of 10

Thank you for your contributions

​

...and many others

​

​

Ce Gao

Johnu George

Toshihiro Iwamoto

Alexandra Johnson

Hougang Liu

Yuji Oshima

John Platt

Andrey Velichkevich

Constantinos Venetsanopoulos

Jinan Zhou