The Machine Learning Toolkit for Kubernetes

Krishna Durai, Cisco Systems, Bangalore

Impact of Machine Learning

2

Makes it Easy for Everyone to Develop, Deploy and Manage a Portable, Distributed and Scalable ML system on Kubernetes

Agenda

4

• ML Lifecycle
• Kubeflow - Architecture and Features
• Demo
• Alternatives to Kubeflow and Comparison
• Kubeflow Community

ML Lifecycle: Perception vs. Reality

5

Source: KubeCon 2018 (Budapest) Talk: Building ML Products with Kubeflow

Building ML Products

6

Building�a�Model

Logging

Source: Kubeflow Contributor Summit 2019 talk: Kubeflow and ML Landscape

Makes it Easy for Everyone to Develop, Deploy and Manage a Portable, Distributed and Scalable ML system on Kubernetes

Kubeflow runs on Kubernetes

8

General Applications

Infrastructure

(Cloud/On-Prem)

Container Runtimes

ML Workloads

(Modelling, training, roll-out, serving, ...)

Infrastructure

(Cloud/On-Prem)

Container Runtimes

Kubeflow Current Features

9

Building�a�Model

Logging

Source: Kubeflow Contributor Summit 2019 talk: Kubeflow and ML Landscape

Kubeflow Architecture

Make it easy to deploy and administer a platform

• Leverage Kubeflow native & non Kubeflow components

Tie it together using

• Orchestration
• Metadata

​

10

Libraries and CLIs

​

Higher Level Services

​

Low Level APIs / Services

Arena

kfctl

kubectl

Katib

Pipelines

Notebooks

Fairing

TFJob

PyTorchJob

Jupyter CR

Seldon CR

Kubebench

Metadata

Orchestration

Pipelines CR

Argo

Study Job

MPIJob

Spark Job

Model DB

TFX

Developed By Kubeflow

Developed Outside Kubeflow

Adapted from Kubeflow Contributor Summit 2019 talk: Kubeflow and ML Landscape (Not all components are shown)

Job Operators (TF/PyTorch/MPI/...)

Provides a set of K8s Custom Resources that turns distributed concepts in ML frameworks into K8s resources

Makes it easy to configure and run local/distributed training jobs of various ML frameworks on K8s

11

TFJob Operator

Notebooks (and beyond)

Orchestration for Notebooks

• Create and manage multiple Notebook servers in one place

Integration with advanced DSLs

• Kubeflow Fairing: build, train and serve, all from Notebooks
• Kubeflow Pipeline SDK: create and deploy workflows from Notebooks

12

Pipelines

Combine individual tasks into end-to-end workflows

Provide orchestration and service integration

Enable components & sharing

Help with job tracking, experimentation, monitoring

13

Katib - AutoML and Hyperparameter Tuning

Provides a service that automates optimization of model hyperparameters and neural network architectures

Makes it easy to track, view and manage experiment results for hyperparameter candidates

Growing suggestion algorithms

• Grid search
• Bayesian Optimization
• Reinforcement Learning
• ...

14

Zhou, Jinan, et al. "Katib: A Distributed General AutoML Platform on Kubernetes." {USENIX} OpML 19. 2019

Kubebench

A harness for benchmarking ML workloads on Kubernetes

Pipelines and automates common benchmarking tasks (configuration, deployment, result collection, ...)

Evaluate performance of ML models as well as the system stack

Enable deployment of state-of-the-art benchmarking workloads on K8s

15

TensorFlow MNIST Handwritten Digits Detection with Hyperparameter Tuning

Demo

16

Available at: https://github.com/CiscoAI/KFLab/tree/master/pipelines/tf-mnist

Use Case: Conversational Chatbot

Goal: Train, evaluate, and test an AI-powered chatbot for suggesting appointments

The Rasa Stack is a set of open source machine learning tools for developers to create contextual AI assistants and chatbots:

• NLU model understands the user’s message (intents, entities, etc.) based on your training data.
• Core model decides what happens next in this conversation.
• It’s machine learning-based dialogue management predicts the next best action based on the input from NLU, the conversation history, and the training data.

​

​

​

17

rasa.com

Use Case: Initial Solution

Rasa Model Training Pipeline

TensorFlow Serving

Trains Rasa NLU and Rasa Core

models

TF trained models’ metadata

Apache Beam Data Processing

Rasa JSON input format

Training Data

Prepares Training Data for Rasa

Deployed on AWS EC2 machine

Use Case: Kubeflow Solution

• Define a Kubeflow Pipeline using the Pipelines Python SDK�
• Compile and upload the pipeline package to Kubeflow Pipelines dashboard�
• Configure and run the pipeline to train and evaluate the model end-to-end�
• Run the trained model with a simple deployed application

​

19

deploy-model

Comparing Kubeflow with Similar Frameworks

Cloud Platform Integrations

21

Community

22

What’s Next?

23

• Jupyter Notebook
• Profiles
• Isolation
• Katib: Neural Architecture Search
• Fairing
• On-premise usage
• Experiment Tracking with Metadata Schema
• Model Management

Agenda

24

• ML Lifecycle
• Kubeflow - Architecture and Features
• Demo
• Alternatives to Kubeflow and Comparison
• Kubeflow Community

Makes it Easy for Everyone to Develop, Deploy and Manage a Portable, Distributed and Scalable ML system on Kubernetes

Action Items

26

• Visit www.kubeflow.org
• Visit https://github.com/CiscoAI/KFLab/tree/master/pipelines/tf-mnist
• Contribute to community

​