1 of 70

Machine Learning as Code: A Year of Democratizing ML with Kubernetes and Kubeflow

David Aronchick - Co-founder, Kubeflow�@aronchick

Jason “Jay” Smith - Customer Engineer, Google�@thejaysmith

2 of 70

One Year Ago...

2

3 of 70

What is Machine Learning?

3

4 of 70

Machine Learning is a way of solving problems without explicitly knowing how to create the solution.

4

5 of 70

Google DC Ops

5

6 of 70

6

PUE == Power Usage Effectiveness

7 of 70

But ML is hard!

7

8 of 70

Containers & Kubernetes

8

9 of 70

Cloud Native Apps

10 of 70

Cloud Native ML?

11 of 70

Platform

11

Building a model

12 of 70

Platform

12

Building�a model

Data ingestion

Data analysis

Data transformation

Data validation

Data splitting

Trainer

Model�validation

Training�at scale

Logging

Roll-out

Serving

Monitoring

13 of 70

User Experience

13

Deploy�Kubeflow

Build Docker Image

Training�at scale

Operate

Build Model Server

Deploy�Model

Integrate Model into App

Experiment in Jupyter

14 of 70

14

Model

UX

Tooling

Framework

Storage

Runtime

Drivers

OS

Accelerator

HW

Experimentation

15 of 70

15

Experimentation

16 of 70

Multi-Cloud is the Reality

16

17 of 70

And Not Just One Cloud!

17

18 of 70

18

Experimentation

19 of 70

19

Training

Experimentation

20 of 70

20

Cloud

Training

Experimentation

21 of 70

Kubecon 2017

21

22 of 70

Make it Easy for Everyone

to Develop, Deploy and Manage Portable, Distributed ML�on Kubernetes

22

23 of 70

23

Cloud

Training

Experimentation

24 of 70

Cloud Native ML!

25 of 70

Momentum!

1900+ commits
100+ Community contributors
30+ Companies contributing, including:

26 of 70

Community Contributions

NOT�GOOGLE

GOOGLE

Kubernetes

Kubeflow

NOT�GOOGLE

GOOGLE

27 of 70

Community Contribution Katib from NTT

Pluggable microservice architecture for HP tuning

Different optimization algorithms
Different frameworks

StudyJob (K8s CRD)

Hides complexity from user
No code needed to do HP tuning

27

28 of 70

Community Contribution TensorRT from NVidia

Production datacenter inferencing server
Maximize real-time inference performance of GPUs

Multiple models per GPU per node
Supports heterogeneous GPUs & multi GPU nodes

Integrates with orchestration systems and auto scalers via latency and health metrics

28

29 of 70

Community Contribution Argo from Intuit

Argo CRD for workflows
Argo CRD is engine for Pipelines (more on that later)
Argo CD for GitOps

29

HP Tuning is critical to training high quality models.

With Katib, we are creating a K8s native implementation of a hyperparameter tuning system

* Implemented as a set of containerized microservices

* Depend on K8s to deploy and manage the microservices

* Using K8s custom controllers to implement the control loop

The StudyJob controller makes it easy for users to declaratively specify the optimization space, the algorithm they want to use, and how they want to train their model.

* Users can use any K8s resource (including Kubeflow ML specific resources like TFJob or PyTorch job) to train their model.

Algorithms are implemented as microservices, so users can select one of the canned algorithms or implement their own.

Finally, the user needs a database to keep track of models and other data needed for the search.

The end result is a system consisting of multiple components that need to be deployed and wired together. Making this easy for user depends on the work Kubeflow is doing to create a simple story for deploying and managing the platform.

30 of 70

Community Contribution

Jupyter Spawner

Simplifies starting a new notebook with all dependencies on KF
Contributions by Arrikto, Red Hat and Intel

Seldon

Rich serving solution for multiple model types
Both commercial and OSS offering

Kubebench

Run benchmark jobs on Kubeflow with various system and model settings
Leverages TFJobs & Argo
Major contributions from Cisco, others

30

Kubebench

Spawner

31 of 70

Introducing Kubeflow 0.4

31

32 of 70

(almost) Introducing Kubeflow 0.4

32

33 of 70

What’s new in 0.4?

Deploy

Application CRD
Simplified Setup

Develop

Kubeflow Pipelines
TFJob/PyTorch beta

34 of 70

Click to Deploy

34

35 of 70

Click to Deploy

Problem: It’s too hard to install Kubeflow!
Solution: A one-click installation tool, available via a clean web interface
How:

Click to deploy uses a bootstrap container and kfctl.sh with all the necessary dependencies included
Also enables use of declarative infrastructure deployment (e.g. Deployment Manager on GCP)
NO TEMPLATING TOOL NEEDED

36 of 70

Demo

37 of 70

Kubeflow GitOps

37

38 of 70

GitOps

Problem: Maintaining a cluster application is hard
Solution: Implement a GitOps (coined by WeaveWorks) driven solution to manage the infrastructure and cluster code
How:

ArgoCD runs the GitOps
Synchronize Kubeflow deployment with Git repository
https://www.kubeflow.org/docs/guides/gitops-for-kubeflow/

39 of 70

Demo

40 of 70

Kubeflow Pipelines

40

41 of 70

Pipelines

Problem: ML solutions are often multi-stage
Solution: Microservices platform designed to enable reusable components and workflow orchestration
How:

Kubeflow Pipelines = a Python SDK for describing and containerizing ML tasks
Runs on Argo (already in the box) and offers experiment logging and analytics
Containerized steps lets you extend to your needs

42 of 70

Demo

43 of 70

Auto-scaling

43

44 of 70