1 of 45

KServe Essentials: Building a Production-Ready Cloud-Native Model Serving Platform

Yuan Tang

Principal Software Engineer, Red Hat, OpenShift AI

Project Lead, Argo & Kubeflow

Co-chair, Kubernetes WG Serving

Maintainer, KServe

October 21-24, 2024

Mandalay Bay Convention Center�Las Vegas, Nevada

2 of 45

Agenda

01

02

03

04

05

06

07

Complexity of Cloud Native AI Platform

What is KServe

History

Open Source Community

Architecture

Key Features

Project Roadmap

3 of 45

The Complexity of Cloud Native AI Platform

From Cloud Native AI WG

4 of 45

What is KServe?

Highly scalable, standard, cloud agnostic model inference platform on Kubernetes

5 of 45

Why KServe?

Provides performant, standardized inference protocol across ML frameworks.

Supports modern serverless inference workload with autoscaling including Scale to Zero on GPU.

Provides high scalability, density packing and intelligent routing using ModelMesh

Simple and pluggable production serving for production ML serving including prediction, pre/post processing, monitoring and explainability.

Advanced deployments with canary rollout, experiments, ensembles and transformers.

5

#IBMTechXchange

6 of 45

History of KServe

6

From Kubeflow/KFServing to KServe

Developed collaboratively by Google, IBM, Bloomberg, NVIDIA, and Seldon in 2019 under the Kubeflow project.
The project graduated from Kubeflow and was rebranded from KFServing to standalone KServe project in Sep 2021.
KServe 0.7 released outside of the Kubeflow with migration guide for minimal disruptions in Oct 2021.
KServe was donated to LF AI & Data Foundation in Nov 2021.

#IBMTechXchange

7 of 45

History of KServe

7

Kubeflow Ecosystem

#IBMTechXchange

8 of 45

Community: Maintainers and Contributors

8

15 maintainers and 250+ contributors!

#IBMTechXchange

9 of 45

Community: Adopters

9

30+ companies varying from vendors to end users!

#IBMTechXchange

10 of 45

KServe Architecture: Single Model

10

#IBMTechXchange

11 of 45

KServe Architecture: Multi Model (ModelMesh)

11

Designed for high-scale, high-density and frequently-changing model use cases.
Intelligently loads and unloads models to and from memory to strike an intelligent trade-off between responsiveness to users and computational footprint.

#IBMTechXchange

12 of 45

KServe Feature: Serving Runtimes

12

Pluggable, reusable, extensible runtimes

Framework specific: PyTorch, TensorFlow, Sklearn, Paddle, XGBoost, LightGBM, PMML
Generic/Multi-framework support: MLServer, ONNX, Hugging Face (vLLM and Triton backends)
Custom runtime support

#IBMTechXchange

13 of 45

KServe Feature: Serving Runtimes

13

Hugging Face runtime

#IBMTechXchange

14 of 45

KServe Feature: Serving Runtimes

14

Custom runtime:

Predictor definition

#IBMTechXchange

15 of 45

KServe Feature: Serving Runtimes

15

Custom runtime:

InferenceService

#IBMTechXchange

16 of 45

KServe Feature: Pre/Post Processing

16

Transforms image via torchvision

#IBMTechXchange

17 of 45

KServe Feature: Pre/Post Processing

17

v1 REST API

#IBMTechXchange

18 of 45

KServe Feature: Pre/Post Processing

18

v2 gRPC API

#IBMTechXchange

19 of 45

KServe Feature: Inference Graph

19

TBA

#IBMTechXchange

20 of 45

KServe Feature: Model Storage

20

Supported storage formats

Azure, S3, GCS
URI
PVC (K8s Persistent Volume Claim)
OCI
Storage Containers

#IBMTechXchange

21 of 45

KServe Feature: Model Storage

21

PVC example

#IBMTechXchange

22 of 45

KServe Feature: Model Storage

22

S3 example

#IBMTechXchange

23 of 45

KServe Feature: Model Storage - ModelCars

23

Efficient model pulling from OCI image registry

Reduced Startup Times: By avoiding repetitive downloads of large models, startup delays are significantly minimized.
Lower Disk Space Usage: The feature decreases the need for duplicated local storage, conserving disk space.
Enhanced Performance: ModelCars allows for advanced techniques like prefetching images and lazy-loading, improving efficiency.

#IBMTechXchange

24 of 45

KServe Feature: Model Storage - ModelCars

24

OCI URI schema as the model reference.
Replacing the init-container with a sidecar container.
Configuring the pod with shareProcessNamespace: true to facilitate access between sidecars.
Accessing to model data in a sidecar container without copying it into a shared volume. No init-container is needed.

#IBMTechXchange

25 of 45

KServe Feature: Model Storage - ModelCars

25

Build Image with Model

Specify StorageURI

#IBMTechXchange

26 of 45

KServe Feature: Model Storage

26

Custom Storage Initializer

Custom Model Registry / Storage Containers
Problem: KServe downloads models using a built-in storage initializer and cannot support custom storage systems.
Solution: KServe provides ClusterStorageContainer CRD to allow users to define custom model registry protocol, e.g. model-registry://.

#IBMTechXchange

27 of 45

KServe Feature: Model Storage

27

ClusterStorageContainer

Custom container spec for a list of supported URI formats

#IBMTechXchange

28 of 45

KServe Feature: Model Storage

28

Custom model-registry:// protocol

#IBMTechXchange

29 of 45

KServe Feature: Request Batching

29

Requests go to the model agent container.
The batcher in sidecar container batches the requests.
The batcher sends the inference request to the predictor container.

#IBMTechXchange

30 of 45

KServe Feature: Autoscaling

30

Knative autoscaling

#IBMTechXchange

31 of 45

KServe Feature: Autoscaling

31

Send traffic in 30 seconds spurts maintaining 5 in-flight requests

hey -z 30s -c 5 …

#IBMTechXchange

32 of 45

KServe Feature: Autoscaling

32

Scale down to zero

#IBMTechXchange

33 of 45

KServe Feature: Explainer / TrustyAI

33

Pluggable explainer runtimes

“Why did my model produce this inference result?”

#IBMTechXchange

34 of 45

KServe Feature: Monitoring

34

Prometheus Metrics

Supported serving runtimes exports prometheus metrics on a specified port in the inference service's pod

#IBMTechXchange

35 of 45

KServe Feature: Monitoring

35

Prometheus Metrics��

Prometheus latency histograms are emitted for each of the steps (pre/postprocessing, explain, predict).
The latencies of each step are logged per request.
Each model server may implement its own set of metrics.
Knative/Queue proxy emits metrics be default

#IBMTechXchange

36 of 45

KServe Feature: Monitoring

36

Grafana Dashboards

Knative HTTP Dashboard
KServe Dashboards for Runtimes

ModelServer Latency Dashboard
TorchServe Latency Dashboard
Triton Latency Dashboard
…

#IBMTechXchange

37 of 45

KServe Feature: Canary Rollout

37

#IBMTechXchange

38 of 45

KServe Feature: Deployment Mode

38

Serverless vs. Raw Deployment

Serverless Mode:

Requires Knative
Uses Knative Pod Autoscaler (KPA)
Supports canary rollout

Raw Deployment Mode:

No dependency on Knative
Uses K8s Deployment, Service, HPA
Does not support Scale down and from Zero

#IBMTechXchange

39 of 45

KServe Feature: GitOps Friendly

39

ArgoCD includes custom health checks for KServe CRDs

#IBMTechXchange

40 of 45

Future of KServe

40

GenAI Inference

#IBMTechXchange

41 of 45

Future of KServe

41

Stable APIs and SDK

Release stable CRDs

InferenceService
ClusterServingRuntime, ServingRuntime
InferenceGraph

Graduate Python SDK to v1

#IBMTechXchange

42 of 45

Future of KServe

42

Security and Documentation

#IBMTechXchange

43 of 45

Future of KServe

43

Community Collaboration

Kubernetes WG Serving
Knative
Envoy
Kubeflow
vLLM
…

#IBMTechXchange

44 of 45

Other Relevant Talks Today

44

Check them out!

1:30pm Scaling ML Workflows with Kubeflow
4:30pm Accelerating Data Workflows with Project Codeflare: Bridging Kueue, Ray, and Kubeflow

#IBMTechXchange

45 of 45

Stay Connected!

Email: terrytangyuan@gmail.com

LinkedIn/X/GitHub: @TerryTangYuan

45

#IBMTechXchange