1 of 50

KServe Deep Dive: Evolving Model Serving for the Generative AI Era

Yuan Tang

Senior Principal Software Engineer, Red Hat AI

​

Project Lead: KServe, Argo, and Kubeflow

Co-chair: K8s WG Serving and WG AI Conformance

Co-chair: CNCF TAG Workloads Foundation

​

2 of 50

Agenda

01

02

03

04

05

​

​

What is KServe

History

Open Source Community

Architecture

Key Features

​

3 of 50

What is KServe?

Highly scalable, standard, cloud agnostic model inference platform on Kubernetes

​

​

4 of 50

Why KServe?

Provides performant, standardized inference protocol across ML frameworks.

​

​

Supports modern serverless inference workload with autoscaling including Scale to Zero on GPU.

​

Provides high scalability, density packing and intelligent routing using ModelMesh

Simple and pluggable production serving for production ML serving including prediction, pre/post processing, monitoring and explainability.

​

​

Advanced deployments with canary rollout, experiments, ensembles and transformers.

​

GenAI capabilities: Envoy AI Gateway, KEDA, LMCache. Model Cache, vLLM multi-node inference

​

4

#IBMTechXchange

5 of 50

History of KServe

5

From Kubeflow/KFServing to KServe

  • Developed collaboratively by Google, IBM, Bloomberg, NVIDIA, and Seldon in 2019 under the Kubeflow project.
  • The project graduated from Kubeflow and was rebranded from KFServing to standalone KServe project in Sep 2021.
  • KServe 0.7 released outside of the Kubeflow with migration guide for minimal disruptions in Oct 2021.
  • KServe was donated to LF AI & Data Foundation in Feb 2022.
  • KServe was accepted as a CNCF incubating project in Sep 2025.

#IBMTechXchange

6 of 50

History of KServe

6

Kubeflow Ecosystem

#IBMTechXchange

7 of 50

Community: Maintainers and Contributors

7

19 maintainers and 300+ contributors!

#IBMTechXchange

8 of 50

Community: Adopters

8

30+ companies varying from vendors to end users!

#IBMTechXchange

9 of 50

KServe Architecture

9

#IBMTechXchange

10 of 50

KServe Feature: Serving Runtimes

10

Pluggable, reusable, extensible runtimes

  • Framework specific: PyTorch, TensorFlow, Sklearn, Paddle, XGBoost, LightGBM, PMML
  • Generic/Multi-framework support: MLServer, ONNX, Hugging Face (vLLM and Triton backends)
  • Custom runtime support

#IBMTechXchange

11 of 50

KServe Feature: Serving Runtimes

11

Hugging Face runtime

  • ​

#IBMTechXchange

12 of 50

KServe Feature: Serving Runtimes

12

Custom runtime:

Predictor definition

#IBMTechXchange

13 of 50

KServe Feature: Serving Runtimes

13

Custom runtime:

InferenceService

#IBMTechXchange

14 of 50

KServe Feature: Pre/Post Processing

14

Transforms image via torchvision

  • ​

#IBMTechXchange

15 of 50

KServe Feature: Pre/Post Processing

15

v1 REST API

  • ​

#IBMTechXchange

16 of 50

KServe Feature: Pre/Post Processing

16

v2 gRPC API

  • ​

#IBMTechXchange

17 of 50

KServe Feature: Inference Graph

17

TBA

#IBMTechXchange

18 of 50

KServe Feature: Model Storage

18

Supported storage formats

  • Azure, S3, GCS
  • URI
  • PVC (K8s Persistent Volume Claim)
  • OCI
  • Storage Containers

#IBMTechXchange

19 of 50

KServe Feature: Model Storage

19

PVC example

#IBMTechXchange

20 of 50

KServe Feature: Model Storage

20

S3 example

#IBMTechXchange

21 of 50

KServe Feature: Model Storage - ModelCars

21

Efficient model pulling from OCI image registry

  • Reduced Startup Times: By avoiding repetitive downloads of large models, startup delays are significantly minimized.
  • Lower Disk Space Usage: The feature decreases the need for duplicated local storage, conserving disk space.
  • Enhanced Performance: ModelCars allows for advanced techniques like prefetching images and lazy-loading, improving efficiency.

#IBMTechXchange

22 of 50

KServe Feature: Model Storage - ModelCars

22

  • OCI URI schema as the model reference.
  • Replacing the init-container with a sidecar container.
  • Configuring the pod with shareProcessNamespace: true to facilitate access between sidecars.
  • Accessing to model data in a sidecar container without copying it into a shared volume. No init-container is needed.

#IBMTechXchange

23 of 50

KServe Feature: Model Storage - ModelCars

23

Build Image with Model

Specify StorageURI

#IBMTechXchange

24 of 50

KServe Feature: Model Storage

24

Custom Storage Initializer

  • Custom Model Registry / Storage Containers
  • Problem: KServe downloads models using a built-in storage initializer and cannot support custom storage systems.
  • Solution: KServe provides ClusterStorageContainer CRD to allow users to define custom model registry protocol, e.g. model-registry://.

#IBMTechXchange

25 of 50

KServe Feature: Model Storage

25

ClusterStorageContainer

​

Custom container spec for a list of supported URI formats

#IBMTechXchange

26 of 50

KServe Feature: Model Storage

26

Custom model-registry:// protocol

#IBMTechXchange

27 of 50

KServe Feature: Request Batching

27

  1. Requests go to the model agent container.
  2. The batcher in sidecar container batches the requests.
  3. The batcher sends the inference request to the predictor container.

#IBMTechXchange

28 of 50

KServe Feature: Autoscaling

28

Knative autoscaling

  • ​

#IBMTechXchange

29 of 50

KServe Feature: Autoscaling

29

Send traffic in 30 seconds spurts maintaining 5 in-flight requests

hey -z 30s -c 5 …

#IBMTechXchange

30 of 50

KServe Feature: Autoscaling

30

Scale down to zero

#IBMTechXchange

31 of 50

KServe Feature: Explainer / TrustyAI

31

Pluggable explainer runtimes

​

“Why did my model produce this inference result?”

​

​

  • ​

#IBMTechXchange

32 of 50

KServe Feature: Monitoring

32

Prometheus Metrics

Supported serving runtimes exports prometheus metrics on a specified port in the inference service's pod

#IBMTechXchange

33 of 50

KServe Feature: Monitoring

33

Prometheus Metrics��

  • Prometheus latency histograms are emitted for each of the steps (pre/postprocessing, explain, predict).
  • The latencies of each step are logged per request.
  • Each model server may implement its own set of metrics.
  • Knative/Queue proxy emits metrics be default

#IBMTechXchange

34 of 50

KServe Feature: Monitoring

34

Grafana Dashboards

  • Knative HTTP Dashboard
  • KServe Dashboards for Runtimes
    • ModelServer Latency Dashboard
    • TorchServe Latency Dashboard
    • Triton Latency Dashboard
    • …

#IBMTechXchange

35 of 50

KServe Feature: Canary Rollout

35

#IBMTechXchange

36 of 50

KServe Feature: GitOps Friendly

36

ArgoCD includes custom health checks for KServe CRDs

#IBMTechXchange

37 of 50

KServe Feature: Deployment Mode

37

Knative vs. Standard

  • Knative/Serverless Mode:
    • Requires Knative
    • Uses Knative Pod Autoscaler (KPA)
    • Supports canary rollout
  • Standard/Raw Deployment Mode:
    • No dependency on Knative
    • Uses K8s Deployment, Service, HPA
    • Does not support Scale down and from Zero

#IBMTechXchange

38 of 50

GenAI Feature: LMCache Integration

38

Key-Value (KV) cache offloading

with vLLM + LMCache

​

Check out LMCache session at 3:30 at Lake Louise (Lobby Level)

  • Reduced Latency: Faster inference and lower prefill delay (Time To First Token, TTFT) by skipping repeated computation for overlapping context.
  • Lower GPU Utilization: Less computation per request, freeing up GPU resources for more users or larger models.
  • Efficient Memory Usage: Centralized, deduplicated cache storage reduces overall memory footprint.
  • Scalability: Multiple inference servers can access and update the same cache, supporting distributed, high-throughput deployments.
  • Improved User Experience: Especially beneficial for multi-turn QA, retrieval-augmented generation (RAG), and scenarios with repeated or similar prompts.

#IBMTechXchange

39 of 50

GenAI Feature: Model Cache

39

#IBMTechXchange

40 of 50

GenAI Feature: Model Cache

40

#IBMTechXchange

41 of 50

GenAI Feature: KEDA Autoscaling

41

TBA

  • KEDA (Kubernetes Event-driven Autoscaler)
    • Enable event-driven scaling beyond K8s HPA
    • Scale based on external metrics such as vLLM metrics for the number of waiting requests or KV Cache usage
    • Polling can introduce latency and additional load on the cluster
  • OpenTelemetry-based push metrics
    • More efficient and real-time autoscaling, reducing the overhead associated with frequent polling

​

#IBMTechXchange

42 of 50

GenAI Feature: Multi-node Inference

42

TBA

  • workerSpec.tensorParallelSize
    • configure tensor parallelism - how the model weights are sharded across GPUs.
  • workerSpec.pipelineParallelSize
    • configure pipeline parallelism - how the model layers are split across the GPUs.

#IBMTechXchange

43 of 50

GenAI Feature: Rate Limiting with Envoy AI Gateway

43

#IBMTechXchange

44 of 50

GenAI Feature: Rate Limiting with Envoy AI Gateway

44

  • Request Routing: Directs API requests to appropriate GenAI services.
  • Authentication and Authorization: Implements API key validation to secure communication.
  • Backend Security Policy: Introduces fine-grained access control for backend services, controlling LLM/AI backend usage using token-per-second (TPS) policies to prevent overuse.
  • Multi-Upstream Provider Support for LLM/AI Services: The ability to receive requests in the format of one LLM provider and route them to different upstream providers, ensuring compatibility with their expected formats. This is achieved through built-in transformation capabilities that adapt requests and responses accordingly.

#IBMTechXchange

45 of 50

GenAI Feature: llm-d Integration

45

#IBMTechXchange

46 of 50

GenAI Feature: llm-d Integration

46

  • vLLM-Optimized Inference Scheduler: llm-d builds on IGW's pattern for customizable “smart” load-balancing via the Endpoint Picker Protocol (EPP) to define vLLM-optimized scheduling. Leveraging operational telemetry, the Inference Scheduler implements the filtering and scoring algorithms to make decisions with P/D-, KV-cache-, SLA-, and load-awareness. Advanced teams can implement their own scorers to further customize, while benefiting from other features in IGW, like flow control and latency-aware balancing.�
  • Disaggregated Serving with vLLM: llm-d leverages vLLM’s support for disaggregated serving to run prefill and decode on independent instances, using high-performance transport libraries like NIXL. llm-d plans to support a latency-optimized implementation using fast interconnects (IB, RDMA, ICI) and a throughput optimized implementation using data-center networking.

#IBMTechXchange

47 of 50

GenAI Feature: llm-d Integration

47

​

  • Disaggregated Prefix Caching with vLLM: llm-d uses vLLM's KVConnector to provide a pluggable KV cache hierarchy, including offloading KVs to host, remote storage, and systems like LMCache.

​

  • Variant Autoscaling over Hardware, Workload, and Traffic: llm-d plans to implement a traffic- and hardware-aware autoscaler that (a) measures the capacity of each model server instance, (b) derive a load function that takes into account different request shapes and QoS, and (c) assesses recent traffic mix (QPS, QoS, and shapes) to calculate the optimal mix of instances to handle prefill, decode, and latency-tolerant requests, enabling use of HPA for SLO-level efficiency.

​

Check out llm-d session at 3pm at Lake Louise (Lobby Level)!

​

#IBMTechXchange

48 of 50

GenAI Feature: KServe + llm-d + Envoy AI Gateway

48

#IBMTechXchange

49 of 50

Future of KServe

49

Community Collaboration

  • Envoy
  • Kubeflow
  • vLLM
  • llm-d
  • LMCache
  • Kubernetes WG Serving
  • Knative
  • …

#IBMTechXchange

50 of 50

Stay Connected!

Email: terrytangyuan@gmail.com

LinkedIn/X/GitHub: @TerryTangYuan

Bluesky: @terrytangyuan.xyz

​

​

​

50

#IBMTechXchange