Unlocking Potential of Large Models in Production

Yuan Tang, Principal Software Engineer at Red Hat

Adam Tetelman, Principal Product Architect at NVIDIA

Meet Your Speakers

Adam Tetelman

Principal Product Architect

NVIDIA

Yuan Tang

Principal Software Engineer

Red Hat

Session Agenda

History and Problem Statement:

Serving Large Language Models in Production�

Solutions to serving LLMs:

KServe Case Study�

Future Problems to Solve:

Feature Gaps, Roadmaps, AI Advancements, and Community Engagement

​

Serving Large Language Models in Production

Inference, RAG and Fine Tuning Pipelines

Model Fine Tuning

Model Evaluation

Enterprise Data

LLM

Embedding

Reranking

Vector Database

Embedding

LLM Guard

LLM Guard

Fine-tune LLM

LoRA Adapters

Enterprise

AI Flywheel

Inference Servers

Fine Tuning Jobs�Dependencies

LLM Frameworks

Summary of Agentic RAG System

Inference

Inference Server: A server to wrap models in a callable API and provide additional operational services.

Inference Platform: A platform used to manage deployed inference servers and provide advanced operational capabilities.�

RAG System

Embedding Model: Model used to convert text or objects into searchable embeddings

Reranking Model: Model used to improve the search results from a Vector DB.

Vector DB: Specialized database used to store and search embedding.

Data Connectors: APIs and integrations used to pull live data from web services or external data sources.

LLM Agent System

Agent / Chain Server: Application with memory that has access to RAG systems, LLM models, and tooling. Applies reasoning and planning to manage agentic conversations.

Large Language Model:: Foundational model that takes an input of user-generated text and system-provided context and generates a text output.

PEFT/LoRA: A form of fine-tuning that adds or overrides a small section of a foundational LLM to add specialized knowledge.

Tools: External programmatic capabilities that can be made available to an agent.

Guardrails: A lightweight system with models for detecting inappropriate content. Can be run on all user input and all LLM output.

The Building Blocks of a GenAI Workflow

A basic developer-focused LLM MVP

Model

Kubernetes

LLM API

Inference Server

Adding workflow and agentic capabilities for real-world scenarios

Model

Kubernetes

Conversational LLM API

AI Tools & Services

Agentic Workflows

Inference Platform

Inference Server

Introducing explainability with RAG

Model

Kubernetes

Conversational LLM API

AI Tools & Services

Agentic Workflows

Inference Platform

Vector DB

External Data

RAG System

LLM Model

Embedding Model

Inference Server

Improving accuracy with fine-tuning

Fine-tuning

Model

Kubernetes

Conversational LLM API

AI Tools & Services

Agentic Workflows

Inference Platform

Vector DB

External Data

RAG System

LLM Model

Embedding Model

Customization Service

Data Curation

Evaluation Service

Data

Inference Server

LoRA

Going from MVP to production

Fine-tuning

Inference Server

Optimized Model

Kubernetes

Conversational LLM API

AI Tools & Services

Agentic Workflows

Inference Platform

Vector DB

External Data

RAG System

LLM Model

Embedding Model

Customization Service

Data Curation

Evaluation Service

Data

Inference Server

Inference Server

Guardrails

LoRA

Evaluation Techniques

Training Accuracy Evaluation

Various evaluation technique for accuracy/speed

​

• Does the system answer predefined questions correctly? (Academic benchmarks)
• Standard benchmarks
• Industry benchmarks�
• Does the system answer predefined questions about your use-case? (Custom benchmarks)
• Company-specific benchmarks�
• Does the system perform “better” than other LLMs?
• Automated comparisons (LLM-as-judge)
• Manual expert comparisons (Human-as-judge)�
• How does the throughput and latency compare to similar models (performance)

Inference Evaluation

Metrics are evaluated system-wide and per-model

​

• How many tokens are generated? (Throughput)
• Overall
• Per-user
• How fast do users get a response? (Latency)
• Average
• Max
• Min
• 95%
• Time to first token (TTFS)
• Which accelerators are being used? (Cost)

Operational Issues

Scaling

• Scaling metrics
• Startup times
• Scheduling�

​

Caching

• Models
• Containers
• Customizations
• Fine-tuning data

Monitoring

• Standard application metrics
• Infrastructure metrics
• Automated guardrails��

Upgrades

• A/B Testing
• Rolling Upgrades
• Hardware availability
• LLM Routing

Solutions to serving large models, KServe a case study

What is KServe?

Highly scalable, standard, cloud agnostic model inference platform on Kubernetes

Why KServe?

• Provides performant, standardized inference protocol across ML frameworks.�
• Provides high scalability, density packing, and intelligent routing using ModelMesh.�
• Advanced deployments with canary rollout, ensembles and transformers.�
• Supports modern serverless inference workload with autoscaling including Scale to Zero on GPU.�
• Simple and pluggable production serving for production ML serving including predictive/generative AI, pre/post processing, monitoring and explainability.

Community: Maintainers and Contributors

15 maintainers and 250+ contributors!

Community: Adopters

30+ companies varying from vendors to end users!

KServe Architecture: Single Model

KServe Architecture: Multi Model (ModelMesh)

​

​

Designed for high-scale, high-density and frequently-changing model use cases.

​

Intelligently loads and unloads models to and from memory to strike an intelligent trade-off between responsiveness to users and computational footprint.

KServe Key Features: Serving Runtimes

Pluggable, reusable, extensible runtimes

• Framework specific: PyTorch, TensorFlow, Sklearn, Paddle, XGBoost, LightGBM, PMML
• Generic/Multi-framework support: Hugging Face (vLLM and Triton backends), MLServer, ONNX
• Custom runtime support

​

​

KServe Key Features: Serving Runtimes

Hugging Face Runtime (vLLM and Triton backends)

KServe Key Features: Model Storage

Supported storage formats

​

​

• Azure, S3, GCS
• URI
• PVC
• Storage Containers
• e.g. custom mode-registry:// protocol with Kubeflow Model Registry
• OCI (via ModelCar)

KServe Key Features: Model Storage - ModelCars

Efficient model pulling from OCI image registry

​

​

• Reduced Startup Times: By avoiding repetitive downloads of large models, startup delays are significantly minimized.
• Lower Disk Space Usage: The feature decreases the need for duplicated local storage, conserving disk space.
• Enhanced Performance: ModelCars allows for advanced techniques like prefetching images and lazy-loading, improving efficiency.

​

​

KServe Key Features: Model Storage - ModelCars

• OCI URI schema as the model reference.
• Replacing the init-container with a sidecar container.
• Configuring the pod with shareProcessNamespace: true to facilitate access between sidecars.
• Accessing to model data in a sidecar container without copying it into a shared volume. No init-container is needed.

KServe Key Features: Model Storage - ModelCars

Build and Push Image with Model

​

​

​

​

Specify StorageURI

​

​

​

​

KServe Key Features: Multi-node Inference

Multi-node inference via Hugging Face Runtime (vLLM backend)

​

​

apiVersion: serving.kserve.io/v1beta1

kind: InferenceService

metadata:

annotations:

serving.kserve.io/deploymentMode: RawDeployment

serving.kserve.io/autoscalerClass: external

name: huggingface-llama3

spec:

predictor:

model:

runtime: kserve-huggingfaceserver-multinode

modelFormat:

name: huggingface

storageUri: pvc://llama-3-8b-pvc/hf/8b_instruction_tuned

workerSpec:

pipelineParallelSize: 2

tensorParallelSize: 1

KServe Key Features: Autoscaling

Knative autoscaling

KServe Key Features: Autoscaling

Send traffic in 30 seconds spurts maintaining 5 in-flight requests

hey -z 30s -c 5 …

​

​

​

KServe Key Features: Autoscaling

Scale down to zero

​

​

KServe Key Features: Canary Rollout Strategy

KServe Key Features: Monitoring

Prometheus Metrics

• Prometheus latency histograms are emitted for each of the steps (pre/postprocessing, explain, predict).
• The latencies of each step are logged per request.
• Each model server may implement its own set of metrics.
• Knative/Queue proxy emits metrics by default

KServe Key Features: Monitoring

Grafana Dashboards

​

​

• Knative HTTP Dashboard
• KServe Dashboards for Runtimes
• ModelServer Latency Dashboard
• TorchServe Latency Dashboard
• Triton Latency Dashboard
• …

​

​

KServe Key Features: Explainer/TrustyAI

Pluggable explainer runtimes

“Why did my model produce this inference result?”

Future problems to solve, gaps, and community engagement

Agents and Tools

Reasoning and Planning:

Decomposing complex tasks into manageable subgoals through reasoning

​

Memory:

Short-term memory in an LLM-powered agent acts as a record of actions and thoughts during a single query

​

Long-term memory:

Logs interactions between the user and agent over extended periods�

Tools:

Defined executable workflows that agents use to perform tasks

​

​

End to End Inference & Training Workflow

Training Techniques

Inference Optimizations

Inference Platform Features

​

• Response Caching
• Context Caching
• Inflight Batching

Model Features

​

• Multi-LoRA Loading
• Just-in-time Compilation (vLLM)
• Ahead-of-time Compilation (TRT-LLM)

TensortRT runtime

Sequence batcher

Seq 0

Seq 4

Seq 3

Seq 2

Seq 1

Seq 5

Instance 1

Instance 2

Slot 1

Slot 2

Slot 1

Slot 2

Model Cache Management

GPU Node

GPU Node

Encrypted

GPU Node

LoRA Adapters

Fine Tune

…

Namespace A

Namespace B

Model Repos

Decrypted

Model Caching

Deploy

Multi-node Inference: Models too big for 1 node

Scheduling

• Avoiding deadlock (Gang Scheduling)
• Avoiding unusable GPUs (Binpacking)�

Operations

• Monitor individual and groups of Pods
• Handling scaling complexity
• Optimizing for throughput or cost�

Optimizations

• Accelerate model loading (Caching on Shared Storage)
• Accelerate cross-node communication (Advanced Networking with RoCE / RDMA)

Future of KServe: GenAI Inference

Serving Runtimes

• Support LoRA adapters, speculative decoding (vLLM)
• TensorRT-LLM, TGI, and benchmarking
• Improve multi-node/multi-host inference

​

Autoscaling

• Support model caching with automatic PVC/PV provisioning
• Autoscaling based on custom metrics

​

Agent/RAG Pipeline Orchestration

• Support declarative RAG/Agent workflow using KServe Inference Graph

Future of KServe: GenAI Inference

Open Inference Protocol extension to GenAI Task APIs

• Community-maintained Open Inference Protocol repo for OpenAI schema
• Support vertical GenAI Task APIs such as embedding, Text-to-Image, Text-To-Code, Doc-To-Text

​

LLM Gateway

• Support multiple LLM providers.
• Support token based rate limiting.
• Support LLM router with traffic shaping, fallback, load balancing.
• LLM Gateway observability for metrics and cost reporting

​

Roadmap: https://github.com/kserve/kserve/blob/master/ROADMAP.md

Future of KServe: Community Collaboration

• Kubernetes WG Serving
• Check out our talk on Friday: WG Serving: Accelerating AI/ML Inference Workloads on Kubernetes
• Collaboration with corporate partners and community maintainers
• Check out our panel at 5:25pm today: Engaging the KServe Community, The Impact of Integrating a Solutions with Standardized CNCF Projects
• Collaboration and Integration with Ecosystem Projects
• vLLM
• Kubeflow
• Envoy
• …

WG-Serving & KServe Talks

AI Day, Sponsored Keynote: Advancing Cloud Native AI Innovation Through Open Collaboration

​

Optimizing Load Balancing and Autoscaling for Large Language Model (LLM) Inference on Kubernetes

​

Engaging the KServe Community, The Impact of Integrating a Solutions with Standardized CNCF Projects (today at 5:25pm)

​

Best Practices for Deploying LLM Inference, RAG and Fine Tuning Pipelines on K8s

​

WG Serving: Accelerating AI/ML Inference Workloads on Kubernetes

Q&A

Sessions from NVIDIA

Sessions from Red Hat

Observability & Metrics Standards

Custom Metrics API

…

Prometheus Adapter

Operator

DCGM/Model Server Metrics

https://docs.nvidia.com/nim/large-language-models/latest/observability.html

Level 1 – Infra - GPU metrics

Level 2 - Inference Platform metrics

Level 3 – Model Server Metrics

NVIDIA & Community Talks

ArgoCon: Building a Cutting-Edge Kubernetes Internal Developer Platform at NVIDIA | Tuesday 10:40am - 11:05am

Lightning Talk: Evaluating Scheduler Efficiency for AI/ML Jobs Using Custom Resource Metrics | Tuesday 5:45pm - 5:50pm

Lightning Talk: Running Kind Clusters with GPU Support Using Nvkind | Tuesday 6:05pm - 6:10pm

Keynote: NVIDIA Case Study: The Many Facets of Building + Delivering AI in the Cloud Native Ecosystem | Wednesday 10:05am-10:20am

All-Your-GPUs-Are-Belong-to-Us: An Inside Look at NVIDIA's Self-Healing GeForce NOW Infrastructure | Wednesday 11:15am - 11:50am

Maintainer Track: Kubernetes WG Device Management - Advancing K8s Support for GPUs | Wednesday 2:30pm - 3:05pm

AIStore as a Fast Tier Storage Solution: Enhancing Petascale Deep Learning Across Cloud Backends | Wednesday 2:30pm - 3:05pm

A Tale of 2 Drivers: GPU Configuration on the Fly Using DRA | Wednesday 3:25pm - 4:00pm

Tutorial: Get the Most Out of Your GPUs on Kubernetes with the GPU Operator | Wednesday 4:30pm - 6:00pm

From Silicon to Service: Ensuring Confidentiality in Serverless GPU Cloud Functions | Thursday 11:00am - 11:35am

Unlocking Potential of Large Models in Production | Thursday 2:30pm - 3:05pm

Which GPU Sharing Strategy Is Right for You? a Comprehensive Benchmark Study Using DRA | Thursday 4:30pm - 5:05pm

Engaging the KServe Community, The Impact of Integrating a Solutions with Standardized CNCF Projects | Thursday 5:25pm - 6:00pm

Maintainer Track: WG Serving: Accelerating AI/ML Inference Workloads on Kubernetes | Friday 11:55am - 12:30pm

From Vectors to Pods: Integrating AI with Cloud Native | Friday 2:00pm - 2:35pm

Enabling Fault Tolerance for GPU Accelerated AI Workloads in Kubernetes | Friday 2:55pm - 3:30pm

Thousands of Gamers, One Kubernetes Network | Friday 2:55pm - 3:30pm

Best Practices for Deploying LLM Inference, RAG and Fine Tuning Pipelines on K8s | Friday 4:00pm - 4:35pm

Best of Both Worlds: Integrating Slurm with Kubernetes in a Kubernetes Native Way | Friday 4:55pm - 5:30pm

​

​

Extra Slides

Agenda Outline

https://kccncna2024.sched.com/event/1i7ns

​

Thursday November 14, 2024 2:30pm - 3:05pm MST

​

The recent paradigm shift from traditional ML to GenAI and LLMs has brought with it a new set of non-trivial LLMOps challenges around deployment, scaling, and operations that make building an inference platform to meet all business requirements an unsolved problem. This talk highlights these new challenges along with best-practices and solutions for building out large, scalable, and reliable inference platforms on top of cloud native technologies such as Kubernetes, Kubeflow, Kserve, and Knative. Which tools help effectively benchmark and assess the quality of an LLM? What type of storage and caching solutions enable quick auto-scaling and model downloads? How can you ensure your model is optimized for the specialized accelerators running in your cluster? How can A/B testing or rolling upgrades be accomplished with limited compute? What exactly do you monitor in an LLM? In this session we will use KServe as a case study to answer these questions and more.

​

Multi-LoRA

End to End RAG Workflow

Model Caching

Typical Workflow