East Coast vLLM Meetup
2025-03-11
Welcome!
5:00 - 6:00
6:00 - 6:30
6:30 - 6:50
6:50 - 7:10
7:10 - 7:15
7:15 - 7:45
7:45 - 9:00
Meet & Greet
vLLM Update
Model Optimization in vLLM
Hardware Update (Redacted)
Join the Open Source AI Movement!
Q&A
Social Hour & Networking
vLLM Update
Simon Mo
PhD Student, UC Berkeley
vLLM Maintainer
vLLM's Goal
Build the fastest and
easiest-to-use open-source
LLM inference & serving engine
4
vLLM Today
5
$ pip install vllm
41K Stars
Official
release!
vLLM API (1): LLM class
6
from vllm import LLM
# Example prompts.
prompts = ["Hello, my name is", "The capital of France is"]
# Create an LLM with HF model name.
llm = LLM(model="meta-llama/Meta-Llama-3.1-8B")
# Generate texts from the prompts.
outputs = llm.generate(prompts) # also llm.chat(messages)]
A Python interface for offline batched inference
vLLM API (2): OpenAI-compatible server
7
$ vllm serve meta-llama/Meta-Llama-3.1-8B
$ curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-8B",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": 0
}'
A FastAPI-based server for online serving
Server
Client
Q1 Update Agenda
8
vLLM V1
What is vLLM V1?
Re-architect the “core” of vLLM
based on the lessons from V0 (current version)
10
Unchanged
Changed
Why vLLM V1?
11
Key changes in vLLM V1
12
Key changes in vLLM V1
13
Simplified Scheduler
14
| | | | | | | | | |
Step 0
| | | | | | | | | |
Step 1
| | | |
Step 2
| | |
| | | | |
| | | | | | | | | |
| |
R1
R2
R3
Token budget (10)
| | |
Step 3
{R1: 3, R2: 5, R3: 2}
{R1: 1, R2: 1, R3: 8}
{R1: 1, R2: 1, R3: 2}
{R1: 1, R2: 1, R3: 1}
Scheduler Output
{request: num_tokens}
Prompts
Simplified Scheduler (cont’d)
15
AI | is | the | future | of |
| | |
num_computed_tokens: 0
all_token_ids
Schedule 5 tokens (“prefill”)
AI | is | the | future | of |
tech | | |
num_computed_tokens: 5
all_token_ids
Schedule 1 token (“decode”)
Simplified Scheduler (cont’d)
16
AI | is | the | future | of |
| | |
num_computed_tokens: 0
all_token_ids
Schedule 3 tokens out of 5
AI | is | the | future | of |
| | |
num_computed_tokens: 3
all_token_ids
Schedule 2 tokens
Simplified Scheduler (cont’d)
17
AI | is | the | future | of |
| | |
num_computed_tokens: 2
all_token_ids
Schedule 3 tokens
Cache hit!
Simplified Scheduler: Next Steps
18
Piecewise CUDA Graphs
→ Increased development burden
19
Q
K
V
Attention
O
MLP 0
MLP 1
Q
K
V
Piecewise CUDA Graphs (cont’d)
20
Q
K
V
Attention
O
MLP 0
MLP 1
Q
K
V
CUDA graph N
CUDA graph N-1
PyTorch Eager
Graph split using torch.compile
Piecewise CUDA Graphs (cont’d)
21
Q
K
V
Attention
O
MLP 0
MLP 1
Q
K
V
Potential Use Cases of Piecewise CUDA Graphs
Piecewise CUDA graphs will allow vLLM to easily integrate new optimizations such as:
22
V1 Current Status
23
DeepSeek Support
State of DeepSeek Model Support
4 Major Performance Levers:
vLLM’s unique features:
25
V0 Results of MLA and MTP
Multi-token prediction (k=1): Improve TPOT with low QPS
Multi-head Latent Attention: 2.4x decoding throughput against MHA
26
Q1 Roadmap
vLLM Core
28
Features
29
Ecosystem Projects
30
Welcome AIBrix!
Thank you sponsors (funding compute!)
31
Our Goal
Build the fastest and
easiest-to-use open-source
LLM inference & serving engine
32
33
Building the fastest and easiest-to-use open-source LLM inference & serving engine!
https://twitter.com/vllm_project
https://opencollective.com/vllm
Model Optimization in vLLM
Robert Shaw
Director of Engineering, Red Hat
vLLM Committer
Why Optimize Your Model?
35
Breakdown of performance of individual decode iterations for Llama-3-8B FP16 and FP8. The x-axis sweeps over batch size 1 to 128. Pink regions are the linear layers
Quantization Support in vLLM
36
Our goal with vLLM
vLLM quantization design
Kernels
We write custom CUTLASS and Triton kernels for quantized compute
FP8 W8A8 in vLLM is faster than baseline PyTorch implementation
MARLIN kernel in vLLM targets the inference serving regime
Ecosystem Compatible
We work hard to make sure key quantization formats run on vLLM
and AutoAWQ into MARLIN format
Pre Optimized Models
Maintained, pre-optimized models �ready to deploy to production with vLLM
39
Comprehensive Validation
Extensive Selection
Broad Collection
Instinct
GPUs
CPUs
TPUs
Formats
Hardware
Algorithms
Llama
Qwen
Mistral
DeepSeek
Gemma
Phi
Molmo
Granite
Nemotron
LLM Compressor
Unified framework for creating compressed models
Comprehensive set of algorithms in unified interface
Seamless integration w/ HF AutoModel
Safetensors-based checkpoint format compatible with vLLM
Large model support via HF accelerate
40
The Case For Activation Quantization
41
Linear Layers
42
A GEMM computed in a vLLM linear layer.
W is the weight matrix and X is the input activation.
M is the flattened number of tokens being processed.
Drivers of GEMM Latency
Latency is driven by (1) bandwidth from HBM → SRAM and (2) tensor core FLOPs
43
Quantization can enable more ops per second
Quantization reduces data from HBM → SRAM
A100 Memory Bandwidth
H100 Tensor Core FLOPS
Quantization Flavors
We support both weight-only and weight+activation quantization
44
Weight-Only Quantization
Weight+Activation Quantization
Why Activation Quantization?
Activation quantization accelerates both compute-bound and memory-bound cases
45
A GEMM computed in a vLLM linear layer.
W is the weight matrix and X is the input activation.
M is the flattened number of tokens being processed.
What About Accuracy?
46
Fine Grained Quantization
Fine-grained quantization enables us to drive significant accuracy
47
X and W are quantized as:
����Rewrite the computation as:���
Inside: A quantized GEMM.
Outside: An elementwise multiplication happening during the epilogue.
torch._scaled_mm
Our CUTLASS kernels
Scales can be per-tensor: Or per-token/channel:
Quantized DeepSeek-R1 Models�Deployment-Ready Reasoning Models
48
Pass@1 score and standard deviation for quantized models on the popular reasoning benchmarks
Join the Open Source AI Movement
Saša Zelenović
Principal Product Marketing Manager, Red Hat
Contribute to Key vLLM Features
50
Join Bi-Weekly vLLM Office Hours
51
Let Us Know How We Did Today!
Take 1 minute now to complete our 3-question survey.
52
red.ht/4bwOplj
Q&A
Thank You!