1 of 54

East Coast vLLM Meetup

2025-03-11

2 of 54

Welcome!

5:00 - 6:00

6:00 - 6:30

6:30 - 6:50

6:50 - 7:10

7:10 - 7:15

7:15 - 7:45

7:45 - 9:00

Meet & Greet

vLLM Update

Model Optimization in vLLM

Hardware Update (Redacted)

Join the Open Source AI Movement!

Q&A

Social Hour & Networking

3 of 54

vLLM Update

Simon Mo

PhD Student, UC Berkeley

vLLM Maintainer

4 of 54

vLLM's Goal

Build the fastest and

easiest-to-use open-source

LLM inference & serving engine

4

5 of 54

vLLM Today

5

$ pip install vllm

41K Stars

Official

release!

6 of 54

vLLM API (1): LLM class

6

from vllm import LLM

# Example prompts.

prompts = ["Hello, my name is", "The capital of France is"]

# Create an LLM with HF model name.

llm = LLM(model="meta-llama/Meta-Llama-3.1-8B")

# Generate texts from the prompts.

outputs = llm.generate(prompts) # also llm.chat(messages)]

A Python interface for offline batched inference

7 of 54

vLLM API (2): OpenAI-compatible server

7

$ vllm serve meta-llama/Meta-Llama-3.1-8B

$ curl http://localhost:8000/v1/completions \

-H "Content-Type: application/json" \

-d '{

"model": "meta-llama/Meta-Llama-3.1-8B",

"prompt": "San Francisco is a",

"max_tokens": 7,

"temperature": 0

}'

A FastAPI-based server for online serving

Server

Client

8 of 54

Q1 Update Agenda

  • vLLM V1
  • DeepSeek Enhancements
  • Q1 Roadmap Update
  • Ecosystem Projects

8

9 of 54

vLLM V1

10 of 54

What is vLLM V1?

Re-architect the “core” of vLLM

based on the lessons from V0 (current version)

10

  • User-level APIs
  • Models
  • GPU Kernels
  • Utility functions
  • ...
  • Scheduler
  • Memory Manager
  • Model Runner
  • API Server
  • ...

Unchanged

Changed

11 of 54

Why vLLM V1?

  • Main goals:
    • Simple & easy-to-hack codebase
      • vllm/v1/core/scheduler.py 608 LOC (>2k LOC in v0)
    • High performance with near-zero CPU overheads
    • Combining all key optimizations & enabling them by default

11

12 of 54

Key changes in vLLM V1

  1. Optimized engine loop & API server

  • Simplified scheduler

  • Clean implementation of distributed inference

  • Piecewise CUDA graphs

12

13 of 54

Key changes in vLLM V1

  • Optimized engine loop & API server

  • Simplified scheduler

  • Clean implementation of distributed inference

  • Piecewise CUDA graphs

13

14 of 54

Simplified Scheduler

  • Synchronous single-step scheduler
  • Chunked prefills (aka Dynamic SplitFuse) by default
    • The scheduling decision is simply represented as a dictionary of {request_id: num_tokens}

14

Step 0

Step 1

Step 2

R1

R2

R3

Token budget (10)

Step 3

{R1: 3, R2: 5, R3: 2}

{R1: 1, R2: 1, R3: 8}

{R1: 1, R2: 1, R3: 2}

{R1: 1, R2: 1, R3: 1}

Scheduler Output

{request: num_tokens}

Prompts

15 of 54

Simplified Scheduler (cont’d)

  • Unification of “prefill” and “decode”
    • In V1, there’s no concept of prefill and decode
    • Schedule based on the difference between num_compute_tokens and len(all_token_ids)
  • Ex1) “Prefill” & “Decode”

15

AI

is

the

future

of

num_computed_tokens: 0

all_token_ids

Schedule 5 tokens (“prefill”)

AI

is

the

future

of

tech

num_computed_tokens: 5

all_token_ids

Schedule 1 token (“decode”)

16 of 54

Simplified Scheduler (cont’d)

  • Unification of “prefill” and “decode”
    • In V1, there’s no concept of prefill and decode
    • Schedule based on the difference between num_compute_tokens and len(all_token_ids)
  • Ex2) Chunked prefills

16

AI

is

the

future

of

num_computed_tokens: 0

all_token_ids

Schedule 3 tokens out of 5

AI

is

the

future

of

num_computed_tokens: 3

all_token_ids

Schedule 2 tokens

17 of 54

Simplified Scheduler (cont’d)

  • Unification of “prefill” and “decode”
    • In V1, there’s no concept of prefill and decode
    • Schedule based on the difference between num_compute_tokens and len(all_token_ids)
  • Ex3) Prefix caching

17

AI

is

the

future

of

num_computed_tokens: 2

all_token_ids

Schedule 3 tokens

Cache hit!

18 of 54

Simplified Scheduler: Next Steps

  • Current: First-come-first-served policy is baked in the scheduler

  • Next step 1: Support various scheduling policies
    • Priority-based scheduling
    • Fair scheduling
    • Predictive scheduling

  • Next step 2: Pluggable scheduler
    • E.g., workload-specific scheduler
    • E.g., different schedulers for different hardware backends

18

19 of 54

Piecewise CUDA Graphs

  • V0: Single CUDA graph for the entire model

  • Pros: Minimal CPU overheads in model execution

  • Cons: Limited flexibility
    • Static shapes are required
    • No CPU operations are allowed

→ Increased development burden

19

Q

K

V

Attention

O

MLP 0

MLP 1

Q

K

V

20 of 54

Piecewise CUDA Graphs (cont’d)

20

Q

K

V

Attention

O

MLP 0

MLP 1

Q

K

V

CUDA graph N

CUDA graph N-1

PyTorch Eager

Graph split using torch.compile

21 of 54

Piecewise CUDA Graphs (cont’d)

  • V1: Splits the model into pieces
    • Runs the attention op in eager-mode PyTorch
    • Runs other ops with CUDA graphs
      • Easy to capture, since the ops are token-wise
      • Critical to capture the all-reduce op

  • Pros: Maximum freedom in implementing the attention op
    • No restriction on shapes
    • Any CPU operations are allowed

  • Cons: CPU overheads unhidden by CUDA graphs could slow down the model execution
    • Negligible for 8B+ models on H100

21

Q

K

V

Attention

O

MLP 0

MLP 1

Q

K

V

22 of 54

Potential Use Cases of Piecewise CUDA Graphs

Piecewise CUDA graphs will allow vLLM to easily integrate new optimizations such as:

  • Cascade Attention
  • KV cache offloading to CPU memory (#11532)
  • KV cache offloading to disk
  • Sparse KV cache
  • Brand-new attention algorithms
  • . . .

22

23 of 54

V1 Current Status

  • Set VLLM_USE_V1=1 to use the V1 engine
    • Same end-user APIs as V0 (OpenAI server & LLM class)
  • On by default for supported use cases in the upcoming v0.8.0 release.

  • Supported models
    • Decoder-only Transformers (e.g., Llama, Mixtral)
    • Llava-style LMMs (e.g., Qwen2.5-VL, Pixtral)
  • Features
    • Supported: chunked prefills, prefix caching, tensor parallelism, LoRA, spec decoding (n-gram only for now), pipeline parallelism, structured outputs
  • Only supports NVIDIA GPUs for now
    • AMD, TPU, HPU work in progress

23

24 of 54

DeepSeek Support

25 of 54

State of DeepSeek Model Support

4 Major Performance Levers:

  1. MLA: Multi-head Latent Attention (✅ v0, ✅ v1)
  2. MTP: Multi-Token Prediction (✅ v0, v1)
  3. EP: Expert Parallelism (✅v0, ✅ v1)
  4. DP: Attention Data Parallelism (v0, ✅ v1)

vLLM’s unique features:

  • Pipeline Parallelism
  • Integration with SOTA kernels (FlashMLA, FlashInfer, FlashAttention)
  • More spec decode methods (ngrams, draft based, etc)
  • Ecosystem of RLHF integrations, offline inference & serving infra

25

26 of 54

V0 Results of MLA and MTP

Multi-token prediction (k=1): Improve TPOT with low QPS

Multi-head Latent Attention: 2.4x decoding throughput against MHA

26

27 of 54

Q1 Roadmap

28 of 54

vLLM Core

  • Ship a performant and modular V1 architecture
    • 🏎️ V1 on by default
    • Spec decode
    • ⌛Hybrid memory allocator
    • 🏗️Documentation and Design Docs
  • Support large and long context models
    • Sparsity in attention and MoEs
    • 🏎️ Disaggregated prefill support
  • Improved performance in batch mode
    • RLHF
    • ⌛ Long Generation

28

29 of 54

Features

  • Model Support
    • Arbitrary HF model
    • ⌛Alternative checkpoint format
  • Hardware Support
    • 🏎️Blackwell
    • ⌛Improved Tranium/Inferentia, Gaudi
    • 🏎️Productionize and support large scale deployment of vLLM on TPU
    • Out of tree support for IBM Spyre and Ascend
  • Optimizations
    • ⌛ AsyncTP/Flux
    • FlashAttention3
  • Usability
    • ⌛Multi-platform wheels and distributions

29

30 of 54

Ecosystem Projects

  • 🤔 Distributed batch inference
  • ✅ Large scale serving
  • ✅ Prefix aware router
  • 🤔 Multi-modality output

30

Welcome AIBrix!

31 of 54

Thank you sponsors (funding compute!)

31

32 of 54

Our Goal

Build the fastest and

easiest-to-use open-source

LLM inference & serving engine

32

33 of 54

33

Building the fastest and easiest-to-use open-source LLM inference & serving engine!

https://twitter.com/vllm_project

https://opencollective.com/vllm

34 of 54

Model Optimization in vLLM

Robert Shaw

Director of Engineering, Red Hat

vLLM Committer

35 of 54

Why Optimize Your Model?

35

  • Reduce GPU RAM requirements
    • Parameters are most of the RAM usage for reasonable sequence lengths

  • Accelerate linear layers
    • Reduce data movement
    • Leverage low precision tensor cores

  • Negligible impact on model quality with fine-grained quantization

Breakdown of performance of individual decode iterations for Llama-3-8B FP16 and FP8. The x-axis sweeps over batch size 1 to 128. Pink regions are the linear layers

36 of 54

Quantization Support in vLLM

36

Our goal with vLLM

vLLM quantization design

  • Build the fastest and easiest-to-use open-source LLM inference & serving engine
  • Fast inference kernels for serving

  • Ecosystem compatibility

  • Pre-quantized model repository

  • Unified llm-compressor framework for creating quantized models

37 of 54

Kernels

We write custom CUTLASS and Triton kernels for quantized compute

FP8 W8A8 in vLLM is faster than baseline PyTorch implementation

MARLIN kernel in vLLM targets the inference serving regime

38 of 54

Ecosystem Compatible

We work hard to make sure key quantization formats run on vLLM

  • Various quantization formats (AutoAWQ, AutoGPTQ, GGUF) used by the OSS ecosystem

  • Most popular formats are integrated in vLLM can be natively loaded and run with our most optimized kernels
    • Automatic conversion of AutoGPTQ

and AutoAWQ into MARLIN format

39 of 54

Pre Optimized Models

Maintained, pre-optimized models �ready to deploy to production with vLLM

39

Comprehensive Validation

Extensive Selection

Broad Collection

Instinct

GPUs

CPUs

TPUs

Formats

  • W4/8A16
  • W8A8-INT8
  • W8A8-FP8
  • 2:4 sparse

Hardware

Algorithms

  • GPTQ / AWQ
  • SmoothQuant
  • SparseGPT
  • RTN

Llama

Qwen

Mistral

DeepSeek

Gemma

Phi

Molmo

Granite

Nemotron

40 of 54

LLM Compressor

Unified framework for creating compressed models

Comprehensive set of algorithms in unified interface

  • GPTQ, AWQ, SmoothQuant, RTN
  • FP8, INT8, INT4

Seamless integration w/ HF AutoModel

Safetensors-based checkpoint format compatible with vLLM

Large model support via HF accelerate

40

41 of 54

The Case For Activation Quantization

41

42 of 54

Linear Layers

  • Linear layers are GEMMs

  • Large N, K - Llama 3.1 8b sizes:
    • 4096 x 6144 (qkv_proj)
    • 4096 x 4096 (o_proj)
    • 4096 x 28672 (gate_up_proj)
    • 14336 x 4096 (down_proj)

  • Smaller, variable M
    • Equal to number of tokens we are processing in a batch

42

A GEMM computed in a vLLM linear layer.

W is the weight matrix and X is the input activation.

M is the flattened number of tokens being processed.

43 of 54

Drivers of GEMM Latency

Latency is driven by (1) bandwidth from HBM → SRAM and (2) tensor core FLOPs

43

Quantization can enable more ops per second

Quantization reduces data from HBM → SRAM

A100 Memory Bandwidth

H100 Tensor Core FLOPS

44 of 54

Quantization Flavors

We support both weight-only and weight+activation quantization

44

Weight-Only Quantization

Weight+Activation Quantization

  • Both weights and activations are quantized
  • At inference time, compute with the lower precision cores
    • Reduce data from HBM → SRAM
    • Use tensor cores with more FLOPS
  • “Just” the weights are quantized
  • At inference time, “dequantize” the weights and compute with BF/FP16 tensor cores
    • Reduce data from HBM → SRAM

45 of 54

Why Activation Quantization?

Activation quantization accelerates both compute-bound and memory-bound cases

45

  • The number of tokens in the batch “M” drives the “arithmetic intensity” of the GEMM

  • Prefill + high batch decode (Large “M”) are compute-bound
    • Most time is spent “doing FLOPs”

  • Low batch decode (Small “M”) are bandwidth-bound
    • Most time is spend “moving data”

A GEMM computed in a vLLM linear layer.

W is the weight matrix and X is the input activation.

M is the flattened number of tokens being processed.

46 of 54

What About Accuracy?

46

47 of 54

Fine Grained Quantization

Fine-grained quantization enables us to drive significant accuracy

47

X and W are quantized as:

����Rewrite the computation as:���

Inside: A quantized GEMM.

Outside: An elementwise multiplication happening during the epilogue.

torch._scaled_mm

Our CUTLASS kernels

Scales can be per-tensor: Or per-token/channel:

48 of 54

Quantized DeepSeek-R1 Models�Deployment-Ready Reasoning Models

48

Pass@1 score and standard deviation for quantized models on the popular reasoning benchmarks

49 of 54

Join the Open Source AI Movement

Saša Zelenović

Principal Product Marketing Manager, Red Hat

50 of 54

Contribute to Key vLLM Features

  • Comment/review PRs that are interesting to you
  • Join the discussions on RFCs
  • Check out “good first issue” tags
  • Build examples and demos with other tools in the open source AI ecosystem

50

51 of 54

Join Bi-Weekly vLLM Office Hours

  • Happening every other Thursday at 2:00PM ET
  • Hear the bi-weekly vLLM update
  • Give feedback & ask questions
  • Deep dive into cutting-edge developments to accelerate your vLLM inference

51

52 of 54

Let Us Know How We Did Today!

Take 1 minute now to complete our 3-question survey.

52

red.ht/4bwOplj

53 of 54

Q&A

54 of 54

Thank You!