vLLM Office Hours #34
Huamin Chen
Distinguished Engineer, Red Hat
Saša Zelenović
Principal PMM, Red Hat
Special Topic: AI-Powered vLLM Semantic Router
Michael Goin
Principal Software Engineer, Red Hat
vLLM Committer
1
Update confidential designator here
Version number here V00000
What’s new in the past two weeks?
Upcoming vLLM Office Hours Sessions
Register for all sessions here.
View previous recordings here.
vLLM Project Update
2
Update confidential designator here
Version number here V00000
Join our [Virtual] Red Hat AI Events
Your Path to Enterprise-Ready AI
Red Hat AI: Day of Learning�October 16, 2025
Red Hat AI: What’s New & What’s Next �October 14, 2025
3
red.ht/ai-day-of-learning
red.ht/whats-new-whats-next
Sign up →
Sign up →
Update confidential designator here
Version number here V00000
Accelerate You AI with AMD Resources
Free GPU hours for Office Hours attendees
4
Update confidential designator here
Version number here V00000
What is vLLM?
The fastest growing de-facto standard in open source model serving
5
Broad Model Support (>100 arches)
DeepSeek
Wide Hardware Support
CUDA
ROCm
Gaudi/XPU
TPU
Neuron
CPU
$ uv pip install vllm --torch-backend=auto
$ vllm serve deepseek-ai/DeepSeek-V3.1 -tp 8
Llama
GPT-OSS
Qwen
Gemma
Mistral
Granite
Ascend
Phi
Update confidential designator here
Version number here V00000
DeepSeek-V3.2: How does DeepSeek Sparse Attention (DSA) work?
6
Day 0 support for CUDA, Ascend, MLU!
Update confidential designator here
Version number here V00000
PipelineRL: In-flight weight updates during RL
7
Update confidential designator here
Version number here V00000
vLLM Encoder Support via Transformers
8
Update confidential designator here
Version number here V00000
bota: Dota 2 agent powered by GPT-OSS-20B and vLLM
9
🏆 OpenAI Open Model Hackathon Winner - Wildcard category
Update confidential designator here
Version number here V00000
What’s new in vLLM v0.11.0
🎉 538 commits from 207 contributors, including 65 new contributors! 🎉
Model Support
Engine Core
10
Update confidential designator here
Version number here V00000
What’s new in vLLM v0.11.0
🎉 538 commits from 207 contributors, including 65 new contributors! 🎉
Hardware & Performance
Quantization
Large Scale Serving & Performance
11
Update confidential designator here
Version number here V00000
What’s new in vLLM v0.11.0
🎉 538 commits from 207 contributors, including 65 new contributors! 🎉
API & Frontend
Dependencies
V0 Deprecation
12
Update confidential designator here
Version number here V00000
AI-Powered vLLM Semantic Router
Huamin Chen
Distinguished Engineer, Red Hat
Today’s special topic:
13
Problem Statement
Based on MMLU-Pro benchmark, Llama-3.1-8B is behind Qwen2-7B in Math, but excels in Chemistry.
14
curl -X POST http://vllm-semantic-router/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Llama-3.1-8B-Instruct",
"messages": [
{
"role": "user",
"content": "what is the derivative of x^2"
}
]}'
Update confidential designator here
Version number here V00000
15
curl -X POST http://vllm-semantic-router/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-oss-120b",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant",
"role": "user",
"content": "what is the 100th fibonacci number"
}
],
max_tokens: 100,
reasoning_effort: "high",
}'
Update confidential designator here
Version number here V00000
16
GenAI Development Like Assembly Language Programming
GenAI developers must master each model's specifics:
Semantic Router as High-Level Language
Update confidential designator here
Version number here V00000
Semantic Router: Intelligent Model Routing System
Core Value
Core Functionalities
Router Reasoning Bench
Goal:
Feature:
Disclaimer: Work in progress, final numbers may vary
Disclaimer: Work in progress, final numbers may vary
Ecosystem Support
Demo
Get involved with the vLLM Community
Contribute to key vLLM features�Comment and review PRs that are interesting to you. Join the discussion on RFCs. Check out “good first issue” tags.
Give Us Feedback�We’ll email you today’s recording as soon as it’s ready. Respond and tell us what we are doing right and what we can do better with vLLM office hours. Or comment on this slide!
Join vLLM Developer Slack�Ask questions and engage with us via Slack. Join here.
Join Red Hat’s vLLM Mission�Red Hat wants to bring open-source LLMs and vLLM to every enterprise on the planet. We are looking for vLLM Engineers to help us accomplish our mission. Apply here.
26
Update confidential designator here
Version number here V00000
Thank you, and see you in two weeks!
Huamin Chen�Distinguished Engineer, Red Hat
Saša Zelenović�Principal Product Marketing Manager, Red Hat
Michael Goin�Principal Software Engineer, Red Hat�vLLM Committer
27
Update confidential designator here
Version number here V00000