LLM Inference with llm-d
Alexa Griffith · Senior Principal Software Engineer & Forward Deployed Engineer · Red Hat
Where this started
2
Scaling inference
3
Scaling inference
50
LLM workloads
4
Where llm-d fits
5
Where llm-d fits
6
llm-d resources
7
llm-d resources
8
How the Endpoint Picker works
9
How the Endpoint Picker works
10
How the Endpoint Picker works
11
How the Endpoint Picker works
12
Configuring the Endpoint Picker
10
Well-lit paths
11
Intelligent routing
15
Intelligent routing
16
Intelligent routing
17
KV cache offloading
18
KV cache offloading
19
KV cache offloading
20
KV cache offloading
21
Disaggregated Serving
22
Beyond text
Async inference
15
Async inference
16
Sharing Inference with Flow Control
26
Sharing Inference with Flow Control
27
Sharing Inference with Flow Control
28
Sharing Inference with Flow Control
29
Detecting saturation
21
Detecting saturation
22
Detecting saturation
23
Admission limits and queues
24
Admission limits and queues
25
Fairness policies
35
Fairness policies
36
Fairness policies
37
Priority holdback
29
Priority holdback
30
Priority holdback
31
Also tested
31
Overload handling
35
Benchmarking flow control
41
Putting it all together
42
llm-d metrics
Metric
What it shows
Why look
llm_d_epp_flow_control_queue_size
Requests waiting in the Endpoint Picker, by priority
Is work queuing before it reaches vLLM?
llm_d_epp_flow_control_request_queue_duration_seconds
How long requests waited to be dispatched
How much of time to first token is admission wait?
llm_d_epp_flow_control_pool_saturation
The configured saturation signal
Is flow control holding work back?
llm_d_epp_prefix_indexer_hit_ratio
Router's estimate of prefix reuse
Is prefix-aware routing finding warm pods?
vllm:prefix_cache_hits / queries
Actual cache reuse inside vLLM
Does the estimate match real reuse?
vllm:num_requests_waiting / running
Queueing inside the engine
Is work piling up past admission?
vllm:kv_cache_usage_perc, num_preemptions
KV memory pressure
Are requests being preempted and recomputed?
Replay a run
51
Flight Recorder · synthetic demo
Inference decision map
Try it yourself
46
interactive explainer
benchmark repo
Join Merge Forward!
Building a more sustainable open source future together!
community.cncf.io/merge-forward
Learn more
#merge-forward on the CNCF Slack!
Follow us on
Merge Forward (CNCF)
Not your average conference party!
Half PARTY, half INVESTIGATION.
Wed, Nov. 11 @ 7pm
Enlist today
👈 (it's free!)
Thank you!
47