1 of 52

LLM Inference with llm-d

Alexa Griffith · Senior Principal Software Engineer & Forward Deployed Engineer · Red Hat

2 of 52

Where this started

2

3 of 52

Scaling inference

3

4 of 52

Scaling inference

50

5 of 52

LLM workloads

4

6 of 52

Where llm-d fits

5

7 of 52

Where llm-d fits

6

8 of 52

llm-d resources

7

9 of 52

llm-d resources

8

10 of 52

How the Endpoint Picker works

9

11 of 52

How the Endpoint Picker works

10

12 of 52

How the Endpoint Picker works

11

13 of 52

How the Endpoint Picker works

12

14 of 52

Configuring the Endpoint Picker

10

15 of 52

Well-lit paths

11

16 of 52

Intelligent routing

15

17 of 52

Intelligent routing

16

18 of 52

Intelligent routing

17

19 of 52

KV cache offloading

18

20 of 52

KV cache offloading

19

21 of 52

KV cache offloading

20

22 of 52

KV cache offloading

21

23 of 52

Disaggregated Serving

22

24 of 52

Beyond text

25 of 52

Async inference

15

26 of 52

Async inference

16

27 of 52

Sharing Inference with Flow Control

26

28 of 52

Sharing Inference with Flow Control

27

29 of 52

Sharing Inference with Flow Control

28

30 of 52

Sharing Inference with Flow Control

29

31 of 52

Detecting saturation

21

32 of 52

Detecting saturation

22

33 of 52

Detecting saturation

23

34 of 52

Admission limits and queues

24

35 of 52

Admission limits and queues

25

36 of 52

Fairness policies

35

37 of 52

Fairness policies

36

38 of 52

Fairness policies

37

39 of 52

Priority holdback

29

40 of 52

Priority holdback

30

41 of 52

Priority holdback

31

42 of 52

Also tested

31

43 of 52

Overload handling

35

44 of 52

Benchmarking flow control

41

45 of 52

Putting it all together

42

46 of 52

llm-d metrics

Metric

What it shows

Why look

llm_d_epp_flow_control_queue_size

Requests waiting in the Endpoint Picker, by priority

Is work queuing before it reaches vLLM?

llm_d_epp_flow_control_request_queue_duration_seconds

How long requests waited to be dispatched

How much of time to first token is admission wait?

llm_d_epp_flow_control_pool_saturation

The configured saturation signal

Is flow control holding work back?

llm_d_epp_prefix_indexer_hit_ratio

Router's estimate of prefix reuse

Is prefix-aware routing finding warm pods?

vllm:prefix_cache_hits / queries

Actual cache reuse inside vLLM

Does the estimate match real reuse?

vllm:num_requests_waiting / running

Queueing inside the engine

Is work piling up past admission?

vllm:kv_cache_usage_perc, num_preemptions

KV memory pressure

Are requests being preempted and recomputed?

47 of 52

Replay a run

51

Flight Recorder · synthetic demo

48 of 52

Inference decision map

49 of 52

Try it yourself

46

interactive explainer

benchmark repo

50 of 52

Join Merge Forward!

Building a more sustainable open source future together!

community.cncf.io/merge-forward

Learn more

#merge-forward on the CNCF Slack!

Follow us on

Merge Forward (CNCF)

51 of 52

Not your average conference party!

Half PARTY, half INVESTIGATION.

Wed, Nov. 11 @ 7pm

Enlist today

👈 (it's free!)

52 of 52

Thank you!

47