2 of 14

Agenda

1	Why a pure-Python architecture can match C++ inference engines
2	The key design principles behind multi-process runtime scheduling
3	How to hide scheduling overhead behind GPU execution
4	Why CUDA Graph changed Python inference
5	The role of PyTorch custom ops in high-performance serving
6	Practical lessons learned from scaling a Python LLM engine
7	Case Study: TokenSpeed — A Next-Generation LLM Inference Engine

3 of 14

What does modern LLM inference systems look like

GPU kernels execute most tensor computation�
CPU runtime mainly coordinates execution�
Scheduling and synchronization become critical�
Asynchronous execution hides much of the host overhead�
Python increasingly acts as a runtime orchestration layer

Modern inference engines are distributed runtime systems, not traditional Python applications.

4 of 14

Why a pure-Python architecture can match C++ inference engines

Multi-process runtime architecture

CPU-side work such as tokenization and detokenization can be isolated into asynchronous processes, preventing them from blocking GPU scheduling and inference execution. Combined with efficient request routing and continuous batching, the runtime can maintain high GPU utilization despite Python’s GIL.

CPU-GPU overlap execution

The scheduler primarily prepares metadata, batching decisions, and execution plans, while the actual heavy computation runs asynchronously on GPUs. With careful CPU-GPU overlap and minimal synchronization, scheduling overhead can become nearly invisible in the end-to-end pipeline.

Native custom kernels

Performance-critical kernels are ultimately implemented in CUDA, Triton, CUTLASS, vendor libraries, or other native backends. Python acts as the glue layer coordinating these kernels rather than executing the heavy computation itself, allowing a Python architecture to achieve performance close to C++ inference engines.

CUDA Graph��CUDA Graph captures an iteration as a reusable GPU execution graph, replacing many small host-side kernel launches with a single graph replay.

This significantly reduces Python dispatch, CPU launch overhead on the critical path.

5 of 14

Three-process architecture: Separates tokenization and detokenization into asynchronous processes, preventing CPU-side work from blocking GPU scheduling during inference and improving overall GPU utilization and serving throughout.
Efficient router: Provides fine-grained request scheduling and continuous batching for efficient inference execution.

The key design �principles behind multi-�process runtime scheduling

Figure from the LightLLM design (2023)

6 of 14

How to hide scheduling overhead behind GPU execution

While the GPU executes step N, the CPU prepares step N+1.
CPU-GPU overlap hides most scheduling overhead from the critical path

Figure from vLLM v1 design (2024-2025)

7 of 14

CUDA Graph

Capture GPU execution once�
Replay execution efficiently�
Reduce CPU launch overhead�
Reduce Python dispatch overhead

Why CUDA Graph changed Python inference

8 of 14

The role of Pytorch custom ops in �high-performance �serving

Pytorch manages tensor lifecycle, memory ownership, and stream/device semantics.
Custom ops expose CUDA/Triton/vendor Kernels as Pytorch tensor operators
The serving engine calls these ops through Pytorch tensors, while the actual acceleration comes from highly optimized kernels running on vendor GPUs.

Pytorch tensor interface�Tensor lifecycle, memory ownership, �stream/device semantics

Pytorch custom ops�Expose CUDA/Triton/vendor kernels as Pytorch �tensor operators

High-performance kernels�CUDA/Triton/vendor kernels

Vendor GPU hardware�NVIDIA/AMD/etc.

LLM inference engine

9 of 14

Practical lessons learned from scaling a Python LLM engine

What worked well

Fast iteration
Strong ecosystem
Excellent orchestration layer

What became difficult �

State management at scale
Implicit invariants
Resource lifetime coordination
CPU-side orchestration bottlenecks

Python is very well suited for the execution plane, but the complexity of the control plane eventually requires stronger constraints.

In large-scale LLM serving, the hard part is usually not compute — it is managing complex states and resources correctly.
In Python, many important constraints are implicit:

who owns a resource
whether state can still change
whether an operation is still valid
whether rollback is safe

As the system grows, correctness increasingly depends on code review and engineering discipline
State gradually spreads across modules, coupling becomes implicit, and scheduling logic becomes harder to reason about and verify.

10 of 14

Case study: �TokenSpeed — A Next-Generation LLM Inference Engine

11 of 14

TokenSpeed is a high-performance LLM inference engine.

What is TokenSpeed?

TensorRT LLM-level performance
vLLM-level usability
Modular architecture
MIT license, open-source
Github: �https://github.com/lightseekorg/tokenspeed
Blog: https://lightseek.org/blog/lightseek-tokenspeed.html
X: https://x.com/lightseekorg/status/2052048105412141376

12 of 14

The architecture of TokenSpeed

Execution

Python runtime

Python execution plane
Fast iteration
Low engineering friction

Frontend

Tokenspeed-smg

Rust server layer
Everything outside the core GPU inference path

Kernel

TokenSpeed-kernel

High-performance GPU kernels
CUDA/Triton/vendor backends
Modular and independently optimized
Exposed to Python through Pytorch custom ops

Scheduler

TokenSpeed scheduler

C++ control plane
Explicit state machine
Safe resource ownership
Verifiable scheduling logic

13 of 14

Final takeaway

Modern LLM inference engines are no longer just Python applications.

They are heterogeneous systems:

Python execution
Native kernels
Async runtimes
Distributed scheduling

The future is not Python vs C++/Rust.

It is choosing the right abstraction boundary for each layer.

1 of 14