Rachel Teichholz

Avigil Pomerantchik

Zehava Frieman

SwapKV

Naama Dobrovitser

Rachel Ovrani

Problem Statement

• KV-cache must reside entirely in HBM.
• Limits sequence length and model size.
• Especially restrictive on low-memory GPUs.
• HBM is significantly more expensive than DRAM.

Why It Matters

Enables running modern LLMs on cheaper hardware by shifting most KV-cache to DRAM, reducing GPU costs and making long-sequence, large-model workloads more accessible

Project Goal

• SwapKV virtualizes the KV-cache across HBM and DRAM.
• Only active layers stay in HBM. Other data is pulled from DRAM on demand.

Example

85GB

1K tokens → 164 MB

KV-cache size

H100 GPU

HBM capacity

Llama-3-70b-fp8

80

Model layers

80GB

5GB ~= 30K

Tokens

Model consumption

80 → 2

Layers

30K → 1.2M

Increase seq length

Input length analysis

Performance analysis

≈ 56GB/s

PCIe bandwidth

≈ 200ms

TPOT

≈ 200GB/s

CPU DDR bandwidth

56*0.2GB

200ms transfer limit

0.164GB

~= 70K tokens

Reduce block size

Must-Have Objectives

Constraints

Limited hardware

Tight

timeline

Map memory

Manage DRAM allocation

Load KV-cache on time

HBM↔DRAM movement

Ensure correctness and reliability

Lack of relevant knowledge

Enable Cudagraph

Performance hit is under 30%

Make the feature flexible and general, e.g. automatically adjust according to resources

Prepare the code for contributing to vLLM community (PR/RFC)

Stretched goals

02

01

03

LEARNING APPROACH

Built ramp-up for knowledge.

Held daily sync meetings to share insights and align understanding.

Divided the topics among team members for efficient learning.

Concept Exploration & Code Mapping

​

Studied key LLM and vLLM concepts: KV-cache, paged attention, prefill/decode, APC, batching, block allocation

Local Setup

Core Concepts

01

02

Mapped each concept to its implementation and understood the internal model flow

Code Mapping

03

Documentation

04

Installed vLLM locally, ran the LLM, and debugged Python & C++

Recorded findings in spreadsheets and created diagrams to visualize concept and memory flow.

Teamwork

SnakeViz & Nsight

How Connector integrated in vLLM :

01

03

07

02

Reduce block size in block allocation

04

05

06

Map physical memory to virtual space

Evacuate irrelevant KV-cache layers from HBM

Load KV-cache from DRAM in time for computation

Synchronize computation with memory loading

Efficient DRAM allocation and management

Implement HBM↔DRAM data movement

Preliminary Design

Connector related research & development

Reduce block allocation

by Rachel Ovrani

Layer Virtualization

max_model_len = 131,072 requires 1.5GB, which is larger than the available memory 1GB

Turn on layer virtualization

The request was successful in loading and generating tokens.

1. The request consumes 0.125GB instead of 1.5GB of memory

2. The layer size is equal to 0.5GB (available memory 1GB, and configuration of 2 physical layers)

3. The number of requests that can run concurrently is 8

The model ran successfully.

num_physical_layers = 2

524,289 KB * 2 ≈ 1GB

num_layers = 24

43,681 KB * 24 ≈ 1GB

Before

virtualization

After

virtualization

Async connector

by Avigail Pomeranchik

DRAM Allocation

by Rachel Teichholz

DRAM Memory Pool

Creating a memory block in DRAM using mmap.

mmap

Pinned Memory

DRAM allocated and locked by the OS – cannot be moved or swapped.

Memory Pool Structure

DRAM Pool

0

max size

4096

8192

12288

DATA BLOCK A

DATA BLOCK C

DATA BLOCK B

...

TensorRef A

offset: 0

shape: (16, 128)

freed: false

TensorRef B

TensorRef C

offset: 4096

shape: (16, 128)

freed: false

offset: 12288

shape: (32, 128)

freed: false

mapping table

key: tensor name

value: tensor ref

free offset list

(start_offset, length in bytes)

​

Data Movement

by Zehava Frieman

Build Dram-Manager

​

Update connector

Old SSD connector

stored KV blocks as files on SSD

used the file system as a mapping layer

​

New DRAM connector

copies KV blocks directly between HBM and a DRAM tensor pool

(no filesystem, pure memory copies).

​

DRAM Eviction

by Naama Dobrovitser

DRAM’s space is limited

There is no cleanup mechanism

Layers are moved to DRAM, but never evicted.

Result: DRAM fills up with old layers

The Core Problem

DRAM becomes overloaded with KV tensors

Initial Solution: LRU Eviction

Free DRAM by removing the least-recently-used KV tensors

head ->

kv tensor

kv tensor

kv tensor

kv tensor

Problem: Layer-Based LRU Cannot Work

Each generation keeps one KV tensor per layer.

​

​

All tokens of the current generation rely on all layers.

​

​

​

​

There is no such thing as an “old layer” to evict.

​

​

​

Gen 1

Gen 2

Solution: Evict Only Past Sequences

Evict only KV tensors of old clients

Active generation is fully preserved

Safe and predictable DRAM cleanup

Active Gen

DRAM

Demo

Technical &

Personal

Lessons

Structured research

Improved research skills + breaking down complex problems

System insights

Understanding LLM & vLLM internals + navigating complex code

Technical troubleshooting

Installing vLLM & solving issues

Codebase mastery

Confidence in large codebase

Team alignment

Strengthened teamwork + structured decision-making

Performance analysis

Profiling tools & GPU work

Complete SwapKV Flow

Extend the prototype to include full CPU↔GPU KV movement and validation logic.

CUDAGraph Compatibility

Verify that graph capture remains stable and unaffected when SwapKV is enabled.

01

02

Performance Optimization

Profile end-to-end behavior, identify bottlenecks, evaluate alternative mechanisms, and optimize to stay within the <30% overhead target.

Upstream Readiness

Refactor, clean up, and document the implementation for a future vLLM PR/RFC

03

04

Next Steps

Next Steps

(given additional time to productize the concept)

Thank you!