ICGMM: CXL-enabled Memory Expansion with Intelligent Caching Using Gaussian Mixture Model

Hanqiu Chen^1§, Yitu Wang^2§, Luis Vitorio Cargnini³, Mohammadreza Soltaniyeh³, Dongyang Li³, Gongjin Sun³, Pradeep Subedi³, Andrew Chang³, Yiran Chen² and Cong Hao¹

¹Georgia Institute of Technology, ²Duke University, and ³Samsung Semiconductor, Inc.

^§ Equal contribution

Email: hanqiu.chen@gatech.edu

Background

(Hennessy, John L., and David A. Patterson. Computer architecture: a quantitative approach)

Memory wall

Background

(Hennessy, John L., and David A. Patterson. Computer architecture: a quantitative approach)

Memory wall

CXL-enabled memory expansion

Background

(Hennessy, John L., and David A. Patterson. Computer architecture: a quantitative approach)

Memory wall

• DRAM acts as cache for SSD
• FPGA acts as cache controller

CXL-enabled memory expansion

DRAM Cache for SSD: Challenges

DRAM data granularity

SSD data granularity

4 KB

Cache pollution & eviction

Granularity mismatch

DRAM Cache for SSD: Challenges

DRAM data granularity

SSD data granularity

4 KB

Cache pollution & eviction

Granularity mismatch

DRAM

FPGA

HW-inefficient cache controller

DRAM Cache for SSD: Challenges

DRAM data granularity

SSD data granularity

4 KB

Cache pollution & eviction

Granularity mismatch

DRAM

FPGA

Large cache miss penalty

HW-inefficient cache controller

DRAM Cache for SSD: Approaches

DRAM

FPGA

DRAM data granularity

SSD data granularity

4 KB

Lightweight HW cache controller

Host CPU

x

Smart caching

If not accessed frequently, do not cache in DRAM but directly to host

DRAM Cache for SSD: Approaches

DRAM

FPGA

Lightweight HW cache controller

Host CPU

x

Smart caching

If not accessed frequently, do not cache in DRAM but directly to host

Algorithm

2D Gaussian Mixture Model (GMM)

• Light-weight
• Captures spatial and temporal memory access patterns

2D GMM Motivations: Spatial Patterns

Spatial memory access profiling results

dlrm

parsec

sysbench

Memory access spatial distribution can be fitted with a combination of Gaussian functions

2D GMM Motivations: Temporal Patterns

Temporal memory access profiling results

Uneven access frequency within a specific range of addresses

dlrm

parsec

sysbench

2D GMM: Overview

Spatial

patterns

Temporal

patterns

2D GMM: Overview

Spatial

patterns

Temporal

patterns

2D GMM

2D GMM: Overview

Spatial

patterns

Temporal

patterns

2D GMM

Trained using Expectation-Maximization (EM) algorithm

2D GMM: Overview

Spatial

patterns

Temporal

patterns

2D GMM

Trained using Expectation-Maximization (EM) algorithm

Predict future memory access frequency with:

• Physical address
• Time stamp

Trace processing

An example of trace sequence with timestamp annotated

We empirically set len_window = 32,

len_access_shot = 10,000

32

Help capture temporal locality better for long trace

Trace processing

An example of trace sequence with timestamp annotated

We empirically set len_window = 32,

len_access_shot = 10,000

32

An access shot

Help capture temporal locality better for long trace

Trace processing

An example of trace sequence with timestamp annotated

We empirically set len_window = 32,

len_access_shot = 10,000

32

A time window

Help capture temporal locality better for long trace

Intelligent Caching with 2D GMM

[page index, time stamp]

​

Memory request

​

DRAM cache

Host

Intelligent Caching with 2D GMM

[page index, time stamp]

​

Memory request

​

DRAM cache

Host

Hit

Directly send data back to host from DRAM cache

If cache hit

Intelligent Caching with 2D GMM

[page index, time stamp]

​

Memory request

​

DRAM cache

Host

Hit

GMM engine

Miss

GMM score

Directly send data back to host from DRAM cache

If cache hit

SSD

Intelligent Caching with 2D GMM

[page index, time stamp]

​

Memory request

​

DRAM cache

Host

Hit

GMM engine

Miss

GMM score

Directly send data back to host from DRAM cache

If cache hit

SSD

GMM score indicates access frequency

larger score -> higher frequency

Intelligent Caching with 2D GMM

[page index, time stamp]

​

Memory request

​

DRAM cache

Host

Hit

GMM engine

Miss

GMM score

Score < Threshold

Directly send data back to host from DRAM cache

If cache hit

Directly send data from SSD to host

If cache miss

SSD

❶

❶

GMM score indicates access frequency

larger score -> higher frequency

S < T

Intelligent Caching with 2D GMM

[page index, time stamp]

​

Memory request

​

DRAM cache

Host

Hit

GMM engine

Miss

GMM score

Score < Threshold

Directly send data back to host from DRAM cache

If cache hit

Directly send data from SSD to host

If cache miss

Cache SSD data into DRAM

SSD

❶

❷

❶

❷

GMM score indicates access frequency

larger score -> higher frequency

S < T

Intelligent Eviction with 2D GMM

[page index, time stamp]

​

Memory request

​

DRAM cache

Host

Hit

GMM engine

Miss

GMM score

Score < Threshold

SSD

❶

❷

…

GMM score

Data

GMM score is also cached in the DRAM

Eviction when DRAM cache is full

4KB (page size)

Intelligent Eviction with 2D GMM

[page index, time stamp]

​

Memory request

​

DRAM cache

Host

Hit

GMM engine

Miss

GMM score

Score < Threshold

SSD

❶

❷

…

GMM score

to be evicted

Find the cache block with lowest GMM score to evict

Data

GMM score is also cached in the DRAM

Eviction when DRAM cache is full

Hardware Prototype

ICGMM is prototyped on FPGA, using FPGA DRAM as cache

Opens up a new opportunity of using SmartSSD as a CXL-enabled memory expansion device

Potential application

Hardware Prototype

ICGMM is prototyped on FPGA, using FPGA DRAM as cache

Complex run-time dynamic control

Hardware System Design

An overview of ICGMM design on FPGA (Xilinx Alev0 U50)

ICGMM is prototyped as a dataflow architecture with FIFO interfaces between different modules

Three modules

• Cache policy engine
• Cache control engine
• Signal controller

Hardware System Design

An overview of ICGMM design on FPGA (Xilinx Alev0 U50)

ICGMM is prototyped as a dataflow architecture with FIFO interfaces between different modules

Three modules

• Cache policy engine
• Cache control engine
• Signal controller

Module 1: Cache Policy Engine

Trace FIFO

PA

Time

Rsp FIFO

Ctrl.

Cache policy engine

GMM

PE

Control module

Score

Weight buffer

One-time loading from HBM before kernel starts

Control

• Independent and data-driven module
• Control module activates and close GMM PE based on Ctrl. signal

Module 1: Cache Policy Engine

Trace FIFO

PA

Time

Rsp FIFO

Ctrl.

Cache policy engine

GMM

PE

Control module

Score

Weight buffer

One-time loading from HBM before kernel starts

Control

• Independent and data-driven module
• Control module activates and close GMM PE based on Ctrl. signal

Highly parallelized GMM PE

• Deep computation pipeline with II = 1 for GMM score calculation
• Shift register to resolve data dependency in GMM score accumulation

Hardware System Design

An overview of ICGMM design on FPGA (Xilinx Alev0 U50)

ICGMM is prototyped as a dataflow architecture with FIFO interfaces between different modules

Three modules

• Cache policy engine
• Cache control engine
• Signal controller

Module 2: Cache Control Engine

Cache management module

• Check cache hit/miss and control cache loading and eviction
• Only cache tags and GMM scores are transferred to BRAM and partitioned for quick hit/miss determination

Module 2: Cache Control Engine

Cache management module

• Check cache hit/miss and control cache loading and eviction
• Only cache tags and GMM scores are transferred to BRAM and partitioned for quick hit/miss determination

SSD access latency emulator

• Stalls the dataflow in the cache control engine for a set duration to emulate SSD response time upon cache miss

Hardware System Design

An overview of ICGMM design on FPGA (Xilinx Alev0 U50)

ICGMM is prototyped as a dataflow architecture with FIFO interfaces between different modules

Three modules

• Cache policy engine
• Cache control engine
• Signal controller

Module 3: Signal Controller

Controller

Manage control signals on FPGA

Trace FIFO

PA

Time

Rsp FIFO

Ctrl.

Score

HBM bank 1

Trace FIFO

[R/W, PA, Time]

Trace FIFO

R/W

PA

Score FIFO

Score

Rsp FIFO

Hit/miss

To cache control engine

To cache policy engine

• The signal controller acts like a bridge between cache control engine and cache policy engine

​

• Generates control signals, manages data flow between HBM, on-board buffers, and different modules and blocks

Experiment Results: Cache Miss Rate

0.32% ~ 6.14% reduction in cache miss rate

Compared to Least Recently Used (LRU) cache policy, GMM can achieve

The policy we choose is marked in shadow

The cache miss rate comparison

Setting: cach size = 64MB, block size = 4KB, associativity = 8. We use 256 Gaussian functions.

Experiment Results: SSD access latency

Average SSD access latency

Compared to Least Recently Used (LRU) cache policy, GMM can achieve

16.23% ~ 39.14% reduction in SSD access latency

Setting: read latency = 75𝜇𝑠 and write latency = 900𝜇𝑠

Experiment Results: Hardware Performance

Resource utilization and latency comparison between a lightweight LSTM and GMM for cache policy engine

Compared to LSTM cache controller, GMM has following benefits:

More than 10,000x latency reduction with only 2% BRAM utilization

Completely hidden GMM overhead with dataflow streaming

Experiment Results: Hardware Performance

Resource utilization and latency comparison between a lightweight LSTM and GMM for cache policy engine

Compared to LSTM cache controller, GMM has following benefits:

More than 10,000x latency reduction with only 2% BRAM utilization

Completely hidden GMM overhead with dataflow streaming

The lightweight LSTM has low cache hit rate (<50%)

Summary & Thanks

• ICGMM is a hardware-managed DRAM caching system for CXL-enabled memory expansion prototyped on FPGA using Gaussian Mixture Model as cache policy engine

​

• Algorithm – GMM-based cache policy for intelligent caching and eviction

​

• Hardware – Optimized GMM hardware design with dataflow architecture

​

• System – Hardware-managed DRAM cache system design

​

• Performance – Improved DRAM cache hit rate and reduced average SSD access latency

​

​