Limitations of PEBS for

Tracking Main Memory Requests

Yuhong Zhong

Columbia University

​

📨 yz@cs.columbia.edu

🌐 https://cs.columbia.edu/~yz

Two-Level Memory (2LM)

Two-level memory (2LM) is a type of memory tiering scheme where the memory controller decides which cache-lines stay in the fast memory tier

Fast Memory Tier (e.g., Local DRAM)

Slow Memory Tier (e.g., PMEM/CXL)

Hot Cacheline

…

Hot Cacheline

Hot Cacheline

Cold Cacheline

Cold Cacheline

Cold Cacheline

…

Cacheline Demotion

Cacheline Promotion

Decide by

Memory Controller

Two-Level Memory (2LM)

Unlike software memory tiering where data can only be migrated at the page granularity, 2LM can migrate data at the 64 B cache-line granularity

​

Examples:

• DRAM + Optane Persistent Memory (PMEM) in Memory Mode
• Local Memory + CXL-Attached Memory (?)

In Optane Memory Mode, DRAM becomes a direct-mapped cache for PMEM

2LM Misses Are Critical to Performance

Due to its direct-mapped associativity, 2LM systems may experience severe conflict misses if two or more hot PMEM lines are mapped to the same DRAM line

​

​

​

​

​

2LM misses can degrade application performance significantly (up to 4x). Tracking them is important to diagnose and improve the performance of 2LM

PMEM Line

PMEM Line

PMEM Line

PMEM Line

DRAM Line

4:1 Mapping

🔥

🔥

PEBS Can Track 2LM Misses

Available mechanisms to track 2LM misses:

• Access bit in the page table entry (PTE) ^[1]
• PTE poisoning ^[2]
• PEBS (Processor Event-Based Sampling) ^[3]

However, only the main memory requests can cause 2LM misses. Memory requests that hit the CPU cache are irrelevant to 2LM

PEBS can track only main memory requests

[1] DAMON: Data Access MONitor https://www.kernel.org/doc/html/v5.17/vm/damon/index.html

[2] Thermostat: Application-transparent Page Management for Two-tiered Main Memory, ASPLOS 2017

[3] HeMem: Scalable Tiered Memory Management for Big Data Applications and Real NVM, SOSP 2021

Sample 2LM Misses Using PEBS

PEBS (Processor Event-Based Sampling) is a hardware sampling mechanism available on Intel CPUs. It can be used to sample the data linear addresses that trigger certain events.

Some relevant events:

1. A retired load instruction missed the L3 cache (MEM_LOAD_RETIRED.L3_MISS)
2. A retired load instruction missed the local memory (MEM_LOAD_RETIRED.LOCAL_PMM)
3. A store instruction retired (MEM_INST_RETIRED.ALL_STORES)

​

Therefore, PEBS can be used to track main memory requests (#1) or just 2LM misses (#2)

AMD has a similar feature called IBS (Instruction-Based Sampling)

Count Per-Page 2LM Misses Using PEBS

We can aggregate PEBS samples, which include the data linear addresses, to get per-page LLC miss counts and 2LM miss counts

​

​

​

​

These counters can help us understand where the most misses happened and how to reduce 2LM misses (increase DRAM size? resolve conflicts?)

2LM Miss Count: 3

2LM Miss Count: 1

2LM Miss Count: 7

2LM Miss Count: 88

Page 0:

Page 1:

Page 2:

Page 3:

Page 4:

2LM Miss Count: 89

Identify the pages that have high 2LM miss counts

Collect per-page miss counts

PEBS Cannot Cover All the Main Memory Requests

Relevant PEBS events:

• A retired load instruction missed the L3 cache (MEM_LOAD_RETIRED.L3_MISS)
• A retired load instruction missed the local memory (MEM_LOAD_RETIRED.LOCAL_PMM)
• A store instruction retired (MEM_INST_RETIRED.ALL_STORES)

PEBS can only sample the main memory requests that are directly triggered by memory instructions. However, there are other types of main memory requests that are generated by the CPU cache:

• CPU cache write-back
• Read-for-ownership (RFO)
• CPU cache prefetch

These main memory requests cannot be sampled by PEBS

​

Experiments: 2LM Misses In the Wild

Workload: SPEC CPU 2017, GAPBS (a graph processing benchmark)

Memory: DRAM + Optane PMEM in Memory Mode

DRAM to PMEM Ratio: 1:2

Many Workloads Have Skewed 2LM Miss Distributions

Many realistic workloads have very skewed 2LM miss distributions, which indicate that they suffer from 2LM conflicts

SPEC CPU 2017 (602.gcc_s)

5% of the pages contribute to 82% of the 2LM misses

SPEC CPU 2017 (605.mcf_s)

5% of the pages contribute to 34% of the 2LM misses

Resolve Conflicts by Exchanging Pages

How to resolve the 2LM conflicts identified by PEBS?

Exchange the pages with high 2LM miss counts with some other pages randomly

PMEM Page

PMEM Page

DRAM Page

PMEM Page

PMEM Page

DRAM Page

🔥

🔥

❄️

❄️

Exchange!

Resolve Conflicts by Exchanging Pages

How to resolve the 2LM conflicts identified by PEBS?

Exchange the pages with high 2LM miss counts with some other pages randomly

Implemented a page exchange syscall in Linux

PTE 1

PTE 2

Page 1

Page 2

PTE 1

PTE 2

Page 1

Page 2

Page Exchange

Page Table

Page Table

Conflicts Identified by PEBS Can Be Resolved By Exchanging Pages

Workload: 602.gcc_s in SPEC CPU 2017

Completion Time:

• 100% local memory: 281.02 s
• 50% local memory: 334.78 s
• 50% local memory + page exchange: 309.53 s

​

Exchanging 15% of the pages with the highest miss counts with some random pages can reduce the slowdown by ~50%

​

PEBS Does Not Always Work

For 602.gcc_s, we can detect conflicts using PEBS and exchange pages accordingly to improve performance significantly

​

However, we find that PEBS does not always work. For some workloads, even though PEBS indicates that almost all the misses can be resolved by page exchanges, we cannot get a significant performance improvement

This is because PEBS does not track certain types of main memory requests (will discuss more later)

A Failure Case of PEBS

Workload: 607.cactuBSSN_s in SPEC CPU 2017

​

• 99% of the LLC misses and DRAM misses happened in the highlighted hot region
• The size of the hot region is less than the half of the resident set size
• It is possible to resolve all the misses!

​

​

LLC Miss Distribution

DRAM Miss Distribution

Misses Still Exist Even If the Hot Region Is Conflict-Free

However, we find that even if we exchanges pages to make the hot region conflict free, the performance is still far away from optimal

Completion Time:

• 100% local memory: 390.84 s
• 50% local memory: 576.26 s
• 50% local memory (hot conflict-free): 537.39 s

The hot region still has a lot of 2LM misses even if it is conflict-free

Therefore, the cold region must have many 2LM misses that are invisible to PEBS

PEBS Cannot Cover All the Main Memory Requests

Main Memory Requests

Read Requests

Write Requests

(e.g., CPU Cache Write-Backs)

Regular Read Requests

Read-For-Ownership (RFO) Requests

Demand Regular Read Requests

Prefetch Regular Read Requests

PEBS can only sample demand regular read requests (i.e. load instructions that miss the CPU cache)

PEBS Cannot Cover All the Main Memory Requests

Main Memory Requests

Read Requests

Write Requests

(e.g., CPU Cache Write-Backs)

Regular Read Requests

Read-For-Ownership (RFO) Requests

Demand Regular Read Requests

Prefetch Regular Read Requests

PEBS can only sample demand regular read requests (i.e. load instructions that miss the CPU cache)

For 607.cactuBSSN_s:

(4490 GB)

(1201 GB)

(5691 GB)

(561 GB)

(3929 GB)

(2805 GB)

(969 GB)

PEBS samples only represent 2805 GB / 5691 GB = 49% of the main memory traffic!

Although the cold region only has 73 GB demand regular read requests, it has 579 GB other types of requests

Conclusions

PEBS is insufficient to track main memory requests and 2LM misses because it is oblivious to the main memory requests that are not directly associated with memory instructions (e.g., CPU cache write-backs, read-for-ownership (RFO), and prefetch)

For software memory tiering (i.e. using software to manage page placement among tiers), using PEBS for hotness tracking has similar issues. The pages with heavy CPU cache write-backs/RFO requests/prefetch requests may seem “cold” to PEBS