Mitigating the Memory Bottleneck with�Machine Learning-Driven and Data-Aware�Microarchitectural Techniques
Rahul Bera
Doctoral Examination
01.10.2025
Advisor:
Onur Mutlu (ETH Zurich)�
Co-Examiners:
Aamer Jaleel (Nvidia Research)
Boris Grot (University of Edinburgh)
Chris Wilkerson (Intel)
Daniel Jiménez (Texas A&M University)
Pradip Bose (IBM T. J. Watson Research Center)
Modern Workloads Exhibit Massive Data Footprints
2
High-performance computing
Graph analytics
Large-scale artificial intelligence
Genome analysis
Modern Workloads Exhibit Massive Data Footprints
3
High-performance computing
Graph analytics
Large-scale artificial intelligence
Genome read mapping
Memory is (and likely will be)
the key performance & energy bottleneck
Massive data footprints
overwhelm state-of-the-art memory systems
Numerous Microarchitectural Techniques �to Address the Memory Bottleneck
4
Cache management policies
Dead block prediction
Cache bypassing
Hit/miss prediction
Software prefetching
Spatial prefetching
Temporal prefetching
Thread-based precomputation
Runahead execution
Out-of-order execution
Coarse-grained multithreading
Simultaneous multithreading
Load value prediction
Memory renaming
Memory dependence prediction
Load address speculation
Load address precomputation
Memoization
Dead instruction elimination
Key Observation
5
Decisions they make online are often
agnostic to the data they observe
Even though microarchitectural techniques
observe a large amount of
Application data
(e.g., program counter value, memory addresses, memory values)
System data
(e.g., bandwidth usage,
cache utilization and pollution)
Key Observation
6
Limited adaptation to observed data
Make decisions based on rigid, and often myopic, human-crafted heuristics, disregarding the large volume of data present at runtime
Insufficient exploitation of data characteristics
Do not fully exploit or often disregard diverse properties of
application data (e.g., criticality, value locality, compressibility)
Decisions they make online are often
agnostic to the data they observe
Data-agnostic decision making
limits effectiveness
Thesis Statement
7
Adapt its policies by continuously learning
from application data and system-generated data
Tailor its decisions by exploiting characteristics
of application data
By enabling microarchitecture to
We can significantly improve performance and
energy efficiency of state-of-the-art processors
Our Approach
8
Adapt its policies by continuously learning
from application data and system-generated data
Tailor its decisions by exploiting characteristics
of the application data
Exploit lightweight and practical
machine learning methods
Study and exploit
underexplored data characteristics
9
Fetch & Decode
Execute
Access Memory
Commit
Key Contributions
10
Fetch & Decode
Execute
Access Memory
Commit
Hermes [MICRO’22]
Off-chip prediction (OCP)
using perceptron learning
Athena [under review]
Prefetcher-OCP coordination
using reinforcement learning
Constable [ISCA’24]
Safely eliminating load instruction execution
by exploiting load address-value stability
Pythia [MICRO’21]
Hardware data prefetching
using reinforcement learning
Key Contributions
11
Fetch & Decode
Execute
Access Memory
Commit
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Hermes [MICRO’22]
Off-chip prediction (OCP)
using perceptron learning
Athena [under review]
Prefetcher-OCP coordination
using reinforcement learning
Constable [ISCA’24]
Safely eliminating load instruction execution
by exploiting load address-value stability
Pythia [MICRO’21]
Hardware data prefetching
using reinforcement learning
What is a Hardware Data Prefetcher?
12
Program feature 🡪 access pattern
When the program feature repeats, the prefetcher anticipates
the repetition of the same access pattern and predicts future addresses
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
(e.g., load PC)
Three Limitations in Prior Prefetchers
13
Reliance on a single, fixed program feature
for prefetch prediction
Rigidity limits dynamic adaptation in workloads where
the selected feature provides poor correlation with access pattern
Lack of inherent system awareness
Neglecting a prefetcher’s undesirable effects on the system (e.g., bandwidth usage, cache pollution) often harms performance
Lack of in-silicon customizability
Rigid hardware structure provides no customizability to select
new program features and/or prefetching objective at runtime
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Our Goal
14
Design a holistic prefetching framework that can
Learn to prefetch using multiple program features
and inherent system-level feedback information
Be easily customized in silicon to change
program feature and/or prefetching objective on-the-fly
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
15
Key Idea: Formulate prefetching as reinforcement learning
Pythia
Processor and Memory System
RL agent
Environment
Autonomously learn
what to prefetch
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
16
Key Idea: Formulate prefetching as reinforcement learning
Pythia
Processor and Memory System
State
Action
Reward
Program features
(e.g., PC, page offset, cacheline delta)
Prefetch offset
(i.e., delta between the prefetch and demand cacheline)
Prefetch quality
considering system-level feedback
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
17
Two-part evaluation
Hardware prototyping
using Chisel
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Trace-driven simulation
using ChampSim
18
Trace-driven simulation
using ChampSim
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Wide range of prefetchers
Wide range of system-configurations
Wide range of workloads
19
Trace-driven simulation
using ChampSim
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Wide range of prefetchers
Wide range of system-configurations
Wide range of workloads
Open-sourced and artifact-evaluated
with all three ACM badges
Key Results
20
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
SPP
MLOP
Bingo
Pythia
Outperforms prior prefetchers
in wide range of configurations varying core count
Key Results
21
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Outperforms prior prefetchers
in wide range of memory bandwidth configurations
SPP
MLOP
Bingo
Pythia
Key Results
22
More performance gains
with customization
Uto 7.8% perf. gain than baseline Pythia in Ligra
by reward customization
Low-overhead, low-latency, and high-throughput predictions
1.03% area and 0.4% power overhead
Meets latency and throughput of an L2 prefetcher
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Does Pythia Solve the Memory Bottleneck?
23
# demand loads going to off-chip without Pythia
~50%
successfully prefetched
by Pythia
~50%
still going off-chip
even with Pythia
What to do with these loads
that are still going off-chip?
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Contributions
24
Fetch & Decode
Execute
Access Memory
Commit
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Hermes [MICRO’22]
Off-chip prediction (OCP)
using perceptron learning
Athena [under review]
Prefetcher-OCP coordination
using reinforcement learning
Constable [ISCA’24]
Safely eliminating load instruction execution
by exploiting load address-value stability
Pythia [MICRO’21]
Hardware data prefetching
using reinforcement learning
Key Observation for Off-Chip Loads
25
On-chip cache access latency
significantly contributes to the off-chip load latency
40% of the stalls can be eliminated by removing
on-chip cache access latency from critical path
L1
L2
LLC
Main Memory
Saved cycles
L1
L2
LLC
Main Memory
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Our Goal
26
Improve processor performance
by removing on-chip cache access latency
from the critical path of off-chip loads
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
27
L1
L2
LLC
Main Memory
L1
L2
LLC
Main Memory
Predicts which loads are likely to go off-chip
Starts fetching data directly from main memory
while concurrently accessing the cache hierarchy
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
28
Predicts which loads are likely to go off-chip
The first perceptron-based off-chip load predictor
Adaptively learns to identify off-chip loads from multiple program features
Load Request
Σ
Feature
(e.g., PC)
Trainable weights
Off-chip
prediction
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Trace-Driven Simulation Methodology
29
Prefetchers
Workloads
Off-chip Predictors
System Configurations
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Trace-Driven Simulation Methodology
30
Prefetchers
Workloads
Off-chip Predictors
System Configurations
Open-sourced and artifact-evaluated
with all three ACM badges
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Results
31
Provides performance gains by itself
and when added on top of sophisticated prefetchers
11.5%
5.4%
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Results
32
High-accuracy and high-coverage
Off-chip prediction with 77% accuracy & 74% coverage
Low storage and
low power overhead
Only 4 KB storage per core
1.5% increase in power over Pythia
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
One Problem: �Two Fundamentally-Different Approaches
33
Predicts address
of future memory requests
Data prefetcher
Predicts whether or not a given memory request would go off-chip
Off-chip predictor
How do they behave together?
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Contributions
34
Fetch & Decode
Execute
Access Memory
Commit
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Hermes [MICRO’22]
Off-chip prediction (OCP)
using perceptron learning
Athena [under review]
Prefetcher-OCP coordination
using reinforcement learning
Constable [ISCA’24]
Safely eliminating load instruction execution
by exploiting load address-value stability
Pythia [MICRO’21]
Hardware data prefetching
using reinforcement learning
Key Observations
35
Off-chip prediction (OCP) and data prefetching individually provide complementary performance gains
Especially in memory bandwidth-constrained systems, where prefetcher
often degrades performance, but an OCP improves performance
Naively combination is not beneficial
Enabling both mechanisms often negates performance benefits of OCP
Existing mechanisms lack flexibility or have significant performance headroom
TLP [Jamet+, HPCA’24] lacks flexibility.
HPAC [Ebrahimi+, MICRO’09] and MAB [Gerogiannis+, MICRO’23] are not optimal
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Our Goal
36
Design a holistic framework that
Autonomously synergizes OCP with
multiple prefetchers throughout the cache hierarchy
To deliver consistent performance benefits,
regardless of workloads and system configuration
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
37
Key Idea: Formulate OCP-prefetcher coordination
as a reinforcement learning problem
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
38
Athena
Processor and Memory System
RL agent
Environment
Autonomously learn
how to coordinate
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
39
Athena
Processor and Memory System
Statet
Actiont
Rewardt
System features
(e.g., b/w usage, prefetcher accuracy)
Coordination action
(i.e., what to enable
& how much to enable)
Change in telemetry
(e.g., IPC, #LLC miss, #load instructions)
t = N committed instructions
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Trace-Driven Simulation Methodology
40
Prefetchers
Workloads
Off-chip Predictors
System Configurations
Coordination Policies
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Results
41
OCP + 2 prefetchers @ L2
OCP + 1 prefetcher @ L1 & L2
OCP + 1 prefetcher @ L2
OCP + 1 prefetcher @ L1
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Results
42
Consistent performance benefit
In coordinating four types of prefetchers
and three types of off-chip predictors
Low storage overhead
Only 3 KB per core
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Contributions: Recap
43
Fetch & Decode
Execute
Access Memory
Commit
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Hermes [MICRO’22]
Off-chip prediction (OCP)
using perceptron learning
Athena [under review]
Prefetcher-OCP coordination
using reinforcement learning
Constable [ISCA’24]
Safely eliminating load instruction execution
by exploiting load address-value stability
Pythia [MICRO’21]
Hardware data prefetching
using reinforcement learning
Key Contributions
44
Fetch & Decode
Execute
Access Memory
Commit
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Hermes [MICRO’22]
Off-chip prediction (OCP)
using perceptron learning
Athena [under review]
Prefetcher-OCP coordination
using reinforcement learning
Constable [ISCA’24]
Safely eliminating load instruction execution
by exploiting load address-value stability
Pythia [MICRO’21]
Hardware data prefetching
using reinforcement learning
Key Observations
45
A large fraction (34%) of dynamic loads fetch
the same data from the same address
throughout the entire workload
Techniques like load value prediction (LVP) and memory renaming (MRN)
do not fully exploit this data characteristic
These loads cause instruction-level parallelism loss due to resource dependence
i.e., blocking other loads from getting executed due to limited resources.
LVP and MRN fail to mitigate resource dependence
Mitigating both data and resource dependence has higher potential performance improvement
Eliminating execution of such loads provides more than 2X potential performance benefit than just breaking their data dependence
2X
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Our Goal
46
Improve instruction-level parallelism by
mitigating both load data and resource dependence
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Constable: Key Insight
47
mov r8, [rbp+0x8]
sub rax, r8
cmp rsi, rax
jle 0x40230e
add rax, 0x10
mov r8, [rbp+0x8]
sub rax, r8
cmp rsi, rax
jle 0x40230e
Dynamic instruction stream
add rax, 0x10
add rax, 0x10
Two successive dynamic instances
of the same static load instruction
LD1
LD2
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Constable: Key Insight
48
If the source register (rbp)
has not been modified
mov r8, [rbp+0x8]
sub rax, r8
cmp rsi, rax
jle 0x40230e
add rax, 0x10
mov r8, [rbp+0x8]
sub rax, r8
cmp rsi, rax
jle 0x40230e
Dynamic instruction stream
add rax, 0x10
add rax, 0x10
LD2 would have the same address as LD1
Address generation of LD2
can be eliminated
If no store or snoop request
to address [rbp+0x8]
LD2 would fetch the same data as LD1
Data fetching of LD2
can be eliminated
LD1
LD2
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Constable: Key Steps
49
Dynamically identify load instructions
that have historically fetched
the same data from the same load address
(i.e., likely-stable)
Eliminate execution of likely-stable loads
by tracking modifications to
their source registers and their load addresses
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Evaluation with Industry-Grade Simulator
50
Strong Baseline
Workloads
Compared Against
System Configurations
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Results – Without SMT
51
5.1%
3.4%
Provides performance gains by itself
and when added on top of load value predictor
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Results – With 2-way SMT
52
Higher performance benefits
in a 2-way SMT processor
8.8%
11.3%
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Key Results
53
Improved resource efficiency
26% L1-D access reduction,
8.8% RS allocation reduction
Low storage and area overhead
12.4 KB per core, 0.232 mm2 area overhead
Athena [under review]
Pythia [MICRO’21]
Hermes [MICRO’22]
Constable [ISCA’24]
Reduction in dynamic power
9.1% L1-D power reduction, 5.1% RS power reduction
Putting It All Together
Recall: Thesis Statement
55
Adapt its policies by continuously learning
from application data and system-generated data
Tailor its decisions by exploiting characteristics
of application data
By enabling microarchitecture to
We can significantly improve performance and
energy efficiency of state-of-the-art processors
Key Contributions
56
Fetch & Decode
Execute
Access Memory
Commit
Hermes [MICRO’22]
Off-chip prediction (OCP)
using perceptron learning
Athena [under review]
Prefetcher-OCP coordination
using reinforcement learning
Constable [ISCA’24]
Safely eliminating load instruction execution
by exploiting load address-value stability
Pythia [MICRO’21]
Hardware data prefetching
using reinforcement learning
Putting it All Together
57
Hermes [MICRO’22]
Off-chip prediction (OCP)
using perceptron learning
Athena [under review]
Prefetcher-OCP coordination
using reinforcement learning
Constable [ISCA’24]
Safely eliminating load instruction execution
by exploiting load address-value stability
ML-driven techniques
Data-aware technique
Adapt their policies by continuously learning
from application data and system-generated data
Tailor its decisions by exploiting characteristics
of the application data
Pythia [MICRO’21]
Hardware data prefetching
using reinforcement learning
Putting it All Together
58
Hermes [MICRO’22]
Off-chip prediction (OCP)
using perceptron learning
Athena [under review]
Prefetcher-OCP coordination
using reinforcement learning
Constable [ISCA’24]
Safely eliminating load instruction execution
by exploiting load address-value stability
Significantly improve
performance and/or energy efficiency
over best-prior mechanisms
Pythia [MICRO’21]
Hardware data prefetching
using reinforcement learning
Recall: Thesis Statement
59
Adapt its policies by continuously learning
from application data and system-generated data
Tailor its decisions by exploiting characteristics
of application data
By enabling microarchitecture to
We can significantly improve performance and
energy efficiency of state-of-the-art processors
Future Directions
60
Extend
Expand
My Involvements During Ph.D.
61
2019
2020
2021
2022
2023
2024
2025
Thesis
publications
Bera+
MICRO’21
Bera+
MICRO’22
Bera+
ISCA’24
Lang & Bera+
under review
Other major
contributions
Bera+
MICRO’19
Nori+
ISCA’21
ML-driven
storage system
Singh+
ISCA’22
Nadig+
arXiv’25
Address
translation system
Kanellopoulos+
MICRO’23
Kanellopoulos+
ASPLOS’25
Near data
processing
Denzler+
IEEE Access’23
Kabra+
ASPLOS’25
Energy-efficient
architecture
Olgun+
TACO’24
Haj-Yahya+
MICRO’21
Kanellopoulos+
MICRO’23
Kanellopoulos+
arXiv’25
Acknowledgements
62
Acknowledgements
63
My beloved baba,
Mr. R. R. Bera
(1960 - 2020)
Mitigating the Memory Bottleneck with�Machine Learning-Driven and Data-Aware�Microarchitectural Techniques
Rahul Bera
Doctoral Examination
01.10.2025
Advisor:
Onur Mutlu (ETH Zurich)�
Co-Examiners:
Aamer Jaleel (Nvidia Research)
Boris Grot (University of Edinburgh)
Chris Wilkerson (Intel)
Daniel Jiménez (Texas A&M University)
Pradip Bose (IBM T. J. Watson Research Center)
Influence on the Community
65
Open-Sourced Artifacts
66
Impact on the Community
67
MICRO’23
MICRO’23
MICRO’22
MICRO’25
HPCA’24
IISWC’25
Backup
68
69
Hermes
Athena
Constable
Pythia
Pythia Discussion
70
Pythia Discussions
71
(1) Single-Feature Prefetch Prediction
[1] Bakshalipour et al., HPCA’19
[2] Kim et al., MICRO’16
15.4%
3.5%
5.5%
4.6%
Bingo [1] performs better
SPP [2] performs better
72
(2) Lack of Inherent System Awareness
368%
574%
Similar coverage
Lower overpredictions
Yet, lower performance
73
(3) Lack of In-silicon Customizability
Design from scratch
Verify
Fabricate
74
Why RL? Why Not Supervised Learning?
75
What About Large Pages?
76
What is the Prefetch Degree? Is It Managed by the RL Agent?
77
Can the Customization Be Done While the Workload is Running?
78
Can Runtime Workload Mix Create an Issue?
79
How does Pythia Compare Against Other Adaptive Prefetching Solutions?
(a) single-core
(b) four-core
4.5%
6.4%
[5] Jimenez et al., TOPC’14
80
How Does Pythia Compare Against the Context Prefetcher?
Pythia outperforms CP-HW by 5.3% in single-core and 7.6% in four-core system
[6] Leeor et al., ISCA’15
81
How Pythia’s Performance Changes With Various State Definitions You Have Swept?
Performance gain ranges from 20.7% to 22.4%
Coverage ranges from 66.2% to 71.5%
Overprediction ranges from 26.7% to 32.2%
82
Is Pythia Sensitive to Hyperparameter?
Changing 𝜀 from 0.002 to 1.0 drops perf. by 16%
Changing 𝛼 from 0.0065 to 1.0 drops perf. by 5.4%
83
How Does Pythia Compare Against Commercial Multi-level Prefetchers?
Pythia outperforms IPCP [7] by 14.2% on average in 150-MTPS
[6] Prakalapati et al., ISCA’20
84
Does Pythia Perform Equally Well for Unseen Workloads also?
Pythia outperforms MLOP and Bingo by
8.3% and 3.5% in single-core
And 9.7% and 5.4% in four-core
85
Basic Pythia Configuration
86
System Parameters
87
Configuration of Prefetchers
88
Evaluated Workloads
89
List of Evaluated Features
90
Performance S-curve: Single-core
623.xalancbmk_s-592B
603.bwaves_s-2931B
462.libquantum
streamcluster
429.mcf
BFSCC-22B
pagerank-51B
fluidanimate-9500M
91
Performance S-curve: Four-core
429.mcf-184B
pagerank
462.libquantum-1343B
437.leslie3d-271B
Mix-59
raytrace-23.75B
Mix-240
92
Single-core Coverage & Overprediction
93
Detailed Performance
94
Benefit of Bandwidth Awareness
95
Case Study
96
Customizing Rewards
97
Customizing Features
98
Hermes Discussions
99
Hermes Discussions
100
Hermes Overview
Core
L1-D
L2
LLC
MC
Off-Chip
Main Memory
L1
L2
LLC
Main Memory
Baseline
Processor is stalled
Latency tolerance limit of ROB
101
Hermes Overview
Core
L1-D
L2
LLC
MC
Off-Chip
Main Memory
L1
L2
LLC
Main Memory
POPET
L1
L2
LLC
Main Memory
Baseline
Hermes
Saved stall cycles
Processor is stalled
Latency tolerance limit of ROB
Predict
Issue a Hermes request
Wait
Train
Perceptron-based
off-chip load predictor
102
Designing the Off-Chip Load Predictor
Tracking cache contents
Learning from program behavior
Large metadata
May need to track all cache operations
Correlate different program features with off-chip loads
MissMap [Loh+, MICRO’11] for the DRAM cache,
D2D [Sembrant+, ISCA’14], D2M [Sembrant+, HPCA’17], LP [Jalili+, HPCA’22] for the cache hierarchy
History-based prediction
HMP [Yoaz+, ISCA’99] for the L1-D cache
Using branch-predictor-like hybrid predictor:
Global, Gshare, and GSkew
Low storage overhead
Low design complexity
POPET provides
both higher accuracy and higher performance
than predictors inspired from these previous works
103
POPET: Perceptron-Based Off-Chip Predictor
[1] D. Tarjan and K. Skadron, “Merging Path and Gshare Indexing in Perceptron Branch Prediction,” TACO, 2005
104
Predicting using POPET
0x7ffe0+12
42
-4
12
3
3 >= -2
-5
Predict that the load would go off-chip
Extract features from the load request
105
Training POPET
0x7ffe0+12
42
-4
12
3
3 >= -2
-5
Predict that the load would go off-chip
Shouldn’t be activated
Cumulative weight < 𝜏act
-1
-1
-1
106
Latency Configuration
Depends on
0 cycles
24 cycles
6 cycles
107
Single-Core Performance Improvement
11.5%
20.3%
5.4%
Hermes alone provides nearly
50% performance benefits of Pythia �with only 1/5th storage overhead
Hermes on top of Pythia
outperforms Pythia alone in every workload category
Hermes provides nearly 90% performance benefit of Ideal Hermes that has an ideal off-chip load predictor
108
Increase in Main Memory Requests
5.5%
38.5%
5.9%
11.5%
20.3%
5.4%
For every 1% performance benefit,
increase in main memory requests
Pythia
Hermes on top of Pythia
Hermes alone
2%
1%
0.5%
Hermes is more bandwidth-efficient
than even an efficient prefetcher like Pythia
109
Performance with Varying Memory Bandwidth
~AMD Threadripper 3990x (Zen 2, 64C/4ch, 2020)
~AMD EPYC Rome 7702P (Zen 2, 64C/8ch, 2019)
~Intel Xeon 6258R
(Cascade Lake, 28C/6ch, 2020)
Pythia
Hermes
Pythia+Hermes
In bandwidth-constrained configurations,
Hermes alone outperforms Pythia
Hermes+Pythia outperforms Pythia
across all bandwidth configurations
Baseline
110
Performance with Varying Baseline Prefetcher
5.4%
6.2%
5.1%
7.6%
7.7%
Hermes consistently improves performance
on top of a wide range of baseline prefetchers
111
Overhead of Hermes
4 KB storage overhead
1.5% power overhead*
*On top of an Intel Alder Lake-like performance-core [2] configuration
[2] https://www.anandtech.com/show/16881/a-deep-dive-into-intels-alder-lake-microarchitectures/3
112
More in the Paper
113
Initial Set of Program Features
114
Selected Set of Program Features
Five features
A binary hint that represents whether or not a cacheblock has been recently touched
115
When A Feature Works/Does Not Work?
Trace: 462.libquantum-1343B
PC: 0x401442
Cacheline #42
Cacheline #43
…
…
Without prefetcher
With a simple stride prefetcher
116
What Happens in case of a Misprediction?
No need for misprediction detection and recovery
No need for misprediction detection and recovery
117
Performance Headroom of Off-Chip Prediction
118
System Parameters
119
Evaluated Workloads
120
Observation: Not All Off-Chip Loads are Prefetched
50%
Nearly 50% of the loads are still not prefetched
1
121
Observation: Not All Off-Chip Loads are Prefetched
70% of these off-chip loads blocks ROB
2
122
Observation: With Large Cache Comes Longer Latency
58
On-chip cache hierarchy access latency
123
Observation: With Large Cache Comes Longer Latency
58
On-chip cache hierarchy access latency
40% of stall cycles caused by an off-chip load can be eliminated
by removing on-chip cache access latency from its critical path
124
What Fraction of Load Requests Goes Off-Chip?
125
Off-Chip Prediction Quality: Defining Metrics
Predicted off-chip
Actual off-chip
Predicted and actual off-chip
Accuracy
Coverage
126
Off-Chip Prediction Quality: Analysis
Accuracy
Coverage
47%
22%
16%
95%
77%
74%
127
Off-Chip Prediction Quality: Analysis
Accuracy
Coverage
47%
22%
16%
95%
77%
74%
POPET provides off-chip predictions with high-accuracy and high-coverage
128
Effect of Different Features
Combination of features provides both higher accuracy and higher coverage than any individual feature
129
Are All Features Required? (1)
No single feature individually provides
highest prediction accuracy across all workloads
130
Are All Features Required? (2)
No single feature individually provides
highest prediction coverage also across all workloads
131
Single-Core Performance
Hermes in combination with Pythia
outperforms Pythia alone in every workload category
132
Single-Core Performance Line Graph
133
Single-Core Performance Against Prior Predictors
POPET provides higher performance benefit
than prior predictors
Hermes with POPET achieves nearly 90% performance improvement of the Ideal Hermes
134
Effect on Stall Cycles
Hermes reduces off-chip load induced stall cycles
on average by 16.2% (up-to 51.8%)
135
Eight-Core Performance
Hermes in combination with Pythia
outperforms Pythia alone by 5.1% on average
136
Effect of Hermes Request Issue Latency
3.6%
5.7%
Hermes in combination with Pythia outperforms Pythia alone even with a 24-cycle Hermes request issue latency
137
Effect of Cache Hierarchy Access Latency
3.6%
6.2%
Hermes can provide even higher performance benefit in
future processors with bigger and slower on-chip caches
138
Effect of Activation Threshold
With increase in activation threshold
139
Power Overhead
140
Effect of ROB Size
6.7%
5.3%
141
Effect of LLC Size
1.3%
2.5%
142
Accuracy and Coverage with Different Prefetchers
POPET’s accuracy and coverage increases significantly
in absence of a data prefetcher
143
Increase in Main Memory Requests
144
Increase in Main Memory Requests
145
Hermes Limit Study
Athena Discussions
147
Athena Discussions
148
Motivational Data
149
Motivational Data
150
Features Considered
151
Q-Value Table Organization
152
Final Configuration
153
Overhead Analysis
154
Simulation Parameters
155
Evaluated Workloads
156
Evaluated Cache Designs
157
Evaluated Mechanisms
158
Performance Gain in CD1
159
Performance Gain in CD2
160
Performance Gain in CD3
161
Performance Gain in CD4
162
Sensitivity to Prefetcher Type
163
Sensitivity to OCP Type
164
Sensitivity to Main Memory Bandwidth
165
Performance Improvement in Four-Core
166
Performance Improvement in Eight-Core
167
Impact of Uncorrelated Reward
168
Understanding Athena
169
Constable Discussions
170
Constable Discussions
Motivation & Design
Methodology & Evaluation
171
Why Do Global-Stable Loads Exist?
Random* Random::s_rng = 0;
Random* Random::get_Rng(void)
{
if (s_rng == 0)
s_rng = new Random();
return s_rng;
}
Example code from 541.leela_r
Global scope variable
Function to access the variable
172
Why Do Global-Stable Loads Exist?
Random* Random::s_rng = 0;
Random* Random::get_Rng(void)
{
if (s_rng == 0)
s_rng = new Random();
return s_rng;
}
624: mov rax, [rip+0x1f4ac5]
62b: test rax,rax
62e: je 0x638
630: ret
631: nop
638: sub rsp,0x8
63c: mov edi,0xc
641: call 0x460
Example code from 541.leela_r
Disassembly (compiled by GCC* with –O3)
*GNU GCC 13.2
173
Why Do Global-Stable Loads Exist?
Random* Random::s_rng = 0;
Random* Random::get_Rng(void)
{
if (s_rng == 0)
s_rng = new Random();
return s_rng;
}
624: mov rax, [rip+0x1f4ac5]
62b: test rax,rax
62e: je 0x638
630: ret
631: nop
638: sub rsp,0x8
63c: mov edi,0xc
641: call 0x460
Example code from 541.leela_r
Gets initialized only once
and never changes
Global-stable load
Disassembly (compiled by GCC* with –O3)
*GNU GCC 13.2
174
Why Do Global-Stable Loads Exist?
175
Why Do Global-Stable Loads Exist?
Global-stable loads exist for many reasons:
176
Effects of Increasing Architectural Registers
Fraction of global-stable loads
are nearly same without or with APX
Compiled with Clang 18.1.3 with and without -mapxf
177
Effects of Increasing Architectural Registers
Fraction of global-stable loads (i.e., Constable’s opportunities)
are much higher than reduction in loads by APX
Compiled with Clang 18.1.3 with and without -mapxf
178
Effects of Increasing Architectural Registers
The profile of global-stable loads stays largely unchanged
after doubling registers using APX
179
Characterization of Global-Stable Loads (I)
180
Characterization of Global-Stable Loads (II)
181
Characterization of Global-Stable Loads (III)
182
Resource Dependence by Global-Stable Loads
Global-stable loads cause significant resource dependence
183
Performance Headroom Analysis
184
Constable Overview
185
Architecting SLD
186
Effect of Wrong-Path Update
187
An Example of Constable’s Operation
188
Ensuring Coherence
189
Ensuring Coherence
190
Constable’s Storage Overhead
191
Area and Power Overhead of Constable’s Structures
192
Simulated System Parameters
193
Evaluated Workloads
194
Workload-Wise Performance
195
Load Category-Wise Performance
196
Performance Comparison with Prior Works
197
Elimination Coverage
198
Coverage of Global-Stable Loads
199
Reduction in Dynamic Power
200
Performance Sensitivity to Load Execution Width Scaling
201
Performance Sensitivity to Pipeline Depth Scaling
202
Eliminated Loads that Violates Memory Ordering
Only 0.09% of all eliminated loads
violate memory ordering
203
Eliminated Loads that Violates Memory Ordering
Eliminated loads that violate memory ordering
increase number of allocated instructions only by 0.3%
204