1 of 91

�Synergizing Data Prefetching and Off-Chip Prediction �via Online Reinforcement Learning

https://github.com/CMU-SAFARI/Athena

Rahul Bera, Zhenrong Lang

Caroline Hengartner, Konstantinos Kanellopoulos, Rakesh Kumar,

Mohammad Sadrosadati, Onur Mutlu

2 of 91

Executive Summary (I)

2

Long memory access latency significantly limits performance of modern high-performance processors

Prefetching and off-chip prediction (OCP) are two orthogonal techniques proposed to hide long memory access latency

Background

Key Insights

Existing coordination policies leave a large performance potential behind

Naively combining OCP with prefetching often fails to realize their full

performance potential

OCP and prefetching provide complementary performance benefits

Goal

Autonomously coordinates off-chip prediction with multiple prefetchers
Provides consistent performance benefits, regardless of workloads and system configurations

3 of 91

Executive Summary (II)

3

Contributions

Works as a prefetcher-OCP coordinator and a prefetcher throttler, at the same time, without any additional hardware

Introduces a composite reward framework that isolates the true impact of Athena’s own action from inherent variations in workload behavior

Athena

Models the coordination between prefetchers and off-chip predictor (OCP) as a reinforcement learning (RL) problem
Observes multiple system-level features over an epoch of program execution
Takes a coordination action (i.e., enabling the prefetcher and/or OCP, and adjusting prefetcher aggressiveness)
Receives a numerical reward to autonomously and continuously learn

Our Proposal

Evaluation

Evaluated with 100 workloads, 6 prefetchers, 3 OCPs, 4 cache designs, and a wide range of main memory bandwidth configurations

Consistently outperforms both heuristic- and learning-based policies in every prefetcher/OCP/bandwidth configurations on average by 3.6% - 10.3%

https://github.com/CMU-SAFARI/Athena

To this end, we propose Athena which models ...

[CLICK] Athena works in epochs of workload execution.

At the end of every epoch, Athena extracts a set of system-level features,

such as memory bandwidth usage, prefetcher or OCP’s accuracy, cache pollution

and use it as a state information to take a coordination action.

[CLICK] An action in this case, is to not only enable or disable each mechanism,

but also to dictate the aggressiveness of the prefetcher

[CLICK] For every action, Athena receives a numerical reward, that measures

the change in multiple system-level metrics. Athena uses this reward to

autonomously and continuously learn the coordination policy.

[CLICK] Athena improves upon prior learning-based coordination mechanisms

in two key ways.

[CLICK] First, Athena introduces a composite reward framework that ...

and

[CLICK] Athena works both as a ...

[CLICK] Our extensive evaluation using a wide range of workloads, prefetchers, off-chip predictors

and system configuration shows that

Athena consistently outperforms both best-prior heuristics-based and learning-based coordination policies.

[CLICK] Athena is open-sourced and fully artifact evaluated, which you can download from our GitHub repo

4 of 91

Outline

4

Background

Motivation

Athena

Evaluation

Conclusion

5 of 91

Key Problem and Its Potential Solutions

5

Long memory access latency

remains a key performance bottleneck in modern processors

Key latency hiding techniques:

1

Data prefetching

2

Off-chip prediction

6 of 91

Data Prefetching and Off-Chip Prediction

6

Predicts address of future memory requests�

Hides the memory

access latency�

Bandwidth overhead

and cache pollution

Predicts whether a given memory request would go off-chip

Often more accurate

predictions�

Lower timeliness than a

prefetcher

Data prefetcher

Off-chip predictor

How do they behave together?

[CLICK] Data prefetching is a widely-explored speculation technique that predicts the address ....

and fetches the corresponding data to the on-chip caches before the program demands it

[CLICK] If the prediction is correct, a prefetcher can partially or fully hide the long memory access latency

[CLICK] However, incorrect speculation can cause severe memory bandwidth overhead and cache pollution

[CLICK] Off-chip predictor, on the other hand, is a recently-proposed technique that predicts whether or not ...

and starts fetching their data directly from the main memory ...

An off-chip predictor often produce more accurate speculation compared to traditional prefetchers

However, it also provides less timeliness than a prefetcher.

Now, technically, both of these mechanisms can be orthogonally applied in a processor.

[CLICK] But the question is: how do they behave in presence of each other?

7 of 91

Outline

7

Background

Athena

Evaluation

Conclusion

Motivation

8 of 91

Ob. 1: Data Prefetching and Off-Chip Prediction Provide Complementary Performance Benefits

8

In bandwidth-constrained processor with 3.2 GB/s of main memory bandwidth

(OCP)

(Prefetcher)

Prefetcher-friendly

Prefetcher-adverse

First, we observe that prefetcher and off-chip predictor often provides complementary performance benefit,

especially in memory bandwidth-constrained processors.

To illustrate this, here I am showing the performance improvement of a state-of-the-art prefetcher Pythia

employed in the L2 cache of a bandwidth-constrained single-core processor.

On the x-axis, we have different workloads, and the y-axis, we have the speedup.

[CLICK] As we can see, Pythia improves performance for the majority of the workloads even in this bandwidth-constrained configuration.

[CLICK] We call such workloads – prefetcher friendly.

[CLICK] However, for a significant number of workloads, Pythia significantly degrades performance,

even with its built-in bandwidth-aware throttling mechanism

[CLICK] In many of these prefetcher-adverse workloads, the state-of-the-art off-chip predictor POPET provides performance benefit.

However, in prefetcher-friendly workloads, POPET significantly underperforms Pythia.

�

9 of 91

Ob. 1: Data Prefetching and Off-Chip Prediction Provide Complementary Performance Benefits

9

In bandwidth-constrained processor with 3.2 GB/s of main memory bandwidth

(OCP)

(Prefetcher)

Prefetcher-friendly

Prefetcher-adverse

Data prefetching and off-chip prediction provide complementary performance benefits

First, we observe that prefetcher and off-chip predictor often provides complementary performance benefit,

especially in memory bandwidth-constrained processors.

To illustrate this, here I am showing the performance improvement of a state-of-the-art prefetcher Pythia

employed in the L2 cache of a bandwidth-constrained single-core processor.

On the x-axis, we have different workloads, and the y-axis, we have the speedup.

[CLICK] As we can see, Pythia improves performance for the majority of the workloads even in this bandwidth-constrained configuration.

[CLICK] We call such workloads – prefetcher friendly.

[CLICK] However, for a significant number of workloads, Pythia significantly degrades performance,

even with its built-in bandwidth-aware throttling mechanism

[CLICK] In many of these prefetcher-adverse workloads, the state-of-the-art off-chip predictor POPET provides performance benefit.

However, in prefetcher-friendly workloads, POPET significantly underperforms Pythia.

�

10 of 91

Ob. 2: Naively Combining OCP with Prefetching Fails to Realize their Full Performance Potential

10

14.1%

6.5%

StaticBest <POPET, Pythia>

Let’s now take a look what happens when we enable both techniques without coordination.

[CLICK] In prefetcher-adverse workloads, we have already seen that

POPET improve performance, whereas Pythia severely degrades performance.

[CLICK] If we naively combine Pythia and POPET, it completely masks out

the performance improvement that POPET alone could have delivered.

[CLICK] However, if we could have controlled these two mechanisms

based on the oracular understanding of their effectiveness,

which we show here as the StaticBest combination,

it could have provided a 14.1% performance improvement over the naive combination.

[CLICK] In prefetcher-friendly workloads, we have already seen that

Pythia provides more performance benefits than POPET.

The naive and StaticBest combination closely follows the trend of Pythia.

[CLICK] Overall across all workloads, the naive combination underperforms the StaticBest combination

by 6.5% on average

11 of 91

Ob. 2: Naively Combining OCP with Prefetching Fails to Realize their Full Performance Potential

11

14.1%

6.5%

StaticBest <POPET, Pythia>

Naively combining OCP with prefetching often

fails to realize their full performance potential

Let’s now take a look what happens when we enable both techniques without coordination.

[CLICK] In prefetcher-adverse workloads, we have already seen that

POPET improve performance, whereas Pythia severely degrades performance.

[CLICK] If we naively combine Pythia and POPET, it completely masks out

the performance improvement that POPET alone could have delivered.

[CLICK] However, if we could have controlled these two mechanisms

based on the oracular understanding of their effectiveness,

which we show here as the StaticBest combination,

it could have provided a 14.1% performance improvement over the naive combination.

[CLICK] In prefetcher-friendly workloads, we have already seen that

Pythia provides more performance benefits than POPET.

The naive and StaticBest combination closely follows the trend of Pythia.

[CLICK] Overall across all workloads, the naive combination underperforms the StaticBest combination

by 6.5% on average

12 of 91

Ob. 3: Existing Coordination Policies Fall Short

12

TLP [Jamet+, HPCA’24] is the only technique proposed

to combine OCP with a prefetcher

13 of 91

Ob. 3: Existing Coordination Policies Fall Short

13

TLP [Jamet+, HPCA’24] is the only technique proposed

to combine OCP with a prefetcher

Can coordinate OCP with prefetcher employed only at the L1 data cache

14 of 91

Ob. 3: Existing Coordination Policies Fall Short

14

9.2%

3.4%

5.9%

15 of 91

Ob. 3: Existing Coordination Policies Fall Short

15

9.2%

3.4%

5.9%

Existing coordination policies

leave a large performance potential behind

16 of 91

Our Goal

16

Design a holistic framework that

Autonomously synergizes OCP with

multiple prefetchers throughout the cache hierarchy

To deliver consistent performance benefits,

regardless of workloads and system configuration

17 of 91

Our Proposal

17

Formulates the coordination between prefetchers and OCP

as a reinforcement learning problem

18 of 91

Outline

18

Background

Motivation

Evaluation

Conclusion

Athena

19 of 91

Basics of Reinforcement Learning (RL)

An algorithmic approach to learn to take an action in a given state to maximize a numerical reward

Agent stores Q-values for every state-action pair

Expected reward for taking an action in a state

Given a state, the agent selects the action that provides the highest Q-value

19

Environment

Action (a_t)

Reward (R_t+1)

Agent

State (s_t)

20 of 91

Formulating Coordination as RL

20

Environment

Action (a_t)

Reward (R_t+1)

Agent

State (s_t)

Processor & Memory Subsystem

Action�1) enable/disable prefetcher and OCP

2) set prefetcher aggressiveness

Reward

Athena

State

(e.g., bandwidth usage, prefetcher/OCP accuracy)

21 of 91

What is State?

21

Vector of system-level features
State := [Feature 1, Feature 2, …]

Captures runtime behavior of the memory subsystem

Perform offline feature selection to determine the final subset

of features that Athena uses to construct the state vector

Prefetcher accuracy
OCP accuracy
Bandwidth usage
Cache pollution

Prefetch bandwidth
OCP bandwidth
Demand bandwidth

Example Features

Automated Design-Space Exploration

22 of 91

What is Action?

Prefetcher /

OCP /

Adjust the aggressiveness of prefetcher

22

23 of 91

What is Action?

23

>> avg( , , )

OCP Only

Prefetcher Only

Neither Enabled

Both Enabled

2

28

4

9

The magnitude of the selected action’s Q-value implicitly encodes Athena’s confidence in taking that action

State vector

24 of 91

Q-Value-Driven Prefetcher Aggressiveness Control

24

Key Idea: Use the Q-value difference to control prefetcher aggressiveness

Low ∆Q

🡪 Low aggressiveness

High ∆Q 🡪 High aggressiveness

Calculate Q-value difference

∆Q ← Q(a*) − avg (Q(remaining actions))

Drive prefetcher aggressiveness using ∆Q

25 of 91

Q-Value-Driven Prefetcher Aggressiveness Control

25

Key Idea: Use the Q-value difference to control prefetcher aggressiveness

Low ∆Q

🡪 Low aggressiveness

High ∆Q 🡪 High aggressiveness

Calculate Q-value difference

∆Q ← Q(a*) − avg (remaining actions)

Drive prefetcher aggressiveness using ∆Q

Athena works as a prefetcher-OCP coordinator and a prefetcher throttler, at the same time, using the same hardware

26 of 91

What is Reward?

Defines the objective of Athena

Intuitive reward: instructions committed per cycle (IPC)

26

Using IPC as the sole reward can be unreliable and may mislead the learned policy

Change in IPC

Coordination actions taken by Athena

Inherent variation in workload behavior

27 of 91

Athena's Composite Reward Framework

27

Allows Athena to autonomously learn a coordination policy

by isolating the true impact of its actions

from inherent variations in workload behavior

Athena’s overall reward

Correlated reward

Reflects the effect of

Athena’s action

- Cycles

- LLC misses

- LLC miss latency

Uncorrelated reward

Reflects inherent

workload variation

Load instructions
Mispredicted branches

28 of 91

Athena's Composite Reward Framework

28

Allows Athena to autonomously learn a coordination policy

by isolating the true impact of its actions

from inherent variations in workload behavior

Athena’s overall reward

Correlated reward

Reflects the effect of

Athena’s action

- Cycles

- LLC misses

- LLC miss latency

Uncorrelated reward

Reflects inherent

workload variation

Load instructions
Mispredicted branches

Potentially broadly applicable to many other

microarchitectural decision-making processes

29 of 91

Final Configuration of Athena

29

Four features

Prefetcher accuracy
OCP accuracy
Bandwidth usage
Prefetcher-induced cache pollution

Coarse-grained action

Enable prefetcher
Enable OCP
Enable both
Enable none

Fine-grained action

Q-value driven prefetcher aggressiveness control

Correlated reward

Cycles
LLC misses
LLC miss latency

Uncorrelated reward

Load instructions
Mispredicted branches

State

Action

Reward

30 of 91

31 of 91

32 of 91

Outline

32

Background

Motivation

Athena

Conclusion

Evaluation

33 of 91

ChampSim Trace-Driven Simulation Methodology

33

IPCP [Pakalapati+, ISCA’20]
Berti [Navarro-Torres+, MICRO’22]
Pythia [Bera+, MICRO’21]
SPP+ PPF [Bhatia+, ISCA’20]
SMS [Somogyi+, ISCA’06]
MLOP [Shakerinava+, DPC3’19]

Prefetchers

100 single-core workloads
180 four-core and eight-core workload mixes

Workloads

Hermes [Bera+, MICRO’22]
HMP [Yoaz+, ISCA’99]
TTP [Jalili+, HPCA’22]

Off-Chip Predictors

# cores: 1- 8 cores
Main memory bandwidth: �1.6 GB/s – 12.8 GB/s
4 cache designs with various prefetchers at different cache levels

System Configurations

HPAC [Ebrahimi+, MICRO’09]
MAB [Yoaz+, ISCA’99]
TLP [Jamet+, HPCA’24]

Coordination Policies

34 of 91

Trace-Driven Simulation Methodology

34

IPCP [Pakalapati+, ISCA’20]
Berti [Navarro-Torres+, MICRO’22]
Pythia [Bera+, MICRO’21]
SPP+ PPF [Bhatia+, ISCA’20]
SMS [Somogyi+, ISCA’06]
MLOP [Shakerinava+, DPC3’19]

Prefetchers

100 single-core workloads
180 four-core and eight-core workload mixes

Workloads

Hermes [Bera+, MICRO’22]
HMP [Yoaz+, ISCA’99]
TTP [Jalili+, HPCA’22]

Off-chip Predictors

# cores: 1- 8 cores
Main memory bandwidth: �1.6 GB/s – 12.8 GB/s
4 cache designs with various prefetchers at different cache levels

System Configurations

HPAC [Ebrahimi+, MICRO’09]
MAB [Yoaz+, ISCA’99]
TLP [Jamet+, HPCA’24]

Coordination Policies

Open-sourced and artifact-evaluated

with all three badges

https://github.com/CMU-SAFARI/Athena

35 of 91

Evaluated Cache Designs

35

Cache Design 1

Cache Design 2

Cache Design 3

Cache Design 4

PF

OCP

PF

PF₁

PF₂

PF₁

PF₂

36 of 91

Evaluated Cache Designs

36

Cache Design 1

Cache Design 2

Cache Design 3

Cache Design 4

PF

OCP

PF

PF₁

PF₂

PF₁

PF₂

37 of 91

Speedup in Cache Design 1

37

4.8%

7.5%

1.5%

6.2%

5.0%

38 of 91

Speedup in Cache Design 1

38

4.8%

7.5%

1.5%

6.2%

5.0%

Athena consistently outperforms prior prefetcher control policies

in all workload categories

Athena outperforms the best-prior coordination policy MAB

by 5% on average

39 of 91

Speedup in Cache Design 1

39

40 of 91

Speedup in Cache Design 1

40

Athena provides similar performance gains

as the StaticBest combination

41 of 91

Performance Sensitivity to Varying Prefetchers

41

5.0%

5.4%

3.6%

5.0%

42 of 91

Performance Sensitivity to Varying OCPs

42

5.0%

4.7%

8.2%

43 of 91

Performance Sensitivity to Varying OCPs

43

5.0%

4.7%

8.2%

Athena provides consistent performance benefits

across diverse prefetcher and OCP types

44 of 91

More Evaluations in the Paper

Performance improvement in three other cache designs

Performance sensitivity study across diverse system configurations

Varying L1D prefetcher
Varying OCP request issue latency
Varying main memory bandwidth

Understanding Athena using a case study

Employing Athena for prefetcher-only management

Additional results in the extended version on arXiv:

Effect of Athena on main memory request
Effect of Athena on LLC load miss latency
Performance improvement of Athena on 359 additional unseen traces

44

45 of 91

More Evaluations in the Paper

45

https://www.arxiv.org/pdf/2601.17615

46 of 91

Athena is Open Sourced

46

https://github.com/CMU-SAFARI/Athena

SPEC CPU 2006 & 2017
PARSEC 2.1
Ligra graph processing workloads
Real-world INT, FP

100 Workloads

IPCP [Pakalapati+, ISCA’20]
Berti [Navarro-Torres+, MICRO’22]
Pythia [Bera+, MICRO’21]
SPP+PPF [Kim+, MICRO’16]
SMS [Somogyi+, ISCA’06]
MLOP [Shakerinava+, DPC3’19]

Six Prefetchers

POPET [Bera+, MICRO’22]
HMP [Yoaz+, ISCA’99]
TTP [Jalili+, HPCA’22; Bera+, MICRO’22]

Three OCPs

TLP [Jamet+, HPCA’24]
HPAC [Ebrahimi+, MICRO’09]
MAB [Gerogiannis+, MICRO’23]
Athena (Our Proposal)

Four Policies

47 of 91

Outline

47

Conclusion

Background

Motivation

Athena

Evaluation

48 of 91

48

Evaluation

Evaluated with 100 workloads, 6 prefetchers, 3 OCPs, 4 cache designs, and a wide range of main memory bandwidth configurations

Consistently outperforms both best-prior heuristic- and learning-based policies in every prefetcher/OCP/bandwidth configuration

https://github.com/CMU-SAFARI/Athena

The first reinforcement learning-based mechanism to coordinate

off-chip predictor and multiple prefetchers

Contributions

Works as a prefetcher-OCP coordinator and a prefetcher throttler, at the same time, without any additional hardware

Introduces a composite reward framework that isolates the true impact of Athena’s own action from inherent variations in workload behavior

49 of 91

�Synergizing Data Prefetching and Off-Chip Prediction �via Online Reinforcement Learning

Rahul Bera, Zhenrong Lang, Caroline Hengartner, Konstantinos Kanellopoulos,

Rakesh Kumar, Mohammad Sadrosadati, Onur Mutlu

50 of 91

BACKUP

50

51 of 91

Index

Motivational data

POPET v. Pythia
Perf. headroom
Why TLP fails?
Prior works’ headroom

Athena design

Candidate features
Pref. aggressiveness algo
Reward framework
QVStore retreival
Final config
Storage overhead
Latency overhead

Methodology

System param
Workloads
Cache designs
Storage overhead comparison

51

52 of 91

POPET vs. Pythia Performance Line Graph

52

	Accuracy	Performance
Pythia	28.7%	-10.5%
POPET	84.1%	+10.3%

53 of 91

Athena Performance Headroom

53

54 of 91

Off-Chip Prefetch Fills are not Always Inaccurate

54

55 of 91

Prior Works Leave Performance Potential Behind

55

56 of 91

Candidate Features Considered

56

57 of 91

Q-Value-Driven �Prefetcher Aggressiveness Control

57

58 of 91

Reward Framework

58

59 of 91

Q-Value Retrieval from QVStore

59

60 of 91

Final Configuration of Athena

60

61 of 91

Storage Overhead of Athena

61

62 of 91

Why is Athena’s Latency Overhead Minimal?

62

Epoch_k

Epoch_k+1

Epoch_k+2

Epoch_k+3

2000 instructions

Latency of

Q-Value updates

Latency to

execute an epoch

<<

63 of 91

Evaluated System Parameters

63

64 of 91

Evaluated Workloads

64

65 of 91

Evaluated Cache Designs

65

66 of 91

Storage Overhead Comparison

66

67 of 91

Speedup in CD1

67

68 of 91

CD1 Speedup – Deepdive

68

69 of 91

CD1 Speedup – Compared to StaticBest

69

70 of 91

CD1 – Impact on Main Memory Requests

70

71 of 91

CD1 – Impact on LLC Load Miss Latency

71

72 of 91

CD1 – Sensitivity to L2C Prefetcher Type

72

73 of 91

CD1 – Sensitivity to OCP Type

73

74 of 91

CD1 – Sensitivity to OCP Request Issue Latency

74

75 of 91

CD1 – Sensitivity to Warmup Length

75

76 of 91

Speedup in CD2

76

77 of 91

Speedup in CD3

77

78 of 91

Speedup in CD4

78

79 of 91

CD4 – Sensitivity to L1D Prefetcher Type

79

80 of 91

CD4 – Sensitivity to Main Memory Bandwidth

80

81 of 91

Four-Core Evaluation Results

81

82 of 91

Eight-Core Evaluation Results

82

83 of 91

Understanding Athena using a Case Study

83

84 of 91

Athena Ablation Study

84

85 of 91

Athena for Prefetcher-Only Management

85

86 of 91

Athena on Unseen Workloads

86

Across 359 Google datacenter workload traces

released in DPC4

87 of 91

Old Slides

87

88 of 91

Automated Design Space Exploration

88

4 Final Features

Feature N

Feature 1

Feature 2

Prefetcher accuracy, OCP accuracy, bandwidth usage, and prefetch-induced cache pollution

89 of 91

Athena: Detailed Design

89

Q-Value�Table

QVStore

+

State

Action

90 of 91

Athena: Detailed Design

90

Q-Value�Table

QVStore

+

State

Action

Athena adopts a lightweight and hardware-friendly

tabular organization for storing Q-values, tailored for

low-latency access and online updates

91 of 91

Putting it all Together

91

Feature₁

Feature₂

Feature_F

State vector

.�.�.

concatenate

(32-bit)

𝚺

Q(s, a)

sum partial�Q-values

argmax

a^*

q₁(s, a)

q₂(s, a)

q_k(s, a)

Plane_1��

Plane_2��

Plane_k_��

#

hash₁

index

#

hash₂

index

#

hash_k

index

.�.�.