1 of 117

ECE 5755: Modern Computer Systems and Architecture

Lecture 12: VM & DRAM

Mark D. Hill

Norman Jouppi

Dr. Jouppi has made innumerable contributions to memory hierarchy design, with the most significant multiple ideas presented in his 1990 ISCA paper “Improving direct-mapped cache performance by the addition of a small fully-associative cache and prefetch buffers”

Widely regarded as the leading memory systems researcher in the world today, Hill made seminal contributions to the fields of cache memories, memory consistency models, transactional memory, and simulation. In the 1980s Hill developed the “3C” model of cache misses

2 of 117

Today’s agenda

Part 1: Virtual memory

Why indirection? Page tables, from two levels to x86-64
Making translation fast: TLBs and how they meet the cache
Page size, TLB reach, huge pages, virtual machines
Modern example: paging in LLM serving

Part 2: DRAM

Cell, array and the access protocol
Refresh and errors
From chips to DIMMs to channels; the memory controller
Modern systems: DDR5, HBM, CXL

Photos, illustrations, graphics here.

3 of 117

Virtual memory

Point we have been avoiding

Addresses generated by program are not the addresses that we use to access the memory (physical memory)
Why?

Photos, illustrations, graphics here.

4 of 117

Virtual Memory: Motivation

Original Motivation: Allow main memory to “act as a cache” for secondary storage (disks)

Physical memory expensive and not very dense (too small)
Programmers wrote “overlays” to load memory from disk
Programming nightmare, incompatible code across products

Current Motivation: Use indirection of VM as a feature

Physical memories are quite large
Multiprogramming, sharing, relocation, protection
Memory mapped files, networks

Photos, illustrations, graphics here.

5 of 117

Virtual vs. Physical Memory

Photos, illustrations, graphics here.

6 of 117

Virtual Memory: Cache Analogy

Cache blocks/lines are called pages
Cache misses are page faults or address faults
Processes use virtual addresses (VA)
Physical memory uses physical addresses (PA)
Addresses divided into page offset, page number

Virtual: Virtual Page Number (VPN)
Physical: Physical Page Number (PPN)

Addresses translation: system maps VA to PA

E.g. 4KB pages, 32-bit machine, 64MB physical memory
32-bit VA, 26-bit PA (log₂64MB), 12-bit page offset (log₂4KB)

Photos, illustrations, graphics here.

7 of 117

Virtual Memory: Cache Analogy

Parameter	First-Level Cache	Virtual Memory
Block (page) Size	16-128 Bytes	4KB – 64KB
Hit Time	1-3 clock cycles	50-150 clock cycles
Miss Penalty	8-15 clock cycles	1M-10M clock cycles
(access time)	(6-130 clock cycles)	(.8M – 8M clock cycles)
(transfer time)	(2-20 clock cycles)	(.2M – 2M clock cycles)
Miss Rate	0.1-10%	0.00001 –0.001%
Address Mapping	25-45bit PA to 14-20bit CacheAd	32-64 bit VA to 25-45 bit PA
Replacement Policy	Hardware Replacement	Software Replacement
Total Size	Independent of Address Space	Processor Address Space
Backing Store	Level 2 Cache	Physical Disk

Photos, illustrations, graphics here.

8 of 117

Hardware maps virtual to physical addresses

Virtual Page Number (VPN) => Physical Page Number (PPN)

OS/Hardware perform the mapping, not processes

Same VPNs in different processes have different PPNs

Protection: processes cannot use each other’s PA
Programming Simplicity: Each process thinks its alone
Relocation: Program can be run anywhere in memory

Doesn’t have to be physically contiguous
Can be paged out, paged back to a different physical location

Photos, illustrations, graphics here.

9 of 117

Virtual Memory: 4 Cache Questions

Same four questions as caches

Page Placement: fully associative

Why?

Page Identification: address translation

Indirection through one or two page tables

Page Replacement: Sophisticated LRU + Working set

Why?

Write Strategy: Always write-back + write allocate

Why?

Photos, illustrations, graphics here.

10 of 117

Why?

Backing store for main memory is disk or SSD

DRAM: ~50–100 ns, already 100s of CPU cycles
Flash SSD: ~50–100 µs, about 1,000× slower than DRAM
Hard disk: ~10 ms, about 100,000× slower than DRAM

VA miss (page fault) is expensive

Minimize at all costs
Fully associative + software replacement reduce miss rate
Write-back reduces disk traffic
Large page sizes (4KB – 16KB) amortize reads

Photos, illustrations, graphics here.

11 of 117

Page ID: Address translation

OS performs address translation using page table

Each process has its own page table

OS knows address of each process’s page table

Page table is an array of Page Table Entries

One entry for each VPN of each process, indexed by VPN

Each PTE contains

Physical page number
Permissions
Dirty bit
Valid + referenced bits
~4 bytes (8 on x86-64)

Virtual Page Number Page Offset

Virtual Address

Page

Table

Main

Memory

Photos, illustrations, graphics here.

12 of 117

Page faults: demand paging, copy-on-write

Page fault: PTE says “not present” → trap to the OS
Demand paging: map lazily, allocate on first touch

malloc(1 GB) costs nothing until pages are touched

Minor fault: page already in memory, fix the PTE (~1 µs)
Major fault: read from SSD or disk (100 µs to 10 ms)
Copy-on-write: share a frame until someone writes

fork(), shared libraries, zero pages

After fork(): one frame, mapped read-only

Parent page table

Child page table

Frame A

After the child writes: OS copies that one page

Parent page table

Child page table

Frame A

Frame A′ (copy)

Photos, illustrations, graphics here.

13 of 117

Multi-level page tables

Most processes use only a tiny portion of total VA space
Tree of page tables

L1 table points to L2 tables (and more if needed)

Different VPN bits are offsets at different levels

Save space: not all tables at all levels need to exist
Slow: Multi-hop chains of translations (space savings outweigh)

PT Root

VPN

1^st Level PT

2^nd Level PTs

Photos, illustrations, graphics here.

14 of 117

Multi-level page tables

32-bit address space, 4KB pages, 4 byte PTEs
2 level virtual page table
2^nd-level tables are each the size of 1 data page
Program uses only upper and lower 1MB of address space
How much memory does page table take?

4GB VM, How many pages? => a) 2M b) 4M c) 1M pages

Photos, illustrations, graphics here.

15 of 117

Multi-level page tables

32-bit address space, 4KB pages, 4 byte PTEs
2 level virtual page table
2^nd-level tables are each the size of 1 data page
Program uses only upper and lower 1MB of address space
How much memory does page table take?

4GB VM, How many pages? => 1M pages
How many pages per 2^nd level table => a) 1K b) 2K c) 4K pages

Photos, illustrations, graphics here.

16 of 117

Multi-level page tables

32-bit address space, 4KB pages, 4 byte PTEs
2 level virtual page table
2^nd-level tables are each the size of 1 data page
Program uses only upper and lower 1MB of address space
How much memory does page table take?

4GB VM, How many pages? => 1M pages
How many pages per 2^nd level table => 1K pages
How many 2^nd-level tables => a) 1K b) 2K c) 4K

Photos, illustrations, graphics here.

17 of 117

Multi-level page tables

32-bit address space, 4KB pages, 4 byte PTEs
2 level virtual page table
2^nd-level tables are each the size of 1 data page
Program uses only upper and lower 1MB of address space
How much memory does page table take?

4GB VM, How many pages? => 1M pages
How many pages per 2^nd level table => 1K pages
How many 2^nd-level tables => 1K
Size of first level table => a) 1KB b) 2KB c) 4KB

Photos, illustrations, graphics here.

18 of 117

Multi-level page tables

32-bit address space, 4KB pages, 4 byte PTEs
2 level virtual page table
2^nd-level tables are each the size of 1 data page
Program uses only upper and lower 1MB of address space
How much memory does page table take?

4GB VM, How many pages? => 1M pages
How many pages per 2^nd level table => 1K pages
How many 2^nd-level tables => 1K
Size of first level table => 4KB
1MB VA space, How many PTEs

Photos, illustrations, graphics here.

19 of 117

Multi-level page tables

32-bit address space, 4KB pages, 4 byte PTEs
2 level virtual page table
2^nd-level tables are each the size of 1 data page
Program uses only upper and lower 1MB of address space
How much memory does page table take?

4GB VM, How many pages? => 1M pages
How many pages per 2^nd level table => 1K pages
How many 2^nd-level tables => 1K
Size of first level table => 4KB
1MB VA space, How many PTEs => 256 PTEs

Photos, illustrations, graphics here.

20 of 117

Before the walk: why 9 bits?

THINK · PAIR · SHARE

x86-64 splits the virtual page number into four 9-bit indices.

Why 9? What fixes that number?

1 min think · 2 min with your neighbor. Hint: what is the size of one page-table node?

Photos, illustrations, graphics here.

21 of 117

A real page table: the x86-64 walk

Each table is one 4 KB page: 512 entries × 8 bytes. CR3 holds the root, one per process
A TLB miss costs 4 dependent memory reads, done by a hardware page-table walker
Cheaper in practice: page-walk caches skip the upper levels, and PTEs sit in the data caches
Stop one level early for a 2 MB page, two levels early for a 1 GB page

64-bit virtual address (48 bits used)

unused

bits 63–48

L4 index

bits 47–39

L3 index

bits 38–30

L2 index

bits 29–21

L1 index

bits 20–12

page offset

bits 11–0

CR3

root

PML4

512 entries

1

PDPT

512 entries

2

PD

512 entries

3

PT

512 entries

4

4 KB page

the data

Photos, illustrations, graphics here.

22 of 117

Address translation

How does address translation really work?
Multi-level page tables

Several levels of indirection: every load or store first reads each level of the page table
2 extra memory accesses with two levels, 4 on x86-64 (slow!)
Processes do not read the page table and translate: the system does

Hardware involvement: Translation Lookaside Buffer

Cache dedicated to these translations

Photos, illustrations, graphics here.

23 of 117

What could go wrong?

THINK · PAIR · SHARE

Suppose L1 is indexed and tagged with virtual addresses.

No translation on a hit. What can go wrong?

1 min think · 2 min pair. Hint: think about two processes, then one process with shared memory.

Photos, illustrations, graphics here.

24 of 117

Fast translation: virtual caches

First-level caches are “virtually addressed”
L2 and main memory are “physically addressed”
Address translation only on a miss (not critical)
Why is this not a great idea?

Protection: translation checks page-level protection
Homonyms: same VA, different PA in another process → flush the cache on context switch, or tag with a process ID
Synonyms: different VAs, same PA (shared libraries, copy-on-write) → two cached copies can disagree
I/O: typically uses PAs (would need conversion to access L1)

CPU

L1

L2

xlate

Main Memory

Photos, illustrations, graphics here.

25 of 117

Fast translation: physical caches + TLB

L1 is physically tagged: no homonyms or synonyms

Translation lookaside buffer (TLB) supplies the physical tag
Keeps I/O coherent

Translation runs in parallel with the L1 index

Fast (one step access); how: next slide

Changing address spaces

Entries tagged with an address-space ID: no flush on context switch
Multicore: changing a mapping must invalidate other cores’ TLBs (shootdown)

CPU

TLB

L2

xlate

Main Memory

L1

Photos, illustrations, graphics here.

26 of 117

Cache: Virtual Index, Physical Tag

Physical cache with virtual address

Only cache index matters for access
Only part of virtual address changes during translation
Make sure index is in untranslated part

Index is within page offset
Virtual index == physical index

Fast
Restricts cache size (Block size * #sets <= page size)
Use associativity to increase size

Virtual Page Number

Virtual Address

Tag

Page Offset

Index

Offset

Photos, illustrations, graphics here.

27 of 117

Why can Apple’s L1 be 128 KB?

Constraint from the previous slide: index bits must lie inside the page offset → size / associativity ≤ page size
x86 has been stuck with 4 KB pages since the 386 (1985); to reach 48 KB, Zen 5 needed 12 ways (12 comparators, more energy per access)
Apple controls the whole stack (silicon + OS) → picked 16 KB pages → 4× the L1 for the same associativity, with a 3–4 cycle hit

Same idea in the L1 I-cache: 192 KB on M-series P-cores
Bonus: 4× fewer TLB misses and page-table entries for the same working set (foreshadowing the next slides)

	AMD Zen 5 / Intel Lion Cove	Apple M4 (P-core)
Page size	4 KB	16 KB
L1 D-cache	48 KB, 12-way	128 KB, 8-way
Bytes per way (= sets × block)	48 KB / 12 = 4 KB	128 KB / 8 = 16 KB
Fits the VIPT rule (≤ page size)?	Yes — exactly	Yes — exactly

Sources: Apple M4 / Arm public documentation; AMD Zen 5 disclosures; Intel Lion Cove disclosure (Hot Chips 2024).

Photos, illustrations, graphics here.

28 of 117

TLB Organization

Fully associative (L1); set-associative (L2)
Example: VA = 48 bits, page size = 4 KB, physical memory = 16 GB
VPN bits = bits(VA) − log₂(page size) = 48 − 12 = 36 bits
Physical address = log₂(16 GB) = 34 bits (22-bit PPN + 12-bit page offset)

Virtual Address

VPN offset

Compare Tags/Select PTE

Hit/Miss

Tag PTE

state

Physical Address

PPN page offset

Entries in TLB

Photos, illustrations, graphics here.

29 of 117

Life of a memory access

Common case

hardware, ~1 cycle

TLB miss

hardware, 10s–100s of cycles

Page fault

OS, microseconds to milliseconds

Load or store issues a virtual address

TLB lookup

in parallel with L1 indexing

Physical address

cache, then DRAM

hit

Page-table walk

up to 4 memory reads

Fill the TLB

retry the access

miss

valid PTE

Page fault

trap to the OS

OS maps the page

allocate or read from disk

invalid PTE

restart

Everything in this lecture is about keeping accesses on the top row.

Photos, illustrations, graphics here.

30 of 117

Selecting page size

Larger Page Size

Page table is smaller (inversely proportional to page size)
Larger page size may allow larger caches with virtually indexed, physically tagged caches (larger page offset)
Page transfers can be more efficient
More efficient TLB => reduces number of TLB misses

Smaller Page Size

Internal fragmentation: contiguous region of virtual memory not a multiple of the page size
Process startup time (load in large pages for small processes)

Multiple Page Sizes

Some processors support multiple choices => larger pages are powers of 2 times the smaller page sizes

Photos, illustrations, graphics here.

31 of 117

TLB reach and huge pages

TLB reach = entries × page size

L1 data TLB: 64 entries × 4 KB

256 KB

L2 TLB: 1,536 entries × 4 KB

6 MB

L2 TLB: 1,536 entries × 2 MB

3 GB

L2 TLB: 16 entries × 1 GB

16 GB

Intel Skylake TLB sizes; bar length on a log scale.

Huge pages (2 MB, 1 GB)

Same TLB, 512× the reach, and the walk ends a level early
Linux Transparent Huge Pages promotes 4 KB pages to 2 MB automatically
Cost: needs 2 MB of contiguous free memory, so fragmentation and compaction stalls
Cost: memory bloat, since touching one byte commits 2 MB

~26% of execution time goes to address translation on data-intensive workloads, measured on a real system

Photos, illustrations, graphics here.

32 of 117

TLB reach for an AI model

THINK · PAIR · SHARE

A model has 40 GB of weights. Each token streams through all of them.

How many TLB entries cover it with 4 KB pages? With 2 MB pages?

Recall: L2 TLB on a recent x86 core ≈ 1,536 entries

2 min think · 2 min pair. Bonus: and with 1 GB pages?

Photos, illustrations, graphics here.

33 of 117

TLB reach for an AI model: answer

ANSWER

4 KB pages

40 GB ÷ 4 KB

10,485,760

~6,800× too many

2 MB pages

40 GB ÷ 2 MB

20,480

~13× too many

1 GB pages

40 GB ÷ 1 GB

40

fits

Compared with a 1,536-entry L2 TLB.

Takeaway: huge pages are necessary, not sufficient. GPUs use 2 MB+ pages by default for this reason.

Photos, illustrations, graphics here.

34 of 117

Live demo: huge pages vs. TLB misses

LIVE DEMO

$ grep -i huge /proc/meminfo

$ cat /sys/kernel/mm/transparent_hugepage/enabled

$ ./run_demo.sh 1024

# random pointer chase over 1 GB:

# 4 KB pages, then 2 MB pages

Every load depends on the last and lands on a random page: no prefetching, no overlap
Same data, same accesses; only the page size changes
Predict first: how much faster with 2 MB pages?

4 KB pages

2 MB pages

Pages spanned

262,144

512

Time per load

~230 ns

~160 ns

Reach of 1,536-entry TLB

6 MB

3 GB

Times measured on a cloud VM; bare metal shows a similar gap. What remains is the DRAM access itself. Under perf, only the timed loads are counted.

Photos, illustrations, graphics here.

35 of 117

Live demo: when do huge pages help?

LIVE DEMO

Dependent loads over 1 GB (ns per load)

Pattern

Each load goes to

4 KB

2 MB

Gain

seq

next 64 B line

5.6

4.9

1.15×

stride

next 4 KB page

153

147

1.05×

random

random page

222

162

1.37×

seq: one TLB miss per 64 loads, already cheap
stride: a miss on every load, yet small gain: neighboring PTEs share a cache line, so each walk hits in cache
random: walks miss in cache too, so huge pages pay off

When small pages win

Reserve 2 GB, touch 1 byte every 2 MB (a sparse table, a big malloc used lightly)

Memory used

4 MB

4 KB pages

2,048 MB

2 MB pages

Time to touch

2 ms

4 KB pages

416 ms

2 MB pages

Each touch commits and zeroes a whole 2 MB page: 512× the memory. Many distros default THP to madvise for this reason.

Photos, illustrations, graphics here.

36 of 117

Ongoing virtual memory research

… the data-intensive Graph500 benchmark… has an approximate working set size of 215 MiB, whereas a typical TLB using 4 KiB pages can only cache translations for about 8.6 MiB of physical memory at once. Some of these applications, such as graph analytics, also tend to have irregular, pointer-based memory traversals with poor locality of reference, thwarting common heuristics such as prefetching. As a result, many modern applications report 20–30% overhead attributable to TLB misses [19, 20, 32, 54], and some as high as 83% [5].

Moreover, TLBs tend to be power-hungry, consuming 3-13% of a processor’s power, and higher associativity and deeper TLB hierarchies further increase dynamic energy usage [29].

Photos, illustrations, graphics here.

37 of 117

Virtual machines: the 2D page walk

Two translations: guest virtual → guest physical → host physical
Worst case (4+1) × (4+1) − 1 = 24 memory reads per TLB miss, against 4 natively
Hardware support: Intel EPT, AMD NPT
The TLB caches the end-to-end mapping, so hits cost the same
Misses are what hurt: huge pages and page-walk caches matter even more in the cloud

host L4

host L3

host L2

host L1

read

guest L4

1

2

3

4

5

guest L3

6

7

8

9

10

guest L2

11

12

13

14

15

guest L1

16

17

18

19

20

data

21

22

23

24

data

Each guest page-table address is guest-physical, so it needs its own host walk.

Photos, illustrations, graphics here.

38 of 117

Modern example: paging for LLM serving

PagedAttention (vLLM; Kwon et al., SOSP 2023)

Problem: each request’s KV cache grows token by token. Contiguous allocation wastes scarce GPU memory.

OS virtual memory

LLM serving

Page

KV-cache block (fixed number of tokens)

Process

Request

Page table

Block table: logical → physical blocks

Copy-on-write

Requests share a common prompt prefix

Fixed-size blocks allocated on demand: no external fragmentation, little internal.

“one can think of blocks as pages, tokens as bytes, and requests as processes”

Photos, illustrations, graphics here.

39 of 117

5 minute break!

Photos, illustrations, graphics here.

40 of 117

The memory system

DRAM

CPU

Memory controller

Last-level cache (LLC)

L1

Core

TLB

64-bit memory bus

Sends commands to DRAM

Issues memory requests to memory controller

Issues loads and store instructions

Virtual to physical memory translation

Photos, illustrations, graphics here.

41 of 117

Why DRAM? SRAM vs. DRAM vs. flash

	SRAM	DRAM	NAND flash
Cell	6 transistors	1 transistor + 1 capacitor	1 transistor (trapped charge)
Access latency	~1 ns	~50 ns	~50–100 µs
Cost per GB	< $300	< $6	< $0.08
Keeps data without power	No	No, and it leaks	Yes
Used for	Caches, registers	Main memory	Storage (SSD)

DRAM: ~50× cheaper per bit than SRAM, ~1,000× faster than flash. The price of one transistor and one capacitor: slow, leaky, destructive reads.

Sample values circa 2023. Source: O. Mutlu, DDCA Spring 2026, Lecture 22.

Photos, illustrations, graphics here.

42 of 117

DRAM Array

1 transistor & capacitor per “bit” (recall from physics a capacitor stores charge)

(to memory controller)

Data pins (8 bits)

Row buffer (2 Kbits)

2 Kbits per row

Photos, illustrations, graphics here.

43 of 117

DRAM Array

1 transistor & capacitor per “bit” (recall from physics a capacitor stores charge)

(to memory controller)

Data pins (8 bits)

Row buffer (2 Kbits)

2 Kbits per row

We want to read this byte

Photos, illustrations, graphics here.

44 of 117

DRAM Array

1 transistor & capacitor per “bit” (recall from physics a capacitor stores charge)

(to memory controller)

Data pins (8 bits)

Row buffer (2 Kbits)

2 Kbits per row

1. Precharge: ready bit lines (~10ns)

We want to read this byte

Photos, illustrations, graphics here.

45 of 117

DRAM Array

1 transistor & capacitor per “bit” (recall from physics a capacitor stores charge)

(to memory controller)

Data pins (8 bits)

Row buffer (2 Kbits)

2 Kbits per row

2. Row activation (~10ns)

1. Precharge: ready bit lines (~10ns)

We want to read this byte

Transfer row

Photos, illustrations, graphics here.

46 of 117

DRAM Array

1 transistor & capacitor per “bit” (recall from physics a capacitor stores charge)

(to memory controller)

Data pins (8 bits)

Row buffer (2 Kbits)

2 Kbits per row

2. Row activation (~10ns)

1. Precharge: ready bit lines (~10ns)

We want to read this byte

3. Column selection

4. Transfer data onto bus

10ns

Transfer row

Photos, illustrations, graphics here.

47 of 117

Load next byte from (already active) row

Lower latency operation: can skip pre-charge and row activation steps

(to memory controller)

Data pins (8 bits)

Row buffer (2 Kbits)

2 Kbits per row

1. Column selection

2. Transfer data onto bus

10ns

Photos, illustrations, graphics here.

48 of 117

Inside the cell: destructive reads

Opening the row shares the cell’s tiny charge with a long bitline: cell drained, bitline barely moves
The sense amplifier detects that swing and drives the bitline to a full 0 or 1, which rewrites the cell
Hence the timing rules: sense before reading (tRCD), restore before closing (tRAS), re-center (tRP)

wordline (row select)

bitline

T

storage capacitor (charged = 1)

access transistor

sense amp

Vdd

Vdd/2

bitline voltage, reading a 1

precharged

share charge

sense + amplify

restore cell

precharge

ACT

READ

PRE

Photos, illustrations, graphics here.

49 of 117

Tiny capacitors: refresh and errors

Refresh

one refresh command every 7.8 µs; the chip is busy during each

Charge leaks away within milliseconds
Every row is read and restored every 64 ms (32 ms when hot)
DDR4 8 Gb chip: busy ~350 ns of every 7.8 µs, about 4.5% of the time
Cost grows with capacity, and dominates DRAM power when idle

Errors and ECC

64 data bits

8

ECC DIMM: 8 check bits per 64-bit word, stored in a ninth chip

Smaller cells hold fewer electrons: more weak cells, more bit flips
ECC corrects any 1-bit error and detects 2-bit errors
Standard in servers; DDR5 also adds ECC inside each chip
RowHammer: repeatedly opening one row can flip bits in its neighbors

Photos, illustrations, graphics here.

50 of 117

DRAM access latency is not fixed

Best case latency: read from active row

Column access time (CAS latency, CL)

Worst case latency: bit lines not ready, read from new row

Precharge (tRP) + row activate (tRCD) + column access (CL)
Precharge also writes the row buffer back into the array: reads are destructive

Question 1: when to execute precharge?

After each column access?
Only when new row is accessed?

Question 2: how to handle latency of DRAM access?

Photos, illustrations, graphics here.

51 of 117

Problem: low data-pin utilization

Data pins (8 bits)

PRE

RAS

CAS

PRE

RAS

CAS

PRE

RAS

CAS

Access 1

Access 2

Access 3

Access 4

Data pins in use only a small fraction of time

(red = data pins busy)

This is bad since they are the scarcest resource!

Photos, illustrations, graphics here.

52 of 117

DRAM burst mode

Data pins (8 bits)

PRE

RAS

CAS

Access 1

Idea: amortize latency over larger transfers

Each DRAM command describes bulk transfer

Bits placed on output pins in consecutive clocks

Rest of transfer

PRE

RAS

CAS

Rest of transfer

Access 2

Photos, illustrations, graphics here.

53 of 117

DRAM chip consists of multiple banks

Data pins (8 bits)

PRE

RAS

CAS

All banks share the same pins (only one transfer at a time)
Banks allow for pipelining of memory requests

Pre-charge/active rows/send column address to one bank while transferring data from another
Achieves high data pin utilization

PRE

RAS

CAS

PRE

RAS

CAS

PRE

RAS

CAS

Bank 0

Bank 1

Bank 2

Bank 3

time

Banks 0-3

Photos, illustrations, graphics here.

54 of 117

Multiple chips form a DIMM

CPU

Memory Controller

Last-level cache (LLC)

Read bank B, row R, column 0

64 bit memory bus

Example: Eight DRAM chips (64-bit memory bus)
Note: DIMM appears as a single, higher capacity, wider interface DRAM module to the memory controller. Higher aggregate bandwidth, but minimum transfer granularity is now 64-bits

Photos, illustrations, graphics here.

55 of 117

Reading a 64-byte line the wrong way

CPU

Memory Controller

Last-level cache

(LLC)

Read bank B, row R, column 0

64 bit memory bus

Assume: consecutive physical addresses mapped to same row of the same chip
Memory controller converts physical address to DRAM bank, row, column

Request line w/physical address X

Bits 0:7

Photos, illustrations, graphics here.

56 of 117

Reading a 64-byte line the wrong way

CPU

Memory Controller

Last-level cache

(LLC)

Read bank B, row R, column 0

64 bit memory bus

All data for cache line serviced by the same chip
Bytes sent consecutively over same pins

Request line w/physical address X

Bits 8:15

Photos, illustrations, graphics here.

57 of 117

Reading a 64-byte line the wrong way

CPU

Memory Controller

Last-level cache

(LLC)

Read bank B, row R, column 0

64 bit memory bus

All data for cache line serviced by the same chip
Bytes sent consecutively over same pins

Request line w/physical address X

Bits 16:23

Photos, illustrations, graphics here.

58 of 117

Reading a 64-byte line the wrong way

CPU

Memory Controller

Last-level cache

(LLC)

Read bank B, row R, column 0

64 bit memory bus

All data for cache line serviced by the same chip
Bytes sent consecutively over same pins

Request line w/physical address X

Bits 24:31

Photos, illustrations, graphics here.

59 of 117

Reading a 64-byte line the efficient way

CPU

Memory Controller

Last-level cache

(LLC)

Read bank B, row R, column 0

64 bit memory bus

Cache miss of line X

Bits 0:7

Bits 8:15

Bits 16:23

Bits 24:31

Bits 32:39

Bits 40:47

Bits 48:55

Bits 56:63

Memory controller converts physical address to DRAM bank, row, column
Here: physical addresses are interleaved across DRAM chips at byte granularity
DRAM chips transmit first 64 bits in parallel

Photos, illustrations, graphics here.

60 of 117

Reading a 64-byte line the efficient way

CPU

Memory Controller

Last-level cache

(LLC)

Read bank B, row R, column 0

64 bit memory bus

DRAM controller requests data from new column

Recall modern DRAM’s support burst mode transfer of multiple consecutive columns, which would be used here

DRAM chips transmit next 64 bits in parallel

Cache miss of line X

Bits 64:71

Bits 72:79

Bits 80:87

Bits 88:95

Bits 96:103

Bits 104:111

Bits 112:119

Bits 120:127

Photos, illustrations, graphics here.

61 of 117

Memory controller: a request scheduler

Receives load/store requests from LLC
Conflicting scheduling goals

Maximize throughput, minimize latency, minimize energy consumption
Common scheduling policy: FR-FCFS (first-ready, first-come-first serve)

Services requests to currently open row first (maximize row locality)
Service requests to other rows in FIFO order

Controller may coalesce multiple small requests into large contiguous requests (to take advantage of DRAM “burst mode”)

Photos, illustrations, graphics here.

62 of 117

Dual-channel memory system

Increase throughput by adding memory channels (effectively widen bus)
Below: each channel can issue independent commands

Different row/column is read in each channel
Simpler setup: use single controller to drive same command to multiple channels

Photos, illustrations, graphics here.

63 of 117

The full hierarchy: channel to cell

Channel

2–12 per CPU

Independent bus: more channels, more bandwidth

›

DIMM

1–2 per channel

Shares the channel: adds capacity only

›

Rank

1–4 per DIMM

Chips that answer together to form a 64-bit word

›

Chip

8–18 per rank

Supplies 4 or 8 of the data bits

›

Bank

16–32 per chip

Own row buffer: requests can overlap

›

Row/column

128K × 1K per bank

Open row: fast repeat access

Example: 32 GB DDR4 DIMM = 2 ranks × 8 chips × 16 Gb (2 GB) per chip

Each chip = 16 banks × 128K rows × 1K columns × 8 bits. With ECC, add a ninth chip per rank.

Rank is the level we had not named: the set of chips selected together on one channel.

Photos, illustrations, graphics here.

64 of 117

Live demo: what DRAM is in this server?

LIVE DEMO

$ ./memory_info.sh

(wraps: sudo dmidecode -t memory)

Locator: DIMM_A1

Size: 32 GB

Type: DDR5

Speed: 4800 MT/s

Rank: 2

Total Width: 72 bits Data Width: 64 bits

Illustrative output; fields vary by vendor.

Rank: 2: two sets of chips share this DIMM
72 vs. 64 bits: the extra 8 are ECC check bits
4800 MT/s × 8 B = 38.4 GB/s per channel
Count the Locators: populated slots tell you the channels
Ask the class: peak bandwidth of this server?

Photos, illustrations, graphics here.

65 of 117

Example: DDR5 and the bandwidth wall

DDR = double data rate: data moves on both clock edges
DDR5-4800: 64 data bits × 2.4 GHz × 2 = 38.4 GB/s per channel
Label timings CL-tRCD-tRP, e.g. 40-39-39: CL ≈ 16.7 ns. DDR4-2400 (16-16-16) was ≈ 13 ns

System	Cores	Channels	Peak bandwidth	Per core
Core i7-7700K (2017), DDR4-2400	4	2	38 GB/s	9.6 GB/s
Xeon Platinum 8480+ (2023), DDR5-4800	56	8	307 GB/s	5.5 GB/s
EPYC 9654 (2022), DDR5-4800	96	12	461 GB/s	4.8 GB/s
EPYC 9965 (2024), DDR5-6400	192	12	614 GB/s	3.2 GB/s

Bandwidth wall: core counts grow faster than memory channels, so each core gets less. Latency in nanoseconds has not improved either.

Photos, illustrations, graphics here.

66 of 117

DRAM Summary

DRAM access latency can depend on many low-level factors

State of DRAM chip: row hit/miss? Is precharge necessary?
Buffering/reordering of requests in memory controller

One transistor + one capacitor per bit: cheap and dense, but leaky

Destructive reads, periodic refresh, error correction

Much of the complexity of a modern multi-core processor has moved into the memory controller

Schedules tens to hundreds of outstanding memory requests
Maps physical addresses to the geometry of the DRAM
Area of active computer architecture research!

Photos, illustrations, graphics here.

67 of 117

Capacity scaled; latency did not

Response: move memory closer to the processor

Enables shorter, but wider, interfaces

Capacity

128×

denser chips: what the market paid for

Bandwidth

20×

faster interfaces

Latency

1.3×

access time barely moved

Improvement in DRAM chips, 1999–2017. Source: K. Chang, PhD thesis, CMU 2017 (via O. Mutlu’s lectures).

Photos, illustrations, graphics here.

68 of 117

3D stacking: more bandwidth, less power

Enabling technology: 3D stacking of DRAM chips

DRAMs connected via through-silicon-vias (TSV’s) that run through chips
TSV’s provide highly parallel connection between logic and DRAM
Base layer of stack “logic layer” is memory controller, manages requests from processor
Silicon “interposer” serves as high-bandwidth interconnect between DRAM stack and processor

Technologies:

Hybrid Memory Cube (HMC): Micron/Intel, discontinued
High-bandwidth memory (HBM): today’s standard

1,024-bit interface per stack (2,048 in HBM4)

Photos, illustrations, graphics here.

69 of 117

GPUs adopted HBM early (2015–16)

Photos, illustrations, graphics here.

70 of 117

Why AI accelerators use HBM

Generating one token reads every weight of the model
tokens/s ≲ bandwidth ÷ model size
40 GB of weights: about 15 tokens/s on the CPU, 120 on the GPU, per sequence
The price: capacity fixed at build time, higher cost per GB, tight supply
More when we cover accelerators

Peak memory bandwidth (GB/s)

One DDR5-4800 channel

38

Server CPU, 12 × DDR5-6400

614

One HBM3E stack

1,200

NVIDIA H200 GPU, 141 GB of HBM3E

4,800

HBM3E: 1,024 data wires per stack at about 9 Gb/s each.

Photos, illustrations, graphics here.

71 of 117

CXL: memory beyond the DIMM slot

Compute Express Link: DRAM attached over PCIe; the OS sees a NUMA node without CPUs
Expansion: more capacity and bandwidth than the DIMM slots allow
Pooling: servers share a pool and recover stranded memory (up to 25% at Azure)
Re-use: older DDR4 behind CXL lowers cost and embodied carbon
Now the OS must tier hot and cold pages

Server

CPU + local DDR5

~100 ns

Server

CPU + local DDR5

~100 ns

Server

CPU + local DDR5

~100 ns

CXL memory pool

shared DRAM, can be re-used DDR4

CXL link over PCIe wires: adds roughly 70–90 ns, about one NUMA hop.

Photos, illustrations, graphics here.

72 of 117

Before you go

2 MINUTES

Turn to your neighbor:

one thing that surprised you today,

one thing still fuzzy.

Write the fuzzy one on a sticky note on your way out, no names
Or post it on Ed anytime, anonymously if you like
Next lecture starts with the most common ones

Why don’t TLBs just get bigger?

Who decides when to use huge pages?

Is refresh why my laptop drains when asleep?

Example notes

Photos, illustrations, graphics here.

73 of 117

5 minute break!

Photos, illustrations, graphics here.

74 of 117

The class so far…

DRAM

CPU

Memory controller

Last-level cache (LLC)

L1

Core

TLB

Photos, illustrations, graphics here.

75 of 117

The class so far…

DRAM

CPU

Memory controller

Last-level cache (LLC)

L1

Core

TLB

Photos, illustrations, graphics here.

76 of 117

The class so far…

DRAM

CPU

Memory controller

Last-level cache (LLC)

L1

Core

TLB

Speculative execution with branch prediction

Photos, illustrations, graphics here.

77 of 117

The class so far…

DRAM

CPU

Memory controller

Last-level cache (LLC)

L1

Core

TLB

Out-of-order execution further improves ILP

Photos, illustrations, graphics here.

78 of 117

The class so far…

DRAM

CPU

Memory controller

Last-level cache (LLC)

L1

Core

TLB

Cache architecture to minimize misses

{Capacity, conflict, compulsory}

Photos, illustrations, graphics here.

79 of 117

The class so far…

DRAM

CPU

Memory controller

Last-level cache (LLC)

L1

Core

TLB

Virtual memory & TLB’s

Photos, illustrations, graphics here.

80 of 117

The class so far…

DRAM

CPU

Memory controller

Last-level cache (LLC)

L1

Core

TLB

DRAM Architecture

Photos, illustrations, graphics here.

81 of 117

Current processor

Fetch/

Decode

ALU

Execution unit

Execution Context

Register 0

Register 1

Register 2

Register 3

Register 4

Register 5

L1 Instruction Cache

L1 Data

Cache

L2 cache

Mem controller

Photos, illustrations, graphics here.

82 of 117

Parallelism: Instruction level parallelism

Fetch/

Decode

ALU

Execution unit

Execution Context

Register 0

Register 1

Register 2

Register 3

Register 4

Register 5

L1 Instruction Cache

L1 Data

Cache

L2 cache

Mem controller

What we’ve talked about already:

Pipelining
Speculation: branch prediction
Out-order execution

Photos, illustrations, graphics here.

83 of 117

Parallelism: Instruction level parallelism

Fetch/

Decode

ALU

Execution unit

Execution Context

Register 0

Register 1

Register 2

Register 3

Register 4

Register 5

L1 Instruction Cache

L1 Data

Cache

L2 cache

Mem controller

What we’ve talked about already:

Pipelining
Speculation: branch prediction
Out-order execution

So far we have looked at scalar pipelines

One instruction per stage
With control speculation, bypassing, etc.

Performance limit CPI = IPC = 1

Limit is never even achieved (hazards)
Diminishing returns from “super-pipelining” (hazards + overhead)

Photos, illustrations, graphics here.

84 of 117

Improving performance via parallelism

Execution time_new= Execution time_old× ( (1 - Fraction_enhanced) + )

Fraction_enhanced

Speedup_enhanced

Overall

Speedup

Example: Achieve speedup of 80x using 100 processors

1/80 = [Frac_parallel/100+1-Frac_parallel]
Frac_parallel= 0.9975 = only .25% of the work can be serial!

Photos, illustrations, graphics here.

85 of 117

Parallel Application Domains

Traditional Parallel Programs
True parallelism in one job

Regular loop structures
Data usually tightly shared
Automatic parallelization
Data-level parallelism
Can often exploit vectors as well

Workloads

Scientific simulation codes (FFT, weather, fluid dynamics)
Dominant market segment in the 1980s

Photos, illustrations, graphics here.

86 of 117

Parallelism: Superscalar processors

Fetch/

Decode

ALU

Execution unit

Execution Context

Register 0

Register 1

Register 2

Register 3

Register 4

Register 5

L1 Instruction Cache

L1 Data

Cache

L2 cache

Mem controller

Fetch/

Decode

ALU

Execution unit

Out-of-order control logic

Superscalar execution: processor automatically finds independent instructions in an instruction sequence and can execute them in parallel on multiple execution units

Photos, illustrations, graphics here.

87 of 117

Typical dual-issue pipeline

Fetch an entire 16B or 32B instruction cache block

4 to 8 instructions (assuming 4-byte fixed length instructions)
Predict a single branch per cycle

Parallel decode

Need to check for conflicting instructions
Output of I₁ is an input to I₂
Other stalls, too (for example, load-use delay)

What are the added costs of this design?

Photos, illustrations, graphics here.

88 of 117

Typical dual-issue pipeline

Multi-ported register file

Larger area, latency, power, cost, complexity

Multiple execution units

Simple adders are easy, but bypass paths are expensive

Memory unit

Single load per cycle (stall at decode) probably okay for dual issue
Alternative: add a read port to data cache

Larger area, latency, power, cost, complexity

Photos, illustrations, graphics here.

89 of 117

Superscalar challenges – front end

Wide instruction fetch

Modest: need multiple instructions per cycle
Aggressive: predict multiple branches, trace cache

Wide instruction decode

Replicate decoders

Wide instruction issue (submit for execution)

Determine when instructions can proceed in parallel
Not all combinations possible
More complex stall logic - order N²for N-wide machine

Wide register read

One port for each register read

Each port needs its own set of address and data wires

Example, 4-wide superscalar 🡺 8 read ports

Photos, illustrations, graphics here.

90 of 117

Superscalar challenges – back end

Wide instruction execution

Replicate arithmetic units
Multiple cache ports

Wide instruction register writeback

One write port per instruction that writes a register
Example, 4-wide superscalar 🡺 4 write ports

Wide bypass paths

More possible sources for data values
Order (N² * P)for N-wide machine with execute pipeline depth P

Fundamental challenge:

Amount of ILP (instruction-level parallelism) in the program
Compiler must schedule code and extract parallelism

Photos, illustrations, graphics here.

91 of 117

Pre multi-core era processor

Majority of chip transistors used to perform operations that help make a single instruction stream fast

Fetch/

Decode

ALU

Execution unit

Execution Context

Register 0

Register 1

Register 2

Register 3

Register 4

Register 5

Data cache

Fetch/

Decode

ALU

Execution unit

Out-of-order control logic

Sophisticated branch predictor

Memory pre-fetcher

More transistors = larger cache, smarter out-of-order logic, smarter branch predictor

Photos, illustrations, graphics here.

92 of 117

Computation Taxonomy

Proposed by Flynn (1966)
Dimensions:

Instruction streams: single (SI) or multiple (MI)
Data streams: single (SD) or multiple (MD)

Cross-product:

SISD: uniprocessors
SIMD: multimedia/vector processors (SSE, AVX-512)
MISD: no real examples (Systolic arrays are debatable)
MIMD: multiprocessors + multicomputers

Photos, illustrations, graphics here.

93 of 117

Multi-core era processors

Execution Context

Register 0

Register 1

Register 2

Register 3

Register 4

Register 5

Fetch/

Decode

ALU

Execution unit

Rather than use transistors to increase sophistication of processor logic that accelerates a single instruction stream

(e.g., out-of-order, speculation)

Use increasing transistor count to add more cores to the processor

Photos, illustrations, graphics here.

94 of 117

Multi-core era processors

Execution Context

Register 0

Register 1

Register 2

Register 3

Register 4

Register 5

Fetch/

Decode

ALU

Execution unit

Execution Context

Register 0

Register 1

Register 2

Register 3

Register 4

Register 5

Fetch/

Decode

ALU

Execution unit

Simpler cores: each core may be slower at running a single instruction stream than a more sophisticated core (e.g., 25% slower)

But there are now two cores: 2 * 0.75 = 1.5x (potential for speedup!)

Photos, illustrations, graphics here.

95 of 117

Four cores: compute four elements in parallel

Photos, illustrations, graphics here.

96 of 117

Sixteen cores: compute sixteen elements in parallel

Photos, illustrations, graphics here.

97 of 117

Photos, illustrations, graphics here.

98 of 117

Photos, illustrations, graphics here.

99 of 117

Photos, illustrations, graphics here.

100 of 117

Photos, illustrations, graphics here.

101 of 117

Another idea: Add more execution units to increase compute capacity

Execution Context

Register 0

Register 1

Register 2

Register 3

Register 4

Register 5

Fetch/

Decode

ALU 0

Another idea…

Amortize cost/complexity of managing an instruction stream across many ALU’s

SIMD processing

Single instruction, multiple data

Same instruction broadcast to all ALUs

This operation is executed in parallel on all ALUs

ALU 0

Photos, illustrations, graphics here.

102 of 117

16 SIMD cores: 128 elements in parallel

Photos, illustrations, graphics here.

103 of 117

Common terminology

Instruction stream coherence (“coherent execution”)

Property of a program where the same instruction sequence applies to many data elements
Coherent execution IS REQUIRED for SIMD processing resources to be used efficiently
Coherent execution IS NOT REQUIRED for efficient parallelization across different cores, since each core has the capability to fetch/decode instructions from their thread’s instruction stream

Divergent execution

A lack of instruction stream coherence

Photos, illustrations, graphics here.

104 of 117

SIMD execution: modern CPU examples

Intel AVX2 instructions: 256 bit operations: 8x32 bits or 4x64 bits (8-wide float vectors)
Intel AVX512 instruction: 512 bit operations: 16x32 bits …
ARM Neon instructions: 128 bit operations: 4x32 bits…

Instructions are generated by the compiler

Parallelism explicitly requested by programmer using intrinsic
Parallelism conveyed using parallel language semantics
Parallelism inferred by dependency analysis of loops by auto-vectorizing compiler

Photos, illustrations, graphics here.

105 of 117

SIMD execution: modern GPU examples

“Implicit SIMD”

Compiler generates a binary with scalar instructions
But N instances of the program are always run together on the processor (coherence)
Hardware (not compiler) is responsible for simultaneously executing the same instruction from multiple program instances on different data on SIMD ALUs

SIMD width of most modern GPUs ranges from 8 to 32

Divergent execution can be a big issue (poorly written code might execute at 1/32 the peak capability of the machine!)

Photos, illustrations, graphics here.

106 of 117

Photos, illustrations, graphics here.

107 of 117

Photos, illustrations, graphics here.

108 of 117

SIMD vs. MIMD

MP vs. SIMD

Programming model flexibility

Could simulate vectors with MIMD but not the other way
Dominant market segment cannot use vectors

Again this is changing!

Cost effectiveness

Commodity Parts: high volume (cheap) components
MPs can be made up of many uniprocessors
Allows easy scalability from small to large

Photos, illustrations, graphics here.

109 of 117

Taxonomy of Parallel (MIMD) Processors

Center on organization of main memory

Shared vs. Distributed

Appearance of memory to hardware

Q1: Memory access latency uniform?
Shared (UMA): yes, doesn’t matter where data goes
Distributed (NUMA): no, makes a big difference

Appearance of memory to software

Q2: Can processors communicate directly via memory?
Shared (shared memory): yes, communicate via load/store
Distributed (message passing): no, communicate via messages

Dimensions are orthogonal

e.g. DSM: (physically) distributed, (logically) shared memory

Photos, illustrations, graphics here.

110 of 117

UMA vs. NUMA: Why it matters

Ideal model:

Perfect (single-cycle) memory latency
Perfect (infinite) memory bandwidth

Real systems:

Latencies are long and grow with system size
Bandwidth is limited

Add memory banks, interconnect to hook up (latency goes up)

Photos, illustrations, graphics here.

111 of 117

UMA vs. NUMA: Why it matters

UMA: uniform memory access

From p0 same latency to m0-m3
Data placement doesn’t matter
Latency worse as system scales
Interconnect contention restricts bandwidth
Small multiprocessors only

NUMA: non-uniform memory access

From p0 faster to m0 than m1-m3
Low latency to local memory helps performance
Data placement important (software!)
Less contention = more scalable
Large multiprocessor systems

Photos, illustrations, graphics here.

112 of 117

Interconnection Networks

Need something to connect those processors/memories

Direct: endpoints connect directly
Indirect: endpoints connect via switches/routers

Interconnect issues

Latency: average latency more important than max
Bandwidth: per processor
Cost: #wires, #switches, #ports per switch
Scalability: how latency, bandwidth, cost grow with processors

Interconnect topology primarily concerns architects

Photos, illustrations, graphics here.

113 of 117

Interconnect 1: Bus

Direct Interconnect Style

Cost: f(1) wires
Latency: f(1)
Bandwidth: Not Scalable, f(1/P)

Only works in small systems (<=4)

Only performs ordered broadcast

No other protocols work

Photos, illustrations, graphics here.

114 of 117

Interconnect 2: Crossbar Switch

Indirect Interconnect

Switch implemented as big MUXes

Latency: f(1)
Bandwidth: f(1)
Cost:

f(2P) wires
F(P²) switches
4 wires per switch

Photos, illustrations, graphics here.

115 of 117

Interconnect 3: Multistage Network

Indirect Interconnect

Routing done by address decoding
k: switch arity (#inputs/#outputs per switch)
d: number of network stages = log_k P

Cost

f(d*P/k) switches
f(P*d) wires
f(k) wires per switch

Latency: f(d)
Bandwidth: f(1)
Commonly used in large UMA systems

Photos, illustrations, graphics here.

116 of 117

Interconnect 4: 2D Torus

Direct Interconnect
Cost

f(2P) wires
4 wires per switch

Latency: f(P^1/2)
Bandwidth: f(1)
Good scalability (widely used)

Variants: 1D (ring), 3D, mesh (no wraparound)

Photos, illustrations, graphics here.

117 of 117

Interconnect 5: Hypercube

Direct Interconnect

K: arity (#nodes per dimension)
D: dimension =log_k P

Latency: f(d)
Bandwidth: f(k*d)
Cost

F(k*d*P) wires
F(k*d) wires per switch

Good scalability, expensive switches

new design needed for more nodes

Photos, illustrations, graphics here.