Lecture Title:�Memory Hierarchy: Primary and Auxiliary Memory Devices

Presenter: Anvar Abdullayev, Lecturer, Faculty of Computer Engineering

Date: September 22, 2025

University: Urgench State University (UrSU)

​

Agenda

• Introduction to Computer Memory
• The Need for Memory Hierarchy
• Key Characteristics of Memory Devices
• Levels of Memory Hierarchy: Overview
• Primary Memory: Registers
• Primary Memory: Cache
• Primary Memory: Main Memory (RAM)
• Auxiliary Memory: Secondary Storage
• Auxiliary Memory: Tertiary Storage
• Comparison of Memory Levels
• Performance and Access Patterns
• Virtual Memory Concepts
• Real-World Examples and Case Studies
• Future Trends in Memory Technology
• Challenges and Optimizations
• Summary
• Q&A

Introduction to Computer Memory

What is Computer Memory? Memory is the component that stores data and instructions for the CPU to access quickly. Without it, computers couldn't execute programs efficiently.

• Role in Von Neumann Architecture: Computers follow the Von Neumann model: CPU fetches instructions and data from memory, processes them, and stores results back.
• Types Overview:
• Primary: Fast, volatile, directly accessible by CPU.
• Auxiliary: Slower, non-volatile, for long-term storage.
• Historical Context: Early computers used vacuum tubes and magnetic drums; modern ones leverage semiconductors for speed.

The Need for Memory Hierarchy

• The Memory Gap Problem: CPU speed doubles every 18 months (Moore's Law), but memory speed lags behind—creating a "gap" where CPU waits idly for data.
• Why Hierarchy?
• Balance cost, speed, and capacity: Use small fast memory for frequent data, large slow for bulk storage.
• Principle of Locality: Programs exhibit temporal (reuse recent data) and spatial (access nearby data) locality.
• Benefits:
• Improves overall system performance by 10-100x.
• Cost-effective: Expensive fast memory in small quantities, cheap slow in large.

​

Key Characteristics of Memory Devices

• Speed (Access Time): Time to read/write data (ns for registers, ms for disks).
• Capacity: Amount of data stored (bytes to TB).
• Cost per Bit: Decreases with size (e.g., $0.000001/GB for HDD vs. $0.01/GB for cache).
• Volatility: Loses data without power (RAM) vs. retains (flash).
• Bandwidth: Data transfer rate (GB/s).

Levels of Memory Hierarchy - Overview

• Hierarchy Pyramid (Top to Bottom):
• Registers: Inside CPU, fastest.
• L1/L2/L3 Cache: On-chip, for recent data.
• Main Memory (RAM): System-wide, volatile.
• Secondary Storage (HDD/SSD): Non-volatile, mass storage.
• Tertiary Storage (Tapes/Cloud): Archival, offline.
• Hit/Miss Ratio: Success rate of finding data in upper levels (aim for 95%+ cache hit).

Primary Memory - Registers

• Definition: Small, high-speed storage locations within the CPU (e.g., 32-64 registers in x86).
• Characteristics:
• Access Time: <1 ns.
• Capacity: 64-512 bits per register.
• Used for: Temporary storage during arithmetic/logic operations.
• Types: General-purpose (e.g., AX in x86), special (e.g., Program Counter for next instruction).
• Example: In assembly: MOV AX, 5 loads value into register AX.

Primary Memory - Cache

• Purpose: Bridge between CPU and RAM; stores copies of frequently used data.
• Levels:
• L1: Per-core, split I-cache/D-cache (32-64 KB).
• L2: Per-core or shared (256 KB-2 MB).
• L3: Shared across cores (8-64 MB).
• Mapping Techniques: Direct-mapped, Set-associative, Fully-associative.
• Replacement Policies: LRU (Least Recently Used) to evict old data.

Primary Memory - Main Memory (RAM)

• Definition: Random Access Memory; volatile semiconductor storage.
• Types:
• DRAM: Dynamic (needs refresh, cheaper, e.g., DDR5 up to 64 GB/s).
• SRAM: Static (faster, used in cache, more expensive).
• Organization: Addressed in bytes; ECC for error correction in servers.
• Access: Row/column addressing; latency ~50-100 ns.

Auxiliary Memory - Secondary Storage

• Role: Non-volatile, high-capacity for OS, files, programs.
• HDD (Hard Disk Drive):
• Magnetic platters, mechanical arms.
• Capacity: Up to 20 TB; Speed: 100-200 MB/s; Latency: 5-10 ms seek.
• SSD (Solid State Drive):
• Flash NAND cells, no moving parts.
• Capacity: 1-8 TB; Speed: 500-7000 MB/s; Wear-leveling for endurance.

​

Auxiliary Memory - Tertiary Storage

• Definition: Offline/archival storage for rarely accessed data.
• Examples:
• Magnetic Tapes: Linear access, 10-50 TB/cartridge; Cost-effective for backups.
• Optical Discs: CDs/DVDs/Blu-ray (up to 100 GB); Read-only.
• Cloud Storage: Distributed (e.g., AWS S3), virtually unlimited.
• Access: Sequential (tapes) vs. random (cloud via API). Latency: Minutes to hours.

Comparison of Memory Levels

Performance and Access Patterns

• Locality Principle in Action:
• Temporal: If data accessed, likely reused soon → Cache it.
• Spatial: If address accessed, nearby likely next → Prefetch blocks.
• Metrics:
• Hit Rate: % data found in cache (e.g., 90% L1).
• Miss Penalty: Extra time for lower-level access (e.g., 200 cycles for RAM miss).
• AMAT Formula: Average Memory Access Time = Hit Time + Miss Rate × Miss Penalty.

AMAT = H_t + (MR × MP)

↑ ↑ ↑

Hit Miss Miss

Time Rate Penalty

​

Example Calc:

H_t = 1 cycle

MR = 10% (0.1)

MP = 200 cycles

-------------------

AMAT = 1 + (0.1 × 200) = 21 cycles

[Slider Widgets: Drag MR to 5% → AMAT = 11 cycles]

Virtual Memory Concepts

• What is Virtual Memory? Extends RAM using secondary storage; allows programs > physical RAM.
• Mechanisms:
• Paging: Divides memory into fixed pages (4 KB); maps virtual to physical.
• Segmentation: Variable-sized segments for code/data.
• Page Table: Lookup structure in RAM; TLB (Translation Look aside Buffer) caches translations.
• Thrashing: Excessive swapping degrades performance—avoid with working set model.

Real-World Examples

• CPU Cache in Intel Core i9: L1: 128 KB/core, L3: 36 MB shared.
• RAM in Servers: 1-4 TB ECC DDR5 for data centers.
• SSD in Laptops: 512 GB NVMe for fast boot/OS.
• Case Study: Netflix Data Pipeline: Uses SSD for hot content, HDD for cold archives, cloud for scaling.

Future Trends in Memory Technology

• Emerging Tech:
• Optane (3D XPoint): Faster than NAND, persistent like DRAM.
• HBM (High Bandwidth Memory): Stacked for GPUs (up to 1 TB/s).
• CXL (Compute Express Link): Coherent memory pooling across devices.
• Challenges: Quantum limits on scaling; energy efficiency for AI workloads.
• Predictions: By 2030, hybrid hierarchies with AI-optimized prefetching.

Challenges and Optimizations

• Bottlenecks:
• Cache Coherence in Multi-Core: Snooping/MESI protocol to sync caches.
• Power Consumption: Leakage in SRAM; solutions like low-power modes.
• Optimizations:
• Prefetching: Predict and load ahead.
• Compression: Reduce data size in cache (e.g., Base-Delta-Immediate).
• Software Role: Align data structures for better locality.

​

Hands-On: Simulating Cache Hits

• Activity Idea: Use a simple Python sim to model direct-mapped cache.
• Code Snippet in Python:
• cache = {} # Block -> Data
• def access(addr, data):
• block = addr // 4 # Assume 4-word blocks
• if block in cache:
• print("Hit!")
• else:
• cache[block] = data
• print("Miss! Loaded.")

Discussion: Run with sequential vs. random accesses—what's the hit rate?

Case Study: Memory in Supercomputers

• Example: Frontier (World's Fastest, 2025):
• 9.2 EFLOPS; 10 PB RAM (HBM3); NVMe SSDs for I/O.
• Hierarchy: GPU caches + shared memory pools.
• Lessons: Extreme scale requires fault-tolerant ECC and distributed caching.

Summary

• Key Takeaways:
• Hierarchy balances speed/cost via locality.
• Primary: Fast/volatile (registers → RAM).
• Auxiliary: Slow/non-volatile (SSD → tapes).
• Optimizations like virtual memory enhance usability.
• Impact on Engineering: Design systems considering AMAT and power.

Q&A

• Questions? Open floor for discussion on topics like cache design or future tech.
• Resources:
• Book: "Computer Architecture: A Quantitative Approach" by Hennessy & Patterson.
• Online: CS:APP course on CMU website.
• Lab: Simulate hierarchy in Verilog next week.

​

Thank You for Attention

• End of Lecture
• Anvar Abdullayev
• Faculty of Computer Engineering

​