1 of 39

SOFSEC1 – Software Security

Module 2: Operating System Security

Prof. Justin Pineda

Jan 2026

2 of 39

Learning Strategy

​

“You don’t need to understand how an engine is built to understand why removing the brakes is dangerous.”

3 of 39

Learning Objectives

  • Evaluate OS security mechanisms related to memory, CPU, and I/O (LO3)
  • Compare exploitation methods targeting memory, CPU, and I/O (LO4)
  • Explain why OS-level vulnerabilities are high-impact risks

4 of 39

Motivation Question

​

If your application code is secure, but the OS is vulnerable — are you actually secure? Why or why not?

​

5 of 39

What Is OS Security?

  • OS as the trust anchor of all applications
  • Controls:
    • Process isolation
    • Memory protection
    • Privilege enforcement

OS vulnerabilities = system-wide compromise

6 of 39

7 of 39

Definition

​

The Operating System (OS) is the manager of your computer.

8 of 39

OS as the ATC

​

The OS is the airport control tower

Apps are airplanes. Without the tower, planes crash.

9 of 39

Why OS Security Matters

If someone controls the OS, what can they control?

    • All applications
    • All files
    • All user accounts
    • Even security software

10 of 39

Memory

  • Temporary working space of your computer
  • Where apps place data while running

11 of 39

Memory Attacks: Buffer Overflow

  • Program expects 10 characters
  • Attacker sends 1,000 characters
  • Extra data spills into areas it shouldn’t

12 of 39

Buffer Overflow Result

  • App crashes
  • Attacker controls the program

13 of 39

Other Memory Attacks

  • Buffer overflow
  • Heap overflow
  • Use-after-free
  • Memory corruption → code execution

14 of 39

Memory Protections

Security features like:

  • ASLR ((Address Space Layout Randomization)
  • DEP / NX (non-executable memory)

​

Security guards that:

  • Move rooms around
  • Lock certain doors
  • Stop people from entering restricted areas

15 of 39

Other Memory Protection

  • Stack canaries
  • Kernel address isolation

16 of 39

CPU and Processes

CPU = Brain of the computer

​

It decides:

  • Which program runs now
  • Which program waits

17 of 39

CPU and Processes

  • Processes vs threads
  • Privilege levels (user mode vs kernel mode)
  • Context switching risks

​

Attackers aim to:

  • Escalate privileges
  • Escape process boundaries

18 of 39

CPU-Level Attacks

  • Side-channel attacks
  • Speculative execution abuse
  • Hardware-assisted leakage

19 of 39

CPU Attacks

Some attacks:

  • Trick the CPU into revealing information
  • Abuse how it “thinks ahead”

20 of 39

I/O (Input / Output)

Input: Keyboard, mouse, USB

Output: Screen, printer, network

21 of 39

I/O (Input / Output)

  • File systems
  • Network interfaces
  • Peripheral devices (USB, drivers)

​

High-risk areas:

  • Device drivers
  • File permissions
  • Race conditions in file access

22 of 39

I/O Attacks

  • USB malware
  • Insecure file permissions
  • Malicious downloads

23 of 39

I/O Exploitation Examples

  • Insecure file permissions
  • Malicious device drivers
  • Time-of-check vs time-of-use

24 of 39

OS Security Failures in Real Incidents

  • Privilege escalation exploits
  • Kernel vulnerabilities
  • Patch lag as a risk factor

25 of 39

Perspectives

From a defender’s view:

  • Hardening the OS
  • Least privilege enforcement
  • Monitoring abnormal OS behavior

​

From an attacker’s view:

  • Bypass controls
  • Abuse trusted components

26 of 39

Attacker vs Defender Mindset

Attacker thinks:

“What controls everything?”

​

Defender thinks:

“How do I limit damage if one thing breaks?”

27 of 39

Connecting OS Security to Software Security

Even perfectly written software can be broken if the OS is weak.

​

This is why:

  • Developers
  • System admins
  • Security teams

must work together

28 of 39

Summary

  • OS security is foundational
  • Memory, CPU, and I/O are prime targets
  • Protections exist but have limits
  • Attackers think system-wide, not app-only

29 of 39

Knowledge Check 1

ASLR primarily defends against which attack type?

  • A. Phishing
  • B. Memory exploitation
  • C. Malware delivery
  • D. Password reuse

30 of 39

Knowledge Check 2

Why are device drivers high-risk?

  • A. They are user-controlled
  • B. They run with high privileges
  • C. They are rarely used
  • D. They are encrypted

31 of 39

Knowledge Check 3

Which OS component is most commonly targeted for privilege escalation?

  • A. Application UI
  • B. Kernel
  • C. Database
  • D. Network cable

32 of 39

Knowledge Check 4

Which is an example of a CPU-level vulnerability?

A. SQL injection

B. Buffer overflow

C. Spectre-type attack

D. Cross-site scripting

33 of 39

Knowledge Check 5

True or False:

If application code is secure, OS vulnerabilities no longer matter.

34 of 39

Key Takeaways

​

​

What part of OS security surprised you the most, and why?

35 of 39

References

​

  • SOFSEC1 Course Syllabus
  • Recorded Lectures
  • Recommended Online Resources

36 of 39

Group Exercise 2: Simulating OS Vulnerabilities Using bWAPP

  • This exercise helps students understand how software functionality, security controls, and threat actor behavior intersect early in the SDLC—especially when security is missing or assumed.

37 of 39

Exercise Objectives

At the end of this lab, students will be able to:

  • Explain how OS-level or system-level vulnerabilities affect applications
  • Simulate Heartbleed and Shellshock attacks in a controlled environment
  • Describe the impact of foundation security failures
  • Reflect on why secure applications still fail on insecure systems

38 of 39

Lab Environment

  • Instructor-provided Virtual Machine
  • Web application: bWAPP
  • Attacks simulated:
    • Heartbleed
    • Shellshock

39 of 39

Pre-Requisites