1 of 25

CSE 451: Operating Systems�Spring 2026�Virtual Machines

Rohan Kadekodi

Slides by: Jonathan Trinh

2 of 25

Importance of Virtualization

Most compute and storage-intensive work happens in the cloud�
Example: starting an IT company

No virtualization
First leap: on-prem VMs
Second leap: Cloud�

For providers:

High utilization of resources

2

First, let’s talk a bit about why we need virtualization, or virtual machines.

Today, most of the compute and storage-intensive work goes on in the cloud. There is hardly any work on a large scale that happens on local computers. All the large companies like Google, Meta, have their own datacenters, or use public cloud offerings. These companies manage both internal data and customer data in VMs. So virtualization is in the heart of today’s public and private cloud offerings.

Let’s imagine for a minute that we have an IT company with an accounting department, backend developers, marketing department, and some frontend developers. In a pre-virtualization era, this company would basically need for each employee, a computer with their own monitor and keyboard, where every computer would cost thousands of dollars. Then you would need an IT support staff to manage software on computers, such as updating the OS, etc. So for me to startup such a company, the cost would be atleast 10s of thousands of dollars, in the 90s, which is very very costly.

The first jump in virtualization came via on-premise VMs, where instead of separate desktops, these companies would have server rooms with all machines, where every employee would basically get remote access to the machine, and have their own “virtual machine”, depending on their requirements. While this reduced the cost somewhat, because you can pool machines and share for users, you still need IT support staff to maintain these machines, and update their packages and OS and so on.

The second major jump was the cloud. Today, there is no need of any physical resources. Today, you can simply launch a VM in the cloud, using Amazon EC2, or Google Cloud Platform or Azure. No IT support staff needed. So very little investment needed for resources. Very low overhead to start a company. You only pay for the amount you use – you don’t pay any other cost. In fact, today we have many many different kinds of VM offerings like spot VMs, harvest VMs, Burstable VMs, and so on, that even reduce this cost further.

From the provider’s perspective, they maintain a large cluster of machines, and then they are able to build virtualization software on top of their servers, and share their hardware with external customers, such that they maximize utilization of their resources, and get paid by the customers. So it’s a win-win.

3 of 25

Agenda

Overview of Virtual Machines
General Mechanisms of Virtual Machines

System Calls
Memory Management
I/O

3

4/7/26

4 of 25

Motivation

Isolation & Security

Developing Operating Systems

Isolate system bugs and protect hardware

Running and testing dangerous software (“Sandboxing”)

Legacy Support for Applications

Simulate environments to run legacy programs on modern hardware

Data Centers and Cloud Computing

Cloud Providers own massive data centers with unbelievable amounts of hardware and server racks
Providers run many Virtual Machines on top of their hardware to distribute services to their clients

4

5 of 25

Virtualization

A virtual X is an efficient, isolated duplicate of the real X

X = {processor, memory, disk, machine, network, datacenter}

Virtualization: no changes to code running in the virtual X Paravirtualization: (small) code changes ok

6 of 25

What is a Virtual Machine

General Definition: A complete compute environment with its own isolated processing capabilities, memory, and communication channels

6

4/7/26

7 of 25

VM Terminology

Host OS: The primary operating system running directly on the physical hardware
Guest OS: The operating system running within the Virtual Machine using the virtualized hardware

Hypervisor: Software that creates and runs Virtual Machines with main purpose of virtualizing hardware

Type-1 (Bare Metal):

Ex: Microsoft Hyper-V, VMware, Xen

Type-2 (Hosted):

Ex: VirtualBox, VMware Workstation, KVM

Similarly

Host physical memory, guest physical memory, guest virtual memory

7

4/7/26

8 of 25

Standard Kernel Structure

8

4/7/26

Hardware

(Host) OS

App

Kernel Mode

User Mode

9 of 25

9

4/7/26

Hardware

Hypervisor

Type-1 Hypervisor

Type-2 Hypervisor

VM

Guest

OS

Guest

App

VM

Guest

OS

Guest

App

Hardware

Host OS

VM

Guest

OS

Guest

App

VM

Guest

OS

Guest

App

Hypervisor

Kernel Mode

User Mode

When it comes to VMs, this is how it looks.

Type 1 and Type 2.

Type 1:

Some physical hardware. Type-1 hypervisor, where there is a hypervisor runs on top of hardware. Hypervisor runs set of VMs, which are like processes for Host OS. But the processes have the illusion that they are running on actual hardware. GuestOS could be anything, all coexisting on the same hardware.

Type 2 is more like Hosted VMs, on your laptops. Here you have a Host OS, and a hypervisor that runs on top of the Host OS, and this hypervisor then runs VMs on top of it, each of which has a guest OS. Hypervisor is like a hybrid kind of application that makes use of hardware virtualization so has a kernel driver, but is a user-space process.

Essentially, here the Guest OS thinks that it is running in kernel mode, but it is actually running in user mode. How can we do that? These Guest OSes running concurrently, on an abstraction that looks like raw hardware. In hypervisor, we are providing the illusion that we are running on hardware.

10 of 25

Virtual Machine Advantages

Encapsulate the application runtime environment

Eg, OS version, system call API, machine resource limits
Independent of anything else running on the same physical machine

Server consolidation

Resource sharing for applications that need less than a whole server
Allows sharing of hardware between latency and throughput sensitive applications
And CPU-intensive and memory-intensive and disk-intensive apps

Transparent checkpoint and restart

Fault tolerance, server software/hardware upgrades
Transparent virtual machine migration

Disk storage disaggregation

Redirect I/O from virtual machine to a shared service

OS kernel debugging

Advantages:

Encapsulate all things that affect how you run applications. Bundle not just system-call interface, all other things that can affect execution. How many file descriptors you allow, how much memory you provide to program, etc. Essentially it is an entire OS and all resources hypertuned for a single application. Nothing else running at the same time will be affected by it. Can change versions of packages, OS, etc. It can be totally different from others
I get server consolidation. Run multiple applications – some don’t need a whole server. Modern cloud: 100 cores. An application requires lesser, handful of cores. Can run 20 VMs, don’t need to run on the same version of same OS, and don’t need to upgrade. Can share in. normal way and get better resource usage, and where we get economic advantage in the cloud.
Easier to disaggregate storage, storage can be somewhere else, just be virtualized to provide the illusion that it is local
Transparent checkpoint and restart – take application with state, and start it again. Only make sure that VM starts in same state as before.
Original reason for VMs was OS debugging. QEMU for labs is standard commercial thing that is used for running VMs and KVM. The configuration that gets used in practice is more high performance, which we might talk

11 of 25

Virtual Machine Challenges

What happens when the guest application does a system call?

Traps to the host kernel, not the guest kernel; what then?

How do we emulate guest kernel mode?

Can’t run guest OS in kernel mode (not trusted)
What happens when guest OS tries to execute a privileged instruction, such as to

change the page table or initiate I/O?

How do we emulate application virtual memory?

Host OS page tables: where guest physical memory is in host physical mem
Guest OS page tables: where app virtual memory is in guest physical mem

How do we emulate I/O devices?

Guest OS thinks it has access to physical devices

12 of 25

System-Level Virtual Machines

Owns Virtualized Hardware
Runs an Operating System of its choice

Guest OS runs in user mode*

How does Guest OS run privileged instructions?

12

4/7/26

Hardware

Windows + Virtual Box

VM

Linux

Guest App

13 of 25

Virtualized System Calls

Trap & Emulate: The Guest OS traps into the hypervisor to handle emulating that instruction
Binary Translation: Hypervisor scans and swaps out dangerous instructions
Paravirtualization: The Guest OS gets “enlightened” and starts using Hypercalls
Hardware-Assisted Virtualization: Modern chips (Intel VT-x or AMD SVM) introduce a Non-Root Mode

13

4/7/26

Let’s start with how to virtualize system calls. There are many solutions here.

One of the key ideas, not as widely used now, is called trap and emulate. You can emulate the behavior of a processor in the OS or hypervisor. So for any instruction, even if it does not exist in real hardware, the guest OS can trap into the hypervisor, and hypervisor emulates that instruction.
If we need to change page table entries, or disable interrupts, trap into the host OS, have the host OS do it for us, and resume execution for guest OS.
Another solution is binary translation. This was basically to solve a bug in x86. The Popek-Goldberg requirements say that all sensitive instructions must be privileged. However, in x86, some privileged instructions in user mode did not trap into the kernel, revealing hardware state. To solve this, binary translation was invented, where the hypervisor scans the entire guest code before execution, and rewrites the problematic instruction. Then we use trap & emulate to work.
Paravirtualization is when the guest OS is actually aware that it is running inside a VM, so the guest VM itself perfoms hypercalls to the underlying hypervisor for executing instructions.
Finally, we have hardware-assisted virtualization. It was soon understood that virtualization is very common and important enough that hardware must support it. So, we now have hardware-assisted virtualization. In this, instead of running the guest OS in ring 3, we have 2 modes of operation in the CPU: VMX root and non-root. Guest OS runs in ring 0 of non-root, hypervisor runs in ring 0 of root. SO the hardware knows that guest OS is unprivileged. No need of binary translation. And datastructure called VMCS is used for VM. When guest does privileged, VMCS is stored, and control transfers to root-mode, hypervisor, and handled.

In this, basically, the hypervisor scans through all the sensitive code, and changes dangerous instructions to something else that is not dangerous, or that traps into the hypervisor, so that the application or guest OS is not able to do anything dangerous.

Another solution is

14 of 25

Scenario: Migration!

A finance firm is migrating its infrastructure to a virtualized environment. They have two distinct workloads:

Workload A: A proprietary, closed-source risk assessment engine running on an archaic version of Windows. There's no source code available, and the OS is no longer supported by Microsoft.
Workload B: A high-performance, Linux based trading platform built in-house. The engineering team has full access to the kernel and can modify the OS image as needed.�

What is the optimal approach for each workload?

What's possible? What isn't, given the constraints of each workload?

14

4/7/26

15 of 25

15

16 of 25

Memory Virtualization

16

4/7/26

Guest App

mov rax, addr

OS

PT pointer

Page Table

Memory



data

Virtual Address

Physical

Address (HPA)

TLB

Virtual

addresses

Physical

addresses

17 of 25

Memory Virtualization

17

4/7/26

Guest App

mov rax, addr

Guest OS

Guest PT pointer

Guest Page Table

Memory



data

Virtual Address

TLB

Virtual

addresses

Physical

addresses

Hypervisor

Hardware PT

Pointer

Guest

physical

Address (GPA)

Physical

Address (HPA)

Guest memory map (GPA -> HPA)

18 of 25

Memory Virtualization

18

4/7/26

Guest OS has its own page tables

Map guest virtual addresses to guest physical memory

Host OS has its own page tables

Map guest physical memory to host physical memory

What happens on a TLB cache miss?

Need map from guest virtual address to host physical memory

19 of 25

Shadow Page Table

19

4/7/26

Guest App

mov rax, addr

Guest OS

Virtual PT

Pointer

Guest Page Table

Read Only

Hypervisor

Hardware PT

Pointer

Shadow Page Table

Memory



data

Guest

physical

Address (GPA)

Physical

Address (HPA)

Guest Virtual Address

Guest memory map (GPA -> HPA)

TLB

Virtual

Addresses (GVA)

Physical

Addresses (HPA)

20 of 25

Pros & Cons

Pros

Portable for both the OS and Hardware
Very fast once the Shadow PT is populated

Cons

Lots of overhead

Trap and Emulate
All the Page Tables

TLB flushes very often

20

4/7/26

21 of 25

Paravirtualization

21

4/7/26

Guest App

mov rax, addr

Guest OS

Virtual PT

Hypervisor

Hardware PT

Pointer

Guest PT

Hypervisor’s PT

Memory



data

Physical

address

Hypercall

Guest VA

22 of 25

I/O Virtualization

Full virtualization (Emulation)

The Guest OS Traps to Hypervisor
Hypervisor performs I/O on its behalf
Guest OS is interrupt to start its handler
Guest OS after fulfilling I/O Traps to Hypervisor
Two Traps per I/O (very slow)

VirtIO (Paravirtualization)

Use a Virtual Queue to Minimize Traps
Guest and Hypervisor agree on a region in Ram for this
Requests are batched asynchronously into the hypervisor
When done Hyper-V interrupts Guest OS to receive its result

22

4/7/26

Full virtualization, and para virtualization.

In full virtualization, Here the guest OS thinks it's talking to a real hardware device — say an Intel e1000 NIC. Every interaction with that device involves writing to device registers, which are memory-mapped I/O addresses. Each of those writes traps into the hypervisor because they access addresses that the hypervisor controls.

The flow for a single I/O operation is: the guest OS writes to the emulated device register, which causes trap #1 into the hypervisor. The hypervisor sees what the guest was trying to do, performs the actual I/O on real hardware on the guest's behalf, and when the I/O completes, it injects a virtual interrupt into the guest. The guest's interrupt handler runs, processes the completion, and then traps again (#2) back to the hypervisor to acknowledge or finalize.

Paravirtualization, guest OS knows that it is running in a VM, on a hypervisor. Using this information, it optimizes this I/O path. The guest and hypervisor agree on a shared region in memory (a ring buffer called virtual queue). The guest driver then places the I/O on the virtual queue. Multiple requests can be batched into this queue.

Once there is a batch of requests, the guest VM generates a hypercall to the hypervisor. The hypervisor processes all the queued requests asynchronously against the real hardware, and then sends a single interrupt to the guest OS, and the guest OS reads results from the shared memory.

2 traps per I/O call, to one trap for a batch of I/O call.

23 of 25

VirtIO’s Virtual Queue

23

4/7/26

Tail

RAM

Head

I/O #1

24 of 25

Extra: Containers vs Virtual Machines

Containers

Uses the Host OS to spin up the container
The Host OS handles isolating the Container which acts as a process with its own name space and resources (cgroups)

Virtual Machines

Uses the Hypervisor to virtualize the hardware
Allows the Virtual Machine to run its own Guest OS separate from the Host OS

Trade-offs

Security:

VMs are much more secure since they isolate at a higher level

Cost:

Containers are more scalable and make better use of hardware resources

24

4/7/26

25 of 25

References

Operating Systems Principles and Practice 2^nd Ed.
Hardware and Software Support for Virtualization
Compiler Design: Virtual Machines
VMScape

Blog on breaking the isolation of Virtual Machines to steal information

25