Confidential Containers
Split host/tenant APIs
Starting point: Assume we have PullImage
kubelet
CRIO
containerd
kata-shim-v2
Linux Kernel
KVM
VMM
(Cloud-Hypervisor/QEMU)
Confidential VM
initrd
kernel
kata-agent
Confidential Computing Platform (TDX, SEV, etc)
1 - PullImage
Container
Images
Storage
Container
Images
Storage
Container
Image
Registry
Relying Party
Key Broker
Attestation
Service
2 - PullImage
3 - Download
🔑
4 - GetKey
5 - GetAttestation
6 - SendAttestation
7 - SendKey
8 - Decrypt -
Unpack - Mount
firmware
measured
Color code:
Green: Tenant-owned
Yellow: Host-owned
Problem: Host vs. Tenant realms
Trusted
Trusted
Untrustedsadasd
Untrusted
kubelet
CRIO
containerd
kata-shim-v2
Linux Kernel
KVM
VMM
(Cloud-Hypervisor/QEMU)
Confidential VM
initrd
kernel
kata-agent
Confidential Computing Platform (TDX, SEV, etc)
1 - PullImage
Container
Images
Storage
Container
Images
Storage
2 - PullImage
3 - PullImage
Container
Image
Registry
Relying Party
Key Broker
Attestation
Service
4 - PullImage
5 - Download
🔑
7 - GetKey
8 - GetAttestation
9 - SendAttestation
10 - SendKey
11 - Decrypt -
Unpack - Mount
6 - Image digest+metadata
Whole volume encryption needed
Q: Why are there green arrows in the orange domain?
Objectives
Step 1: Blocking APIs
Reject APIs that present a security risk
Confidential VM
initrd
kernel
kata-agent
VMM
(Cloud-Hypervisor/QEMU)
kubelet
CRIO
containerd
kata-shim-v3
etcd
Host user with host credentials
vsock
Agent checks if API is valid, based on in-image, attested configuration file.
But how do you implement kubectl logs or kubectl exec?
Step 2: Expose tenant APIs
Hey, we have a secure channel already!
Confidential VM
initrd
kernel
kata-agent
VMM
(Cloud-Hypervisor/QEMU)
kubelet
CRIO
containerd
kata-shim-v3
Host user with host credentials
vsock
But how do you talk to the agent over this new channel?
Relying Party
Key Broker
Attestation
Service
Some other API source
Agent had to establish secure connexion before anything else…
etcd
Agent could also accepts API calls from somewhere in the trusted realm (same channel or similar one)
Step 3: Prototype with a tenant-only stack
Different set of secrets (different etcd), might run on different host
Confidential VM
initrd
kernel
kata-agent
VMM
(Cloud-Hypervisor/QEMU)
kubelet
CRIO
containerd
kata-shim-v3
Host user with host credentials
vsock
Do we really need forwarding?
Do we still need vsocks?
Relying Party
Key Broker
Attestation
Service
API forwarding service
kubelet
CRIO
containerd
kata-shim-v3
etcd
Tenant user with tenant credentials
etcd
Step 3b: Variant using encrypted API over vsock
Talking at some other level of the stack
Confidential VM
initrd
kernel
kata-agent
VMM
(Cloud-Hypervisor/QEMU)
kubelet
CRIO
containerd
kata-shim-v3
Host user with host credentials
vsock
Run on other host?
Custom crypto?
Relying Party
Key Broker
Attestation
Service
kubelet
CRIO
containerd
etcd
Tenant user with tenant credentials
etcd
Step 4: Locked-down / immutable pods
Take advantage of Sandbox API effort
Confidential VM
initrd
kernel
kata-agent
VMM
(Cloud-Hypervisor/QEMU)
kubelet
CRIO
containerd
kata-shim-v3
kubelet
CRIO
containerd
kata-shim-v3
Host user with host credentials
No vsock
Tenant user with tenant credentials
Secure (networked) RPC channel
etcd
Create Pod with complete Pod description (not piecewise)
Practical all APIs except pod lifetime (create, kill) go through confidential channel
etcd
Step 5: Dual-secrets user-space command split
The first (only?) major non-Kata change required (but can be manual)
Confidential VM
initrd
kernel
kata-agent
VMM
(Cloud-Hypervisor/QEMU)
kubelet
CRIO
containerd
kata-shim-v3
kubelet
CRIO
containerd
kata-shim-v3
Host user with host credentials
No vsock
Tenant user with tenant credentials
etcd
etcd
Cluster user:�kubectl $blah $args
kubectl create…�kubectl delete…
kubectl exec…�kubectl logs…
Step 6 (long term): Simplify the control plane
Leverage existing efforts: Hypershift, kcp
Confidential VM
initrd
kernel
kata-agent
VMM
(Cloud-Hypervisor/QEMU)
kubelet
CRIO
containerd
kata-shim-v3
kata-shim-v3
Host user with host credentials
No vsock
Tenant user with tenant credentials
etcd
Cluster user:�kubectl $blah $args
kubectl create…�kubectl delete…
kubectl exec…�kubectl logs…
magic-smoke
(TBD)
HyperShift and kcp
https://github.com/openshift/hypershift
Long term view: Host / Tenant Split APIs
Roughly the same APIs as today, but along two paths
Attest boot image
Provide / attest workloads
Start container (+exec, debug)
Provide secrets (e.g. disk keys)
Access workload stdio
Access workload logs
Get workload metrics
Confidential VM
initrd
kernel
kata-agent
VMM
(Cloud-Hypervisor/QEMU)
Host user with host credentials
No vsock
Tenant user with tenant credentials
Create and destroy pods
Allocate physical resources
Access hypervisor logs
Get global metrics (cgroup)
Linux Kernel
KVM
Problem: who splits the specs?
E.g. image name
Open issues