1 of 33

Confidential Containers

2 of 33

Threat Model

Prevent the host software stack from:

Accessing container data and code while in use

Confidential computing removes the host from the container TCB

Expands Kata’s current threat model

3 of 33

(Kata) Containers Gaps

The host sees all container memory

Applies to virtualized containers as well

The host manages all container images

Pulls, unpack and mounts container images on the host

The host controls all containers and pods content

Containers and pod manifests are controlled by the host

4 of 33

Solutions

Use hardware virtualization

Encrypt containers virtualized memory and virtual CPU state

Attest the guest software stack up to the runtime (a.k.a. kata-agent)

Pull, decrypt and unpack container images from the guest

Hybrid model alternative

Download from the host, decrypt and unpack from the guest

Image manifest is not encrypted, the host can tamper with it

Provision the guest with container images credentials

Verify or decrypt all container image layers

5 of 33

CRIO/containerd impact

Sandbox creation

No changes needed, everything handled from the kata shim

Pass a block device from the host for container image storage (optional)

Potentially shared encrypted layers between tenants

Reduce memory overhead (Container images layers stored in RAM without that block device)

Container creation

Image pulling+decryption/verification+mounting moved inside the guest

Digest+metadata passed back to CRIO/containerd

Shim API extension (PullImage)

Sandbox API proposal

6 of 33

RunPodSandbox

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

Container

Images

Storage

Container

Images

Storage

firmware

7 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

Container

Images

Storage

Container

Images

Storage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

firmware

8 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

firmware

9 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

firmware

10 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

firmware

11 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

firmware

12 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

5 - Download

🔑

firmware

13 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

5 - Download

🔑

6 - Image digest+metadata

firmware

14 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

5 - Download

🔑

7 - GetKey

6 - Image digest+metadata

firmware

15 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

5 - Download

🔑

7 - GetKey

8 - GetAttestation

6 - Image digest+metadata

firmware

16 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

5 - Download

🔑

7 - GetKey

8 - GetAttestation

9 - SendAttestation

6 - Image digest+metadata

firmware

measured

17 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

5 - Download

🔑

7 - GetKey

8 - GetAttestation

9 - SendAttestation

10 - SendKey

6 - Image digest+metadata

firmware

measured

18 of 33

PullImage

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

5 - Download

🔑

7 - GetKey

8 - GetAttestation

9 - SendAttestation

10 - SendKey

11 - Decrypt -

Unpack - Mount

6 - Image digest+metadata

firmware

measured

19 of 33

PullImage - Local Attestation

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

5 - Download

🔑

7 - GetKey

8 - Decrypt -

Unpack - Mount

6 - Image digest+metadata

firmware

measured

20 of 33

PullImage - Hybrid model

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

2 - PullImage

3 - Download

🔑

4 - GetKey

5 - GetAttestation

6 - SendAttestation

7 - SendKey

8 - Decrypt -

Unpack - Mount

firmware

measured

21 of 33

Pros

Memory, CPU states and container image protected from the host

Dynamic container addition

Workload content controlled by the relying party

Image layers potentially shared between tenants

22 of 33

Cons

Poor container image sharing abilities

Memory consumption overhead

23 of 33

Opens

Container creation: IO streams

Forbidden APIs (e.g. exec)

Encrypted image layers without the keys

24 of 33

Confidential Containers v0

Container Image

Restricted scope: Only available to the pod (149)

Nothing is shared with the host or other tenants

Temporary block device storage - Wiped when pod terminates

containerd/CRIO should not expose/list the image

​

25 of 33

Confidential Containers v0

Host/Tenant API

Agent API runtime configured restriction (1837 and 1891)

No shadow control plane

No exec, no logs (no stdio)

26 of 33

Confidential Containers v0

Attestation

Triggered by image layers key fetching or during startup

Kata Agent uses ocicrypt-rs (WIP) for decrypting/verifying

ocicrypt-rs keyprovider to call into a attestation agent/crate

Attestation agent/crate to be discussed

27 of 33

Confidential Containers v0

Relying Party

Confidential VM

Key Broker

+

Attestation

Service

VMM

(Cloud-Hypervisor/QEMU)

CRIO

containerd

kata-shim-v2

PullImage

kata-agent

Container

Image

(Pod scope)

🔑

Temporary Block Device

PullImage

image mgmt

ocicrypt

keyprovider

Restricted API via vsock

Attestation

Agent

firmware

28 of 33

Confidential Containers v0

Kata 2.x compatible

No breaking changes needed

First end to end implementation

Reachable in the short term

We can then incrementally improve

​

Restricted

Enlightened workload for exec, logs

Inefficient

Always pull image layers

Requires a dummy block device

29 of 33

Backup

30 of 33

Confidential Kubernetes Nodes

kubelet+CRIO/containerd inside the confidential VM

Different threat model

Protect container data from the host

Does not protect container from each others

E.g. Google’s confidential GKE nodes

31 of 33

Host vs. Tenant realms

Trusted

Trusted

Untrustedsadasd

Untrusted

kubelet

CRIO

containerd

kata-shim-v2

Linux Kernel

KVM

VMM

(Cloud-Hypervisor/QEMU)

Confidential VM

initrd

kernel

kata-agent

Confidential Computing Platform (TDX, SEV, etc)

1 - PullImage

Container

Images

Storage

Container

Images

Storage

2 - PullImage

3 - PullImage

Container

Image

Registry

Relying Party

Key Broker

Attestation

Service

4 - PullImage

5 - Download

🔑

7 - GetKey

8 - GetAttestation

9 - SendAttestation

10 - SendKey

11 - Decrypt -

Unpack - Mount

6 - Image digest+metadata

Whole volume encryption needed

Why are there green arrows in the orange domain?

32 of 33

Locked-down / immutable pods

An alternative to forbidden APIs?

Confidential VM

initrd

kernel

kata-agent

VMM

(Cloud-Hypervisor/QEMU)

kubelet

CRIO

containerd

kata-shim-v3

kubelet

CRIO

containerd

kata-shim-v3

etcd

etcd

Host user with host credentials

No vsock

Tenant user with tenant credentials

Hypershift-like control plane

Secure RPC channel

33 of 33

Host / Tenant Split APIs

Roughly the same APIs as today, but along two paths

Attest boot image

Provide / attest workloads

Start container (+exec, debug)

Provide secrets (e.g. disk keys)

Access workload stdio

Access workload logs

Get workload metrics

Confidential VM

initrd

kernel

kata-agent

VMM

(Cloud-Hypervisor/QEMU)

Host user with host credentials

No vsock

Tenant user with tenant credentials

Create and destroy pods

Allocate physical resources

Access hypervisor logs

Get global metrics (cgroup)

Linux Kernel

KVM

Problem: who splits the specs?

E.g. image name