EVE-OS - a SPOS for the distributed edge

https://github.com/lf-edge/eve

Presented by Erik Nordmark, EVE TSC Chair (and ZEDEDA CTO)

Problem we solve

• Easy/secure deployment of cloud-native to the distributed edge
• Distributed edge - typically single or a few servers on-prem
• Often involves thousands of locations
• Help users on their journeys
• Windows-based monoliths, to VMs, to containers, to kubernetes
• Raises issues about
• (lack of) physical and network security
• Reliable power
• Intermittent Internet connectivity, static IP, MiTM proxies, active standby
• Rich local network and I/O connectivity (serial, USB, etc)
• No (IT) staff on site
• Air-gapped networks
• Already have users with 10,000 (single node) kubernetes clusters

Security challenge

• App or model secret sauce at insecure location
• Security and privacy of customer data on disk
• How secure is the network?
• Physical access - theft of software or data?
• Constraint: after power outage apps need to start
• Even if there is no upstream connectivity

Security challenge - existing approaches

• BIOS, OS, runtime minimization and hardening required
• Plus secure and scalable update mechanism
• Immutable images makes this easier
• A host based firewall can help with network security
• No inbound connectivity? No users/passwords?
• Need for security monitoring on each server?
• Theft?
• Secure boot insufficient. Attacker: Find old signed firmware/software with known CVEs
• Full disk encryption doesn’t help - no user to input password

​

Project EVE Approach

Free self-service

SaaS

​

Open Source

EVE API

Sandbox

Project EVE Approach

Edge Virtualization Engine (EVE)

• Abstraction layer designed for the edge
• Created and donated by ZEDEDA to LF
• Open sourced under Apache License v2
• Part of Linux Foundation LF Edge Project

Any Gateway at IoT Scale

Free self-service

SaaS

​

Open Source

EVE API

Any Application

VM or Container

Sandbox

EVE Architecture

containerd

User Edge Compute Hardware

Open API

Partition A

Partition B

EVE managed, workload-centric storage

EVE services

EVE Controller

Disk overhead: 500M

RAM overhead: 500M�CPU overhead: 1 core

Hypervisor (Xen, KVM, ACRN)

On host/dom0

Dom 2

Dom 1

EVE Controller

Software Stack

Linuxkit

Linuxkit init

Onboot containers

containerd

EVE microservices as containers

Software Stack

Linuxkit

watchdog

newlogd

wlan

wwan

debug

nim

Linuxkit init

Onboot containers

containerd

baseosmgr

downloader

verifier

tpmmgr

vaultmgr

edgeview

guacd

Device, OS, runtime

orchest-�ration

volumemgr

Software Stack

Linuxkit

watchdog

newlogd

wlan

wwan

debug

nim

Linuxkit init

Onboot containers

containerd

baseosmgr

downloader

verifier

tpmmgr

vaultmgr

Runtime support

zedrouter

volumemgr

downloader

verifier

volumemgr

vtpm

kvm

meta-data

domainmgr

containerd

k3s

xen

longhorn

Workloads

VMs

containers

pods

edgeview

guacd

Device, OS, runtime

orchest-�ration

zedmanager

Disks and partitions

• Four partitions (plus more on ARM)
• CONFIG - for device personality (identity also in TPM)
• IMGA/IMGB - squashfs boot/root partitions
• PERSIST - user data volumes, images, spooled logs, config checkpoints
• /persist/vault is encrypted (fscrypt/ext4, or ZFS native)
• CONFIG and IMGx are integrity protected (using TPM)
• Multiple options
• ext4 /persist on one disk
• ZFS pool across multiple disks for larger /persist
• Longhorn (underway) for clustered storage

EVE API

• Dial-home to controller
• Declarative configuration API plus a few operations (e.g., reboot)
• Self-compressing status API (can miss events)
• Self-compressing metrics API (can miss periodic reports)
• E.g., reports CPU seconds since boot, not CPU percentage
• Protobuf encoded payload
• Using Mutual TLS 1.2/1.3 plus end-to-end object signing
• Object signing to support MiTM content-inspecting proxies
• User secrets additionally protected by end-to-end object encryption
• To avoid leaking e.g., datastore credentials and cloud-init secrets to proxy admin
• ​
• ​

​

Project EVE Architecture

Hardware Layer

EVE-EVC API - config, status, metrics, logs

EVEagent:

config, status, metrics

Downloader

EVEmanager:

instance orchestrator

Verifier

sha, sigs

HW info, metrics

Domain mgr

dom0

Crypto device identity

TEE/TPM

Measured boot and remote attestation

Device onboarding

Disk encryption

TLS 1.2/1.3 + object signing

Baseos manager

Grub gpt priority boot

Network interface manager

Device connectivity

log manager

Eth, RS 485, BTLE etc

Instance B

Instance C

Instance D

Driver domain(s)

Instance A

Edge Virtualization Engine

​

Eth, wlan, wwan

Hardware watchdog

Linux watchdog

Self update

Device connect-�ivity

​

​

Edge Container runtime

Deployed

Edge Containers

​

​

​

​

​

​

​

EVC sample: Adam

Commercial EVC:

Device APIs

​

​

Device Identity Onboarding Security Foundation

EVErouter:

DHCP

DNS

ACLs

VPN

​

Instance connectivity

local + NAT

I/O virtualizatiion

and

assignment

switch

cloud

Remote instance local

consoles

​

Volume manager

​

Edge Container connect-

ivity and storage

​

​

​

​

​

Project EVE Architecture

Hardware Layer

EVE-EVC API - config, status, metrics, logs

EVErouter:

DHCP

DNS

ACLs

VPN

​

EVEagent:

config, status, metrics

Downloader

EVEmanager:

instance orchestrator

Verifier

sha, sigs

HW info, metrics

Domain mgr

dom0

Crypto device identity

TEE/TPM

Measured boot and remote attestation

Device onboarding

Disk encryption

TLS 1.2/1.3 + object signing

Baseos manager

Grub gpt priority boot

Network interface manager

Device connectivity

Instance connectivity

log manager

Eth, RS 485, BTLE etc

local + NAT

I/O virtualizatiion

and

assignment

switch

cloud

Instance B

Instance C

Instance D

Driver domain(s)

Instance A

Edge Virtualization Engine

​

Eth, wlan, wwan

Hardware watchdog

Linux watchdog

Remote instance local

consoles

​

Volume manager

​

EVE Robustness Elements

• Immutable, read-only OS image
• A/B booting
• Hardware and software watchdogs - automatic rollback
• Declarative, eventual consistency API
• Including device network connectivity (static IP, https proxy)

EVE Robustness and Security Elements

• Immutable, read-only OS image
• A/B booting
• Hardware and software watchdogs - automatic rollback
• Declarative, eventual consistency API
• Including device network connectivity (static IP, https proxy)
• Disable (USB) ports by default - assignable to applications
• No user accounts/login in EVE
• Zero touch, zero trust onboarding
• Mutual trust EVE - Controller from secure onboarding process
• Mutual TLS plus object signing/encryption
• Distributed firewall for every app
• Support deployment of virtual security appliances

Solving the hard - handle stolen disk or server

• Start with hardware root of trust (TPM chip or firmware)
• Use TCG measured boot and remote attestation
• Use standard sealing of (storage) key under TPM measurements
• Unique: avoid attestation if no hardware/firmware/software change
• Unique: if change retrieve encrypted “for me” key from controller
• Details in https://wiki.lfedge.org/display/EVE/Measured+Boot+and+Remote+Attestation

​

Potentially unique technology

• Mutual auth secure onboarding (integrates with FDO)
• A/B/C network config; test then rollback or commit
• Signed network bootstrap config?
• A/B boot with auto-rollback using watchdog (HW, SW)
• including controller connection check
• Immutable OS + integrity check of hardware/firmware/software
• File system encryption - keys sealed under TPM PCRs
• Datastore concept - level of indirection to access images (local)
• APIs for local operator but single source of declarative truth
• Signed configuration enables air-gapped centralized policies

​

Recent and future work

• Example recent work:
• Better USB virtualization - per device and not whole controller
• Application volume snapshot and rollback (app version rollback)
• EVE features under way:
• Richer 5G/LTE and local connectivity (CAN bus, VLANs)
• Handling broken TPMs better
• Clustered compute and storage at the edge
• Real-time support
• Exploring
• Kata containers, coco
• FDO onboarding