Pre-read
The slides in this deck are publicly available!
These are the slides that correspond to Kris Nóva’s 2023 FOSDEM talk.
All comments on the slides will be publicly visible.
For private feedback please email: nova@nivenly.org.
The Go gopher was designed by Renee French licensed under Creative Commons.
I have written permission to use the Untitled Goose Game art form House House and Panic.
Ferris the Crab was designed by Dzuk and licensed under Creative Commons.
The Kubernetes logo is Apache 2.0 licensed.
Aurae
Distributed Systems Runtime
Kris Nóva
hachyderm.io/@nova
Principal Engineer
Author
Computer Science
Overview and Context
github.com/aurae-runtime/aurae
What is Aurae?
An open source Rust project aimed at simplifying node management at scale.
A generic “execution engine” for containers, VMs, and processes
Thesis Statement
Bringing deliberate runtime controls to a node will unlock a new generation of higher order distributed systems.
Simplifying the execution stack will foster secure and observable systems while reducing complexity and risk.
What is a “node”?
hachyderm.io/@nova
..not Node JS
A single compute unit in a set.
EG: A server in a cluster.
What does Aurae do?
Control of every runtime process on a node.
Runs as a minimal init system with a remote API.
Manages processes, containers, virtual machines.
Runs on Linux.
What doesn’t it do?
Generic desktop support is out of scope.
Higher order scheduling is out of scope.
Does not run on Darwin.
Does not run on Windows.
Where did it come from?
Challenges with complexity at scale.
Desire to simplify and secure the stack.
Hope for a stronger node API.
Hope for managing nodes at scale.
All workloads are untrusted
The reality of scale.
Boring
Functional
Guiding Principles
Attainable
Single Binary
Polymorphic, easy to ship, lightweight, strong scope, low risk
Strong Interfaces
Wanted to build a “standard library” for multiple purposes.
Plays nice with others
Kubernetes Container Runtime Interface
(all of them…)
We want boring systems!
Most problems stem from complexity, drift, and disparate systems
Attack Surface?
Node upgrades?
Pod boundary?
Which YAML?
Container Escapes?
Network interfaces?
Label selector templates?
Template render?
What about systemd?
Docker?
Security context?
Federation?
Service Mesh?
Istio?
DNS?
kube-proxy?
Iptables? nftables?
Resource quota?
Kernel versions?
Observability?
The Kubelet has an API?
Safe
Guiding Principles
Tenant Security
How do we make it easy to do the right thing?
How do we bake security into systems that are already online?
Aurae
We wrote a paper
Started in Go
“Aurae Legacy”
NAML
Daemon Proof of Concept
Created familiar interfaces.
Needed to re-create the JSON/YAML dance.
What about access to libc?
Turing Complete Manifests
“Not Another Markup Language”
Kubernetes objects in Go.
Basically the Go SDK.
github.com/aurae-runtime/aurae-legacy
github.com/krisnova/naml
libc
virtualization
..but
eBPF
hachyderm.io/@nova
Pid 1
Cgroups V2
Clone3, Ptrace
init semantics, udev
Existing Libs
C Libraries
KVM
Firecracker
Cgroup controllers
Probes
We want to instrument our isolation zones.
What about writing the probe bytecode?
Go 1.20 Update
Unwrap()
Clone()
Errors
Go 1.20
Memory
Go is starting to take memory safety seriously
Rust
I made the decision to move to Rust.
Rust
to help us solve the YAML problem
All Config Drifts Towards Turing Completion
hachyderm.io/@nova
AuraeScript
Rust binary that embeds connection logic for auraed
Deno and TypeScript
github.com/denoland/deno
In order to keep the system attainable I wanted a simple and safe Turing complete language in favor of static config.
Deno is a simple runtime for TypeScript written in Rust.
AuraeScript as a Client
hachyderm.io/@nova
AuraeScript embeds Deno and a Rust client.
TypeScript as a replacement for YAML
sleep.ts
sleep.ts
YAML but with math
Patterns Emerged
hachyderm.io/@nova
start
stop
status
nginx.ts
dashboard.ts
Static Service Definitions
Turing Complete Generic Functionality
Rust
to help us solve the sidecar problem with auraed
A sidecar that is always available with the same features as the host.
hachyderm.io/@nova
Rust memory safety is key.
Any sufficiently mature infrastructure service will evolve into a sidecar.
hachyderm.io/@nova
Features for Both:
The host and your workload
Logging
Tokio streams
Authn / Authz
#derive[]
Derive access controls at the feature level in Rust
Identity
RustTLS
Substantially simpler
Network
Direct access to Linux network devices
eBPF facilities
Rust unlocks “Spawn”
Rust gives us access to virtualization, containers, processes, and the kernel
Single API for both nodes and inter-workload features
What we bring to the host, we bring to your workload
All memory safe
Assemble nested images from host filesystem at runtime��No downloading images
No pause containers
Spawn
Aurae Host Daemon
Privileged
/proc/self/exe
Cell Sandbox
Nested Aurae Daemon
Pod Sandbox
Nested Aurae Daemon
VM Sandbox
Nested Aurae Daemon
Rust
to help solve the untrusted workload problem
Reduced attack surface
Borrow checker pays off here
Memory Safety
Init Container
Hi Ian! I love you!
Isolation Zone
(Pod Sandbox, VM, Cells)
Nested Aurae Daemon
pid=1
Untrusted Workload
pid=2
Untrusted Workload
pid=3
Managing Memory for mTLS in Auraed
Rust
to help us solve the maintainability problem
Did somebody say macros?
@future_highway
Third party resources? Custom Resource Definitions? YAML? Controllers? But what does it mean?
API Machinery, Third Party Resources, CRDs
Go/Kubernetes Want Macros
Starting with generic objects
//go:generate
zz_kube_types.go
What are Rust macros?
Rust lets you write code that can write other code.
cells_call.rs
pods_call.rs
vms_call.rs
aurae_macros.rs
ae_macro.rs
aer_macro.rs
What are Rust macros?
Rust lets you write code that can write other code.
cells_call.rs
pods_call.rs
vms_call.rs
aurae_macros.rs
ae_macro.rs
aer_macro.rs
ae_cells.rs
What are Rust macros?
Rust lets you write code that can write other code.
These are unique needs of Aurae!
Macros in Aurae
hachyderm.io/@nova
Rust Client
AuraeScript
Documentation
Deno Ops
TypeScript library
Rust Source Code
Macros as an Investment
Manual Plumbing
Improve
Refine
Rust Macro
Binary
Macros as an Investment
Rust Macro
Binary
The State of the Art
Rust
to help us solve the boring problem
Takeaways
What’s Next?
Getting Involved
GitHub
github.com/aurae-runtime/aurae
Docs
aurae.io
Twitch
twitch.tv/krisnova
Discord
discord.gg/T76jkkMbSj
Twitch Crew
@future_highway
@dma42
@orz
@taniwha3
@kuxaku
@malte_aero
@zunderscore
@lastchain
@quintessence
@esk
@hazelweakly
@rrethy
Demo
hachyderm.io/@nova
Interesting Takeaways
hachyderm.io/@nova
A gopher’s first year of Rust
Is it..?
A microkernel?
A unikernel?
A system system?
It depends.
Rust Development Flow
Idea
Establish feature to add to code base
Implement
Design and fight with the borrow checker
Setup
Basic plumbing, Cargo.toml, build.rs, etc
Profit
It’s kind of just done now
0m
+30m
+10m
+31m
There is only the “right” way
..there isn’t really any other way.
Rust is easy to hate
…if you try to fight the language instead of letting the language show you the way.
- @togglebit
Go vs Rust
Frequently Asked Question
Rust
Go
My Opinion
auraed
Safe, Reliable
Ship “right”
Fight the compiler
Libc is a super power
ae
Fast, Practical
Ship “right now”
Fight the debugger
Libc is a risk
I feel lucky to know both and have access to both.
Node Config
Management
hachyderm.io/@nova
…is the wrong conversation to be having.
Why are we trying to
“get better”
at hacking user level configuration for systems with no users?