BPFChainer
Building Safe Multi-Program eBPF Environments
A Hands-on Tutorial on Cgroup-BPF Chaining and Monitoring
Prankur Gupta · Meta Platforms Takshak Chahande · Meta Platforms
SIGCOMM 2026 · Denver · Mon Aug 17 · Room 502 · Full-day Tutorial
PRESENTERS
Who we are
Prankur Gupta
Host Networking @ Meta
Takshak Chahande
Lead, Container & VM Networking @ Meta
Workshop Check-In : https://forms.gle/Z9r7k5gEjuZ4A3jo6
THE PROBLEM
Multiple independently-written eBPF programs now share the same hook in production — and the kernel gives you no safe way to compose them.
WHY IT MATTERS
This is everyone's problem now
1
FIFO
Kernel runs attached programs first-attached-first, and keeps only the LAST return value
2
SILENT
A later program can override an earlier security decision — with no signal
3
SCALE
At hyperscale these interactions become incidents affecting millions of connections
Cilium · Calico · Katran · bpfman all live here.
ROADMAP
The day, end to end
MORNING
Foundations & why it breaks
AFTERNOON
Building the solution
You leave able to reproduce a conflict AND build a chainer from scratch.
http://8.235.34.118:8080/s0x/��GitHub - https://github.com/takshak/bpfchainer
Workshop Check-In : https://forms.gle/Z9r7k5gEjuZ4A3jo6
LECTURE 1
A virtual machine brings its own kernel
LAB Strong isolation, and you pay for it: a whole kernel per guest, plus an arbiter underneath.
app
app
app
guest kernel
guest kernel
guest kernel
hypervisor
host kernel
hardware
The guest kernel is the isolation. Nothing above it can see anything belonging to another guest.
LECTURE 1
A container shares the kernel it runs on
LAB Same hardware, same kernel, no arbiter. Every separation now has to be built into that one kernel.
app
app
app
no guest kernel
one shared kernel
namespaces decide what each one sees · cgroups decide what each one may use
hardware
Take the guest kernel away and the isolation has to come from somewhere. Lecture 1 is that somewhere.
LECTURE 1
Eight namespaces: one per thing worth lying about
LAB A namespace virtualises one kind of resource. There is no master switch — you pick them one at a time.
Each one answers a single question: what does this process see when it looks at ___ ?
pid
CLONE_NEWPID
process numbers and the tree
mnt
CLONE_NEWNS
the filesystem it can reach
uts
CLONE_NEWUTS
hostname and domain name
ipc
CLONE_NEWIPC
shared memory, queues, semaphores
net
CLONE_NEWNET
interfaces, routes, sockets, conntrack
cgroup
CLONE_NEWCGROUP
its position in the cgroup tree
user
CLONE_NEWUSER
uid and gid mapping
time
CLONE_NEWTIME
boot and monotonic clocks
runc unshares the first six by default · user needs opt-in · time is rarely used
net is the only one with a datapath — which is why it costs the most, and gets its own slide.
LECTURE 1
Same command, two answers
LAB Nothing is blocked and nothing is copied. The kernel simply answers the question differently depending on who asks.
COMMAND
INSIDE c1
ON THE HOST
pid
ps aux | wc -l
init is pid 1 in here, 4212 out there
2
312
mnt
ls /
an overlayfs mount the host assembled
the image rootfs
the host rootfs
uts
hostname
one field, one namespace, no other effect
c1
prod-host-07
ipc
ipcs -q
no shared memory with anything outside
(empty)
14 queues
A namespace is not a wall. It is a different answer to the same question.
LECTURE 1
Two namespaces, two stacks, one queue between them
LAB Every layer on the left has a twin on the right. That duplication is the namespace.
netns c1
the container's own stack
socket
struct sock
TCP / UDP
tcp_sendmsg
IP · route · nft
ip_output
qdisc · txq
dev_queue_xmit
veth-c
veth_xmit
netns host
an identical, separate stack
socket
sk_data_ready
TCP / UDP
tcp_v4_rcv
IP · route · nft
ip_rcv
qdisc · txq
tx only — skipped
veth-h
napi poll
TX ↓
RX ↑
per-CPU backlog softnet_data->input_pkt_queue
raise NET_RX_SOFTIRQ → net_rx_action()
the sender's CPU stops here; the receive half runs in softirq
veth_xmit() does not send anything.
It hands the skb to the peer device and queues it. Nothing is copied — but the whole receive path must run again, on the other side of the boundary.
LECTURE 1
One packet out of the container, two full stacks
LAB Two netdevs, two traversals, one softirq — before the packet has touched the wire.
netns c1
stack 1 — TX
socket
tcp_sendmsg
ip_output
qdisc
veth_xmit
per-CPU backlog → raise NET_RX_SOFTIRQ → net_rx_action()
netns host
stack 2 — RX
napi poll
tc ingress
tc-BPF
ip_rcv
route
forward
then — TX
ip_output
qdisc
tc egress
tc-BPF
eth0
NIC
The container's stack does not send. It queues — and the host runs a whole stack again to finish the job.
LECTURE 1 · MOVE 3
The cgroup: what it may use, and who may watch
LAB cpu, memory, io and pids are metered here. Network is not — it is netns for topology plus eBPF for policy.
cgroup v2 — one tree
/sys/fs/cgroup
cgroup.subtree_control: cpu memory io pids
lab
cgroup.subtree_control: cpu memory io pids
c1
cgroup.procs: 4212 4260 4291
A controller only exists in a child if the parent lists it in subtree_control. Delegation is explicit, top down.
what it can actually limit
cpu
cpu.max 200000 100000
cpu.weight 100
cpu.stat usage_usec
memory
memory.max 2G
memory.high 1800M
memory.current 743M
io
io.max 8:0 rbps=10M
io.weight 100
io.stat rbytes wbytes
pids
pids.max 512
pids.current 37
pids.peak 91
network no such controller
cgroup v2 cannot account, limit or police a single byte of traffic.
cpu, memory, io and pids have controllers. Network has BPF — and that is the whole reason we are here.
LECTURE 1 · MOVE 4
Hiding the floor: the cgroup namespace
LAB unshare --cgroup · the container believes it sits at the root of the hierarchy
ls -Lo /proc/PID/ns/
HOST pid 1
CONTAINER pid 4212
pid
4026531836
4026532418
mnt
4026531840
4026532416
uts
4026531838
4026532417
ipc
4026531839
4026532419
net
4026531992
4026532421
user
4026531837
4026531837
cgroup
4026531835
4026532423
time
4026531834
4026531834
6 of 8 inodes differ from the host — that difference is the entire isolation
TWO VIEWS, ONE CGROUP
FROM THE HOST
cat /proc/4212/cgroup
0::/lab/c1
FROM INSIDE
cat /proc/self/cgroup
0::/
cat memory.max
2G — still enforced
A namespace about cgroups: it changes what the process sees, never what the kernel enforces.
LECTURE 1 · HANDOFF
Lab 1 — build the container by hand
Four verbs. create, enter, exec, destroy — cgroup slice, then unshare pid, mount and uts.�What to notice. One process, two PIDs. The cgroup is just a directory, and membership is a PID written into cgroup.procs.�The trap. nsenter joins namespaces, not cgroups. Skip the cgroup.procs write and every BPF counter in Labs 2–5 reads zero.
namespaces
nsenter
cgroup
cgroup.procs
you need
both
-> 30 minutes. Then Lecture 2 uses that directory.
LAB labs/01-container — sudo make verify must pass before the coffee break.
./container-runtime.sh create c1
./container-runtime.sh enter c1
./container-runtime.sh exec c1 cat /proc/self/cgroup
./container-runtime.sh destroy c1
LECTURE 2
eBPF internals: hooks, types, maps, lifecycle
[ diagram: program → hook → map ]
LAB Lab 2 — compile, load, and attach a cgroup/connect4 program to your Lab-1 container.
LECTURE 2
A program is compiled, proved safe, then wired to an event
LAB Nothing here is a module. The kernel accepts your code only after it can prove the code cannot hurt it.
USER SPACE
connect4.bpf.c
restricted C
clang -target bpf
-O2 -g, emits BTF
connect4.bpf.o
ELF: insns, maps, BTF
libbpf
relocates, opens maps
bpf(BPF_PROG_LOAD, &attr)
KERNEL
verifier
walks every path · bounded loops · no out-of-bounds · ≤ 1M insns · rejects with a log
JIT
BPF bytecode becomes native instructions
prog + BTF
an fd, refcounted, alive while something holds it
bpf(BPF_LINK_CREATE) / bpf_program__attach_cgroup()
ATTACHED
cgroup /sys/fs/cgroup/lab/c1 · BPF_CGROUP_INET4_CONNECT
from now on it runs on every connect() made by any process in that cgroup
LECTURE 2
Three places BPF can sit, and they know different things
LAB Pick the hook by what you need to know, not by what you need to block.
PER NETDEV
XDP
tc / tcx ingress
tc / tcx egress
context
struct xdp_md · struct __sk_buff
knows
a packet, on one interface, in one netns
blind to
which process or cgroup sent it
PER CGROUP
connect4 / connect6
sockops
cgroup_skb
sockopt · sysctl
context
bpf_sock_addr · bpf_sock_ops · __sk_buff
knows
the calling process, its cgroup, its socket
runs
inside the syscall, before the kernel acts
PER FUNCTION
kprobe / kretprobe
fentry / fexit
tracepoint
LSM
context
pt_regs · typed args via BTF
knows
any kernel function's arguments
used for
observability, and LSM for policy
Today is the middle column: the only family that runs with the caller's identity in hand.
LECTURE 2
connect4 runs inside the syscall, before the kernel acts
LAB Not a packet filter. A decision point in the caller's own context, with the address still writable.
connect(fd, &addr, len)
user space
__sys_connect()
security_socket_connect()
LSM
BPF_CGROUP_RUN_PROG_INET4_CONNECT
your program runs here
tcp_v4_connect()
only if the program allowed it
SYN goes out
struct bpf_sock_addr
user_ip4
readable — and writable
user_port
readable — and writable
protocol
IPPROTO_TCP / UDP
sk
the socket, for map keys
return 1 → connect proceeds, using whatever address you left behind
return 0 → connect() returns -EPERM
LECTURE 2
A map is the only thing a program can keep
LAB A BPF program has no globals and no heap. Anything that must survive one run lives in a map.
IN THE KERNEL
connect4 program
bpf_map_lookup_elem()
bpf_map_update_elem()
bpf_map_delete_elem()
BPF_MAP_TYPE_HASH policy_map
KEY
VALUE
10.0.0.7
{ deny, hits=41 }
10.0.0.9
{ allow, hits=2 }
10.0.1.4
{ deny, hits=0 }
fixed key and value size, declared before load
IN USER SPACE
your agent · bpftool
bpf(BPF_MAP_LOOKUP_ELEM)
bpf_map_get_next_key()
bpftool map dump
hash
any key, shared
array
index 0..n, fastest
percpu_*
one copy per CPU
lru_hash
evicts cold keys
ringbuf
events out, ordered
sockhash
sockets, for redirect
prog A: update(K, A) prog B: update(K, B) the kernel orders neither
Shared state is shared hazard. Two programs on one hook, one map, no ownership — Lecture 3 makes this happen on purpose.
Anatomy of cgroup-BPF execution
LECTURE 3
As we learned
One program per hook does not scale operationally.�Multiple programs per hook introduces composition problems.��
BPF_F_ALLOW_MULTI: mechanism, not policy
What it gives. Multiple programs can attach to one cgroup hook.�What it lacks. No product-level priority, ownership, rollback, or conflict signal.�Core lesson. Attachment coexistence is not semantic composition.
without flag
replace
ALLOW_MULTI
both run
conflict policy?
not defined
-> The kernel runs programs; it does not know feature intent.
LAB This is why Lab 3 studies hook-specific behavior before building a chainer.
LECTURE 3
Anatomy of cgroup-BPF execution
Different hooks expose different outputs
connect4
verdict
sockopt/sysctl
ctx value
sockops
reply
-> Different output channels mean different conflict rules.
LAB Do not generalize one hook behavior to every cgroup-BPF attach point.
Anatomy of cgroup-BPF execution
cgroup BPF does not provide a uniform hook��
Return verdicts. connect4, sock_create, cgroup_skb pass/drop.�Mutable context. sock_addr, sockopt, sysctl can change fields/values.�Mutable reply. sockops uses skops->reply for selected TCP callbacks.
The output channel defines the failure mode
LAB This is the taxonomy that Scenario A/B/C instantiate in lab 3
Anatomy of cgroup-BPF execution
Conflict Class | What conflicts | Examples | Risks |
return verdict | allow vs deny | connect4 | wrong final decision |
mutable context | final ctx value | sock_addr/sockopt/sysctl | validated value changes |
shared side effect | map key/value ownership | shared maps | misleading state |
mutable reply | final skops->reply | sockops ECN/timeout | TCP behavior changes |
packet mutation | packet metadata/data | cgroup_skb | mark/packet clobber |
The kernel runs an effective program list
userspace
cgroup hook
program array
-> No single universal last writer wins rule
LAB The hook-specific result handling is where most confusion starts.
Anatomy of cgroup-BPF execution
operation - userspace enters a kernel subsystem such as
connect() or TCP setup.
attach point - the subsystem invokes the cgroup-BPF hook
for that operation.
result handling - the hook decides whether to fold returns,
read ctx, or preserve side effects.
hook specific result handling
kernel continues or rejects operation
LECTURE 3 · CORE
Anatomy of cgroup-BPF execution
FIFO order. Programs run first-attached-first.�Who WIns? . Depending on hook type the kernel may keep only the final
program's update value or the first.�Shared-map races. Two programs, one map, no coordination → corruption
under load.
prog-1
DENY
prog-2
—
prog-3
ALLOW
→ ALLOW wins. Policy silently bypassed.
LAB Return-value override, BPF_F_ALLOW_MULTI semantics, shared-map races — invisible to standard tooling.
LECTURE 4 · PRODUCTION
The multi-program challenge, in the wild
Cilium + a security agent + an observability probe — same hook, three teams, zero coordination.
FAILURE 1
Sticky return
Later programs security pass not honored with sticky rejection
FAILURE 2
Map clobbering
Two programs write one shared map — state corrupts under load
FAILURE 3
Order dependence
Behavior changes when an unrelated team reattaches
Thesis: multi-program coordination must become a first-class discipline — not an accident.
LAB Lab 3 — reproduce these conflicts yourself
Lecture 4
connect4: later programs run, deny is sticky
Execution. Programs in the effective cgroup array are run in order.�Return folding. A zero return records -EPERM for connect4.�Important. A later allow does not clear the accumulated deny.
prog-1
DENY
prog-2
ALLOW
kernel
-EPERM
-> Not first-deny-stops. Not last-return-wins. Deny is sticky.
LAB Scenario A makes this visible with two connect4 programs.
while ((prog = READ_ONCE(item->prog))) {
func_ret = run_prog(prog, ctx);
if (!func_ret && !IS_ERR_VALUE((long)run_ctx.retval))
run_ctx.retval = -EPERM;
item++;
}
return run_ctx.retval;
Lecture 4
Return verdict conflict
program 1. returns deny for the target connect attempt.
program 2. returns allow and still executes after program 1.
observed the connection fails with EPERM because deny was folded.
deny prog ret = 0
allow prog ret = 1
connect() EPERM
-> Execution continued, enforcement stayed denied.
LAB This disproves the simplistic model that the last program always wins.
Lecture 4
Side effects are not rolled back
program 1. denies the connection and writes shared map state.
program 2. still runs and writes the same map key.
observed connection denied, but the map can show program 2 as last writer.
prog A deny + map=A
prog B map=B
final deny, B
-> Verdict and side effects are seperate outputs
LAB A denied operation can still leave misleading or clobbered shared state.
Lecture 4
sockops uses a mutable reply field
callbacks. sockops programs run during TCP lifecycle events.
context. the program receives struct bpf_sock_ops.
reply. selected callbacks use skops->reply as the answer back
to TCP code. Multiple programs can write the same reply field.��
struct bpf_sock_ops {
__u32 op;
union {
__u32 args[4]; /* passed to BPF */
__u32 reply; /* returned by BPF */
__u32 replylong[4];
};
...
};
* reply is different from the BPF program return value.
Lecture 4
sockops call chain
if (cgroup_bpf_enabled(CGROUP_SOCK_OPS) && sock_ops->sk) {
__sk = sk_to_full_sk(sock_ops->sk);
if (__sk && sk_fullsock(__sk))
__ret = __cgroup_bpf_run_filter_sock_ops(
__sk, sock_ops, CGROUP_SOCK_OPS);
}
return __ret;
TCP enters the cgroup sockops runner BPF_CGROUP_RUN_PROG_SOCK_OPS_SK
ret = BPF_CGROUP_RUN_PROG_SOCK_OPS(&sock_ops);
if (ret == 0)
ret = sock_ops.reply;
else
ret = -1;
return ret;
TCP reads skops->reply after BPF runs
static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
{
return (tcp_call_bpf(sk,
BPF_SOCK_OPS_NEEDS_ECN,
0, NULL) == 1);
}
ECN is one concrete reply consumer
Lecture 4
skops->reply affects more than ECN
timeout RTO
rwnd window
ecn bit
-> Same shared reply field, different TCP decisions.
LAB ECN is easiest to show in packets, but the pattern is broader.
BPF_SOCK_OPS_TIMEOUT_INIT. reply can select the SYN retransmission timeout.
BPF_SOCK_OPS_RWND_INIT. reply can select the initial advertised receive window.
BPF_SOCK_OPS_NEEDS_ECN. reply determines whether ECN is required for the connection.
BPF_SOCK_OPS_BASE_RTT: reply can report base RTT
base rtt
Lecture 4
Ordering-dependent sockops behavior
program A. sets skops->reply = 1 for BPF_SOCK_OPS_NEEDS_ECN.
program B. sets skops->reply = 0 for the same callback.
observed. TCP observes the final reply value for that callback.
ecn_on reply = 1
ecn_off reply = 0
TCP sees 0
-> Reverse the order and TCP sees 1.
LAB Packet capture can show the ECN capability bit changing during handshake.
Lecture 4
Why this is hard in production
explicit order, implicit ownership. You may know who runs first, but not who is allowed to own the final output.
cgroup hierarchy. parent/root programs may run outside the local workload team boundary.
testing gap. a program can pass isolated tests and still conflict with another program.
team A policy
team B tuning
team C telemetry
--> Silent behavior change, not just load failure.
LAB The production problem is coordination across owners and hooks.
Lecture 4
What each scenario proves
LAB Lab 3 is the evidence section for why deterministic composition is needed.
Scenario | Hook | Conflict | Lesson |
A | connect4 | return verdict | deny sticks; later still runs |
B | connect4 + map | side effect | state can be clobbered after deny |
C | sockops | mutable reply | final reply changes TCP behavior |
Lecture 4
Safe coexistence requires explicit semantics
know the hook. different attach types expose different output channels.
know the output. return value, ctx field, map state, packet metadata, or reply.
know the rule. folding, final-value observation, or side-effect persistence.���What’s Next?��Lecture 5 and lab 4. a small chainer owns the real attach point and runs priority slots.
Lecture 6 and lab 5. monitoring reports conflicts such as double writers.
LAB The production problem is coordination across owners and hooks.
LECTURE 5 · THE SOLUTION
From conflict to composition
The problem: cgroup-BPF runs programs FIFO and keeps only the last return. Independently-written programs clobber each other.��The need: call many programs, in a deterministic order, and mediate their returns — safely, without patching the kernel.��The primitive that makes it possible: the BPF trampoline.
We build a coordination layer in BPF itself — one program that calls the others.
LECTURE 5 · THE SOLUTION
Trampoline architecture for chaining
Interpose a coordination layer via BPF trampolines (freplace).��Deterministic order — not FIFO-by-luck�Mediated returns — explicit composition (e.g. any-DENY-wins)�Priority slots — programs register at a defined position
Lineage: xdp-chainer (2019, on freplace) → generalized to cgroup-BPF → BPFChain
LAB Lab 4 (centerpiece) — build a sockops chainer: add programs by priority slot, mediate skops→reply deterministically.
CONCEPT
What is a BPF trampoline?
A tiny piece of generated code that sits between a caller and a BPF program — it marshals the context and jumps into the target. It is the mechanism behind fentry/fexit and, crucially, freplace.
caller
calls f(ctx)
TRAMPOLINE
marshal ctx · jump
target program
runs instead
The trampoline is what lets one BPF program hand control to another at runtime — the foundation everything else here stands on.
KEY fentry / fexit / freplace are all built on the trampoline primitive.
CREDIT · THE ENABLING PATCH
BPF Trampoline
"Introduce BPF Trampoline" — kernel patch series, by Alexie bpf-next, Nov 2019.�
This patch introduced the trampoline into the kernel and made BPF_PROG_TYPE_EXT / freplace possible — attaching an extension program that replaces a specific function in an already-loaded BPF program.��Everything BPFChainer does is built on this primitive.
Our contribution in this tutorial: a clear block-diagram representation of this deep kernel concept — so you can see, not just read, how program replacement works.
BPFCHAIN · THE MECHANISM
20 slots, replaced by freplace
bpf_chainer(ctx) — the skeleton
int f1(ctx) { return PASS; } // stub
int f2(ctx) { return PASS; } // stub
...
int f20(ctx) { return PASS; } // stub
int bpf_chainer(ctx) {
s = f1(ctx); if (s) return s;
s = f2(ctx); if (s) return s;
...
s = f20(ctx); if (s) return s;
return s;
}
freplace attaches EXT programs into slots
f1
firewall
f5
load-balancer
f12
observability
libbpf freplace(EXT_prog → f{N}) → trampoline redirects f{N} to the attached program
KEY Deterministic slot order + return mediated at each step (non-zero = terminate) = safe composition.
LIFECYCLE
Lifecycle of a chained program
1 · OPEN
Load the user's EXT program object; set its attach target to a f{N} slot in bpf_chainer.
2 · PRIORITY
Translate the reserved priority slot into the target f{N} — this fixes execution order.
3 · LOAD
Verifier checks the EXT program; it is loaded against the chainer's slot signature.
4 · ATTACH
freplace wires the EXT program into the trampoline for f{N} — it now runs in place of the stub.
5 · EXECUTE
bpf_chainer calls f1..f20 in order; each attached program runs; non-zero return terminates the chain.
Same idea as xdp-chainer (2019), generalized to cgroup-BPF.
LECTURE 6
Monitoring & observability
LAB Lab 5 — emit ringbuf conflict events from a monitored chainer; inspect chain state.
Lecture 6
Monitoring multi-program eBPF chains
chain. deterministic composition is not enough without visibility
fleet. the real question is which chain and how many chains are
attached to which workload
lab. a constrained local demo showcasing how to monitor conflicts
workload -> chain?
extensions -> present?
conflicts -> seen?
-> Monitoring turns a chainer from code into an operable system.
LAB Lab 5 is a small local model of a larger production monitoring problem.
Lecture 6
Production monitoring starts with inventory
workload. service, container ID, host, cgroup path
attachment. hook type, chainer program ID, attach mode, attach timestamp
intent. define the expected chainer, slots, owners, and versions before checking for drift
LAB In the lab this is one cgroup; in production it is a reconciled fleet inventory.
Lecture 6
What cgroup inspection can prove
bpftool. shows direct programs attached to one cgroup and attach type
dispatcher. for a chainer design, the cgroup should show the dispatcher
limit. it does not show the logical extension chain behind the dispatcher
cgroup -> sockops
attached -> dispatcher
missing -> slots
-> cgroup show sees the cgroup surface, not the chainer internals.
LAB Useful first-level truth: is the dispatcher attached where we think it is?
Lecture 6
What cgroup inspection can prove
bpftool. shows direct programs attached to one cgroup and attach type
dispatcher. for a chainer design, the cgroup should show the dispatcher
limit. it does not show the logical extension chain behind the dispatcher
cgroup -> sockops
attached -> dispatcher
missing -> slots
-> cgroup show sees the cgroup surface, not the chainer internals.
LAB Useful first-level truth: is the dispatcher attached where we think it is?
Lecture 6
freplace slots are visible differently
dispatcher. is the cgroup-attached program
slots. are BPF_PROG_TYPE_EXT prog replacing chainer func
context. slot, priority, owner, and workload come from the registry
cgroup show -> dispatcher
prog show -> EXT
registry -> slot/owner
-> Loaded EXT program is not the same as expected chain membership.
LAB This is why “program is loaded” and “program is in this chain” are different checks.
Lecture 6
Kernel facts are necessary, not sufficient
actual. bpftool reports local kernel state at one point in time
expected. production needs owner, rollout, allowed callbacks, and version
drift. stale or unexpected programs require reconciliation and alerts
kernel -> facts
control plane -> intent
diff -> alert
-> Production monitoring is expected state plus actual state.
LAB bpftool is a diagnostic; it is not the fleet source of truth.
Lecture 6
Production monitoring architecture
orchestrator. provides workload identity and cgroup lifecycle
attach manager. records intended chainer and extension state
host agent. reconciles kernel attachments and loaded EXT programs
event stream. reports runtime conflicts, ownership violations, and unexpected behavior
orchestrator -> identity
registry -> intent
host agent -> actual
-> Events report violations; reconciliation catches drift.
LAB The lab approximates one host slice with local commands.
event stream -> violations
Lecture 6
How production chainer monitoring works
no maps. normal introspection should not require datapath chainer maps
metadata. loader knows slot, priority, target function, owner, and version
reconcile. host verifies dispatcher attachments, EXT slot programs, live links/pins, and BTF targets
registry -> intent
kernel -> actual
events -> violations
-> Use registry-backed monitoring; keep datapath overhead low.
LAB Debug maps in the tutorial are teaching instrumentation, not the production design.
Lecture 6
Runtime conflict signals
conditional. some conflicts appear only for specific traffic or callbacks
event. should identify hook/op, winner, conflicting writer, attempted value
signal. emit violations and conflicts, not every normal decision
op -> NEEDS_ECN
winner -> slot 1
conflict -> slot 6
-> Static inventory says what should run; events say what conflicted.
LAB Lab 5 uses ringbuf events to make this visible in a small setup.
Lecture 6
What the lab showcases
container. one cgroup stands in for a production workload cgroup
bpftool. shows dispatcher attachment and loaded EXT slot programs
instrumentation. debug maps and ringbuf events make conflicts teachable
one cgroup -> local
one chain -> visible
conflict -> reproducible
-> The lab is intentionally observable; production optimizes overhead.
LAB The lab demonstrates concepts, not the full production monitoring system.
Lecture 6
Lab vs. production monitoring
LAB This is the bridge between tutorial mechanics and production monitoring.
Lecture 6
Takeaways
bpftool. shows kernel facts, some ownership but not expected state
freplace. slots are EXT programs, not direct cgroup attachments
production. needs registry-backed reconciliation plus runtime violation events
-> Monitoring adds identity, ownership, expected state, and history.
LAB This sets up Lab 5 without making the lab architecture look like production.
Monitoring multi-program eBPF chains requires four views: actual kernel state, expected control-plane state, ownership metadata, and runtime conflict signals. The chainer provides the coordination point, but monitoring makes that coordination safe to operate.
WRAP-UP
What you built today
1
container by hand
2
attach a program
3
reproduce a conflict
4
build a chainer
5
monitor the chain
Production lessons at Meta scale · NetEdit, internal bpf-chainer, xdp-chainer [cleared numbers only — verify with OSS review]
Roadmap: planned architecture improvements · path toward open-sourcing BPFChain
Takeaway: multi-program coexistence is a discipline — chain deterministically, mediate returns, and make the chain observable.
prankur.07@gmail.com · takshak@gmail.com · github.com/takshak/bpfchainer