Understanding eBPF & XDP
Deep Dive into Kernel Architecture, Packet Processing & High-Performance Networking
SUSHANTH SATHESH RAO
Why eBPF?
Problems before eBPF:
eBPF Solution:
eBPF - Extended Berkley Packet Filter
eBPF is an in-kernel virtual machine that executes user-supplied programs safely, verifiably, and efficiently at specific kernel hook points.
Key properties:
eBPF Program Lifecycle (Step-by-Step)
Verifier: Why It Is Critical
Verifier enforces:
Verifier Internal Model
Verifier performs:
eBPF Registers, Stack, and Memory Model
BPF Instruction Set
eBPF MAPS
eBPF maps are key-value data structures residing in the Linux kernel that enable communication and data sharing between eBPF programs, and between eBPF programs and user-space applications.
Types of maps
General-purpose maps (common to all subsystems):
Networking-specific maps:
Linux Kernel Network Stack
OSI Layer | Linux Kernel | Kernel Source Paths | Key Files and/or Functions | What the code does |
Layer 1 — Physical | NIC hardware drivers | drivers/net/ and per-vendor dirs like drivers/net/ethernet/intel/, drivers/net/ethernet/mellanox/ | - Driver probe code - DMA ring setup - ndo_open, ndo_start_xmit implementations - IRQ handlers - NAPI polling functions | - Hardware initialization - DMA buffer management - Interrupt moderation - RX/TX descriptor handling - Integrates with NAPI framework |
Layer 2 — Data Link (Ethernet, ARP, VLAN) | Networking device layer + Ethernet stack | net/core/, net/ethernet/, net/802/ | - dev.c (core net_device logic) - eth.c (Ethernet helpers) - arp.c (ARP implementation) - vlan/ (802.1Q/802.1ad) - link_watch.c | - net_device struct & registration - Link state changes - ARP protocol - MAC address handling - VLAN tagging/stripping - Carrier detection |
Linux Kernel Network Stack (Contd.)
Layer 2.5 — Bridging & Tunneling | Linux bridge, VXLAN, GRE, Geneve, LWT | net/bridge/, net/core/lwtunnel/, net/ipv4/ip_gre.c, net/ipv6/ip6_gre.c, net/vxlan/ | - STP/RSTP - br_forward.c - VXLAN (udp tunnel driver) - GRE/IPIP - Lightweight tunnels | - Software L2 bridging - Overlay networks - Encapsulation / decapsulation |
Layer 3 — Network Layer (IPv4/IPv6 routing) | IP stack, routing table, netfilter | net/ipv4/, net/ipv6/, net/core/ | - ip_input.c, ip_forward.c - ip_output.c - route.c, fib_trie.c - IPv6: ip6_input.c, ip6_output.c, ip6_route.c | - Packet forwarding & routing - FIB lookup - PMTU discovery - ICMPv4/v6 - IP fragmentation / reassembly |
Layer 3.5 — Netfilter + XDP/TC ingress | Firewalling, packet mangling, eBPF attach points | net/netfilter/, kernel/bpf/, net/sched/ | - Netfilter hooks (nf_hook.c) - xtables matches/targets - cls_bpf, act_bpf (TC BPF) - XDP in drivers | - Firewall filtering - NAT - TC ingress classifiers - XDP pre-processing |
Linux Kernel Network Stack (Contd.)
Layer 4 — Transport (TCP/UDP/SCTP) | TCP/UDP protocols, congestion control | net/ipv4/tcp*.c, net/ipv4/udp.c, net/ipv6/tcp*.c, net/ipv6/udp.c, net/sctp/ | - tcp_input.c - tcp_output.c - tcp_ipv4.c, tcp_ipv6.c - udp.c - congestion control modules in net/ipv4/tcp_cong.c | - TCP/UDP processing - Sockets for transport layer - Retransmission, RTT estimation - Congestion control algorithms (CUBIC, BBR, etc.) |
Layer 4.5 — Socket Layer | BSD sockets interface, sk_buff, queues | net/core/ | - sock.c, skbuff.c - datagram.c - stream.c | - Socket API implementation - SKB allocation/free - Receive/transmit queues |
Layer 5 — Session Layerl | Not explicitly present | N/A | - N/A | - Session-layer protocols (like TLS) live in userspace libraries, not kerne |
Layer 6 - Presentation Layer | Not implemented in the kernel | N/A | - N/A | - Encoding/decoding handled in userspace |
Layer 7 - Application Layer | Kernel only supports some of them | net/{rpc, dns_resolver, sunrpc}/ | - SUNRPC - NFS client/server modules | - Limited app-layer protocols run in kernel - Almost all application-layer functionality is user space |
Quick Summary
Code | Purpose |
net/core/ | Central networking framework. SKB, qdisc, GRO/GSO, netdevice core. |
net/sched/ | Traffic control (TC) qdiscs, classifiers, BPF TC integration. |
net/bpf/ & kernel/bpf/ | eBPF interpreter, JITs, maps, program types. |
drivers/net/ | All NIC drivers: Ethernet, tunneling, bonding, wireless, etc. |
net/rxrpc | In-kernel RXRPC protocol |
net/mac80211/, drivers/net/wireless/ | Wi-Fi stack |
net/packet/ | AF_PACKET socket implementation |
High Level Overview
RX Path
RX Path - Physical / Data Link Layer
sk_buff
sk_buff cont*(header)
/* These two members must be first. */
struct sk_buff *next;
struct sk_buff *prev;
struct sock *sk; /* owner socket */
struct skb_timeval tstamp; /* arrival time */
struct net_device *dev; /* output dev */
struct net_device *input_dev; /* input dev */
/* Headers of the above OSI layers */ . . .
. . .
}
RX Path - Network Layer
RX Path - Transport Layer
RX Path - Application Layer
That completes the receive path
TX Path - Application Layer
TX Path - Transport Layer
Important data structures:
TX Path - Network Layer
Important Data structure:
TX Path - Data Link Layer
Queue Disciplines
The Qdisc (Queueing Discipline) controls how packets are queued and scheduled for transmission in Linux.
What a Qdisc is
It determines:
Types of Qdiscs
Special Qdiscs
Where Qdiscs live in the kernel
Key kernel structs:
Role of Qdiscs
Traffic Control Subsystem
TC is the high-level userspace API that configures:
It is configured by the tc command-line utility and interacts with the kernel via netlink.
Core components of TC:
Hooks in the network stack
TC Ingress
TC ingress is a hook for incoming packets after they have entered the kernel networking stack, before routing.
Key details:
Capabilities of TC ingress:
TC ingress limitations:
Usage examples:
__netif_receive_skb_core()
→ sch_handle_ingress() → tc_run() → tcf_classify()
→ tc_classify()
→ run filters (cls_bpf, cls_flower, etc.)
TC Egress
TC egress is triggered when packets are being sent out of an interface.
Egress responsibilities:
Capabilities:
dev_queue_xmit()
→ qdisc_run()
→ qdisc->enqueue()
→ qdisc->dequeue()
→ tc_classify()
TC-BPF
1. cls_bpf (Classifier BPF)
2. act_bpf (Action BPF)
Where TC-BPF fits in the kernel:
TC-BPF program behavior:
eXpress Data Path (XDP)
xdp_md
struct xdp_md {
__u32 data; --> Packet start pointer
__u32 data_end; --> Packet end pointer
__u32 data_meta; --> Metadata region (optional)
__u32 ingress_ifindex;
__u32 rx_queue_index;
};
XDP Actions
ACTION | Description |
XDP_ABORTED | Program error, drop with warning |
XDP_DROP | Silently drop packet |
XDP_PASS | Continue normal stack processing |
XDP_TX | Send packet out of same interface (hairpin) |
XDP_REDIRECT | Redirect to another interface/CPU/map |
Different modes of XDP
Native Mode:
Generic Mode:
Offload Mode:
Networking specific maps
Xskmap: The xskmap is used to redirect raw XDP network frames directly to user space via AF_XDP sockets (XSKs), completely bypassing the standard Linux networking stack. This allows for extremely high-speed packet processing in user-space applications. User space creates an AF_XDP socket and inserts its file descriptor into the map using an integer key (typically the queue ID). An XDP eBPF program then uses the bpf_redirect_map() helper to forward incoming packets to the corresponding socket specified by the key.
Cpumap: The cpumap is used to implement software-based Receive Side Scaling (RSS) or to offload packet processing from one CPU to another. This enables efficient distribution of network traffic across multiple CPU cores, particularly useful on hardware that lacks native RSS support. An XDP program running on the initial receiving CPU redirects the packet (using bpf_redirect_map()) to an entry in the cpumap. Each entry in the map is associated with a dedicated kernel thread running on a different, "remote" CPU. This remote CPU handles the rest of the processing, including allocating the necessary SKB (socket buffer) and feeding it into the normal network stack. A second XDP program can even be attached to the remote CPU entry for further processing.
AF_XDP
AF_XDP (Contd)
UMEM is a contiguous user-space memory region that you register with the kernel for AF_XDP. The UMEM is:
UMEM:
AF_XDP (Contd)
UMEM
A frame (UMEM slot) cycles through ownership states. Typical lifecycle for receiving a packet:
Thus ownership transfers are always mediated by these rings.
Key invariants / rules:
Typical AF_XDP (userspace)
TC-BPF vs XDP
XDP:
TC:
Feature | TC-BPF | XDP |
Operates on | SKB | Raw frame (XDP buffer) |
Runs | Later | Earliest possible |
Can reshape traffic | Yes | No |
Can modify SKB metadata | Yes | No SKB at all |
Speed | Slower than XDP | Fastest |
Use cases | shaping, classification | filtering, early drop, AF_XDP redirect, L{2-4} load balancing |
Deep Packet Inspection
Bpftool - Lifeline in debugging eBPF
pL4S - Low Latency Low Loss Scalable Throughput
IPv6 Extension Header
Traffic Shaping
# create clsact qdisc
tc qdisc add dev eth0 clsact
# attach BPF to ingress (direct-action)
tc filter add dev eth0 ingress bpf da obj tb_policer.o sec classifier
Traffic Shaping
eBPF Program Performance
Verifier Performance
Interpreting eBPF JIT Output
if (cursor + sizeof(struct ethhdr) > data_end) return XDP_ABORTED;
eth = cursor;
if (cursor + sizeof(struct ethhdr) + sizeof(struct iphdr) > data_end)
return XDP_DROP;
10: mov rax,[r1]
14: mov rcx,[r1+8]
18: add rax,0xe
1c: cmp rax,rcx
1f: jae abort
25: mov rax,[r1]
29: add rax,0x22
2d: cmp rax,rcx
30: jae drop
cursor += sizeof(struct ethhdr) + sizeof(struct iphdr);
if (cursor > data_end)
return XDP_DROP;
10: mov rax,[r1]
14: mov rcx,[r1+8]
18: add rax,0x22
1c: cmp rax,rcx
1f: jae drop
Interpreting eBPF JIT Output
if (cursor + sizeof(struct ethhdr) > data_end) return XDP_ABORTED;
eth = cursor;
if (cursor + sizeof(struct ethhdr) + sizeof(struct iphdr) > data_end)
return XDP_DROP;
10: mov rax,[r1]
14: mov rcx,[r1+8]
18: add rax,0xe
1c: cmp rax,rcx
1f: jae abort
25: mov rax,[r1]
29: add rax,0x22
2d: cmp rax,rcx
30: jae drop
cursor += sizeof(struct ethhdr) + sizeof(struct iphdr);
if (cursor > data_end)
return XDP_DROP;
10: mov rax,[r1]
14: mov rcx,[r1+8]
18: add rax,0x22
1c: cmp rax,rcx
1f: jae drop
eBPF Limitations
Guaranteed Termination: Programs must be proven to terminate. Unbounded loops or infinite recursion are not allowed, though bounded loops (where the maximum iteration count can be determined statically) are now supported.
Resource Limits: Programs are subject to size and complexity limits. The current instruction count limit per execution path is approximately one million instructions (as of 2025), which the verifier uses to ensure performance and safety.
Memory Safety: Programs cannot access arbitrary kernel or user memory. Memory accesses must be within valid bounds and are validated using helper functions like bpf_probe_read_user or by accessing data within the program's defined context.
Restricted Function Calls: eBPF programs can only call a predefined set of stable, version-agnostic kernel "helper" functions. They cannot call arbitrary kernel functions or use the full kernel API, which aids compatibility across different kernel versions.
Privilege Requirements: Loading eBPF programs typically requires root privileges or the CAP_BPF Linux capability. Unprivileged eBPF is available but comes with stricter limitations on functionality and access.
Complexity of Development: Writing safe and efficient eBPF code requires a deep understanding of kernel internals, the eBPF virtual machine, and the nuances of the verifier.
Limited Stack Space: The maximum stack space for an eBPF program is limited to 512 bytes. This necessitates using eBPF maps to store larger data structures or pass data between different programs or to user space, adding complexity.
Kernel Version Dependency: eBPF features and verifier behavior can change between Linux kernel versions, leading to potential compatibility issues in mixed environments.
Debugging Difficulties: Debugging eBPF programs can be challenging because they run inside the kernel with limited access to traditional debugging tools used for user-space applications.
Time-of-Check to Time-of-Use (TOCTOU) Issues: Due to the lack of concurrency primitives (like reliable locks that can be acquired from eBPF) and concurrent execution, programs may encounter race conditions when reading kernel data.
Thank you!
Questions?��Let’s discuss!