1 of 14

Kubernetes Network Drivers

The Future of Kubernetes Networking

aojea@google.com

Mar 2025

2 of 14

KEP-3698: Multi-Network #3700

closed

Dec 2022

A Multi network working groups was created with several groups with different goals and understanding

CNI / Multus: defending the existing market share
Kubevirt/VMs in Kube: Openstack Neutron
Kubernetes as a Router: Sonic and other complex and niche
Service Mesh to improve the relation with the Network plugins and improve the integration

The group KEP only presented APIs and never a solid plan of implementations, implementation details, undefined behavior, no commitment for work … leaving a lot of doubts about the implementation and sustainability of the project

The CNI was identified as a problem by this groups and as a solution a new project spun off called KNI was promoted, unclear on what problem it solved, some of the details were to rewrite the entire logic of the kubelet and make networking part of the kubelet and expose network capabilities through a new API that people will develop.

After reading all the comments, making suggestions to people to find a path forward it was clear that most of the people just wanted to have some indirection layer,. After a quick research, I could find the device plugins were enough to solve most of the CNI problems and DRA

3 of 14

User stories

I want to deploy an application that needs to use GPUs and some special NICs with RDMA capabilities to be able to use GPUDirect
I want to deploy an storage application that have to use a secondary interface connected to the vlan corresponding to the storage network.
I want to deploy an application that implements a CNF using two additional interfaces for the data plane and the main interface for the control plane
I want to attach a Pod to multiple VPCs in my cloud provider

4 of 14

KEP-3698: Multi-Network #3700

closed

Dec 2022

KEP-4477:

KNI [Kubernetes Networking Interface]

closed

Jan 2024

5 of 14

Kubernetes networking (today)

… and its consequences

Running a Pod are ~ two CRI operations

RunPodSandbox creates the network namespace and exec the CNI ADD call executing the plugins configured
CreateContainers start running the applications inside the created network namespace

Kubernetes use network plugins to provide the network functionality:

Pod IP and Interface
network policies (opt)
Services (opt)
advanced features (opt)

6 of 14

CNI is often mistakenly seen as the only way to handle container networking.
CNI's plugin chaining model creates dependencies between components, making development and updates complex.
CNI assumes full control of network operations, causing friction with external solutions and specialized needs.
The CNI model is stateless, making it hard to adapt to new technologies and build highly reliable network solutions.

REST

Network CNI plugin process

CNI binary

Container Runtime

OCI Runtime

CNI Spec

OCI Spec

CRI-API

REST

1. create namespaces

2. create network interface and assign IPs

3. create containers

7 of 14

The Future of K8s Networking

KEP-3698: Multi-Network #3700

closed

Kubernetes Contributor Summit EU

Apr 2024

Dec 2022

KEP-4477:

KNI [Kubernetes Networking Interface]

closed

Jan 2024

Things you can do today with the network

SIG Network, Mar 2024

A Multi network working groups was created with several groups with different goals and understanding

CNI / Multus: defending the existing market share
Kubevirt/VMs in Kube: Openstack Neutron
Kubernetes as a Router: Sonic and other complex and niche
Service Mesh to improve the relation with the Network plugins and improve the integration

The group KEP only presented APIs and never a solid plan of implementations, implementation details, undefined behavior, no commitment for work … leaving a lot of doubts about the implementation and sustainability of the project

The CNI was identified as a problem by this groups and as a solution a new project spun off called KNI was promoted, unclear on what problem it solved, some of the details were to rewrite the entire logic of the kubelet and make networking part of the kubelet and expose network capabilities through a new API that people will develop.

After reading all the comments, making suggestions to people to find a path forward it was clear that most of the people just wanted to have some indirection layer,. After a quick research, I could find the device plugins were enough to solve most of the CNI problems and DRA

8 of 14

The Future of Kubernetes Networking

1. The Need for a High-Level API

Abstraction:

Focusing on "what" we want to achieve rather than in the "how".
"Connect to Network A", "Isolate traffic between these Pods” instead of fiddling with low-level rules.

Flexibility:

Give developers the freedom to create higher-level abstractions.
Flexible Input but Standardized Output

2. The Need for a Low-Level API

Breaking Free from CNI Limitations:

Address the shortcomings of CNI.
Better integration with diverse networking technologies.
Create a more interconnected and interoperable ecosystem moving away from monopolies.

Declarative Power:

Makes applications more reliable and easier to manage.
More robust and self-healing applications.

9 of 14

High Level API: DRA

control

plane

pods

dra-net

dra- net

DRA

apiVersion: resource.k8s.io/v1alpha3

kind: ResourceSlice

…

spec:

devices:

- basic:

attributes:

vpc:

string: "blue"

encapsulation:

string: ether

ip:

string: 169.254.123.1/24

apiVersion: resource.k8s.io/v1alpha3

kind: ResourceSlice

10 of 14

Request Resources

apiVersion: resource.k8s.io/v1alpha3

kind: ResourceClaimTemplate

metadata:

spec:

devices:

requests:

- name: net1

deviceClassName: net1

control

plane

pods

dranet

DRA

apiVersion: resource.k8s.io/v1alpha3

kind: DeviceClass

metadata:

spec:

selectors:

- cel:

expression: device.driver == "networking.k8s.io"

apiVersion: v1

kind: Pod

metadata:

labels:

app: pod

spec:

containers:

- name: ctr1

image: registry.k8s.io/e2e-test-images/agnhost:2.39

resourceClaims:

- name: nccl-networks

resourceClaimTemplateName: nccl-interfaces

- name: nccl-network

resourceClaimName: nccl-interface

apiVersion: resource.k8s.io/v1alpha3

kind: ResourceClaim

metadata:

spec:

devices:

requests:

- name: net1

deviceClassName: net1

Scheduler allocates based on the resourceslices and claims

User reference in the Pod the ResourceClaim or ResourceClaimTemplate

Admin defines some DeviceClass on the cluster

11 of 14

Low Level API: OCI Spec “Linux Network Devices”

OCI Specification

CRI API

OCI defines how runtime implementations run OCI-compliant bundles and container configuration, containing low-level OS and host-specific details, on a particular platform.

Support for declarative configuration of Linux Network Devices

https://github.com/opencontainers/runtime-spec/pull/1271

No root capabilities required for network plugins

12 of 14

Low Level API: Network Interfaces

CNI:

Just add the PodIPs and the default interface. This is the problem it solves and solves it well, we don’t want to build on this hook.

NRI participates on all the Pod lifecycle hooks, has access to the networking information and runs as a daemon.

Preprovision of the Network Interface.

CDI

Can generate the json file to patch the OCI spec to include the network interface

13 of 14

The “bumpy” road so far

The Future of K8s Networking

KEP-3698: Multi-Network #3700

closed

Kubernetes Contributor Summit EU, Apr 2024

Dec 2022

KEP-4477:

KNI [Kubernetes Networking Interface]

closed

Jan 2024

Things you can do today with the network

SIG Network, Mar 2024

KEP-4817:Resource Claim Status (Network data)

alpha

Oct 2024

KEP-4817:Resource Claim Status (Network data)

beta

Apr 2025

A Multi network working groups was created with several groups with different goals and understanding

CNI / Multus: defending the existing market share
Kubevirt/VMs in Kube: Openstack Neutron
Kubernetes as a Router: Sonic and other complex and niche
Service Mesh to improve the relation with the Network plugins and improve the integration

The group KEP only presented APIs and never a solid plan of implementations, implementation details, undefined behavior, no commitment for work … leaving a lot of doubts about the implementation and sustainability of the project

The CNI was identified as a problem by this groups and as a solution a new project spun off called KNI was promoted, unclear on what problem it solved, some of the details were to rewrite the entire logic of the kubelet and make networking part of the kubelet and expose network capabilities through a new API that people will develop.

After reading all the comments, making suggestions to people to find a path forward it was clear that most of the people just wanted to have some indirection layer,. After a quick research, I could find the device plugins were enough to solve most of the CNI problems and DRA

14 of 14

Kubernetes Network Drivers

Kubernetes Network Drivers use DRA to expose Networks resources at the Node level that can be referenced by all the Pod (or all containers)

The network driver, before the RunPodSandbox is called, receives the NodePrepareResources rpc with the Devices and Configuration to use in the NRI hook.

This allows to keep backwards compatibility and remove the dependency in the existing monolithic network plugins and make networking composable.

CRI-API

Network

Driver

NRI / CDI

DRA

Secondary Network Interfaces

Additional network functionality

PodIPs