COSI - The Common Operating System Interface

Table of Contents

Table of Contents        1

Summary        1

Motivation        2

Container Linux Distributions        3

Goals & Non Goals        4

Goals        4

Non-Goals        4

Advantages to the Community        5

References        5

Systemd        5

IPC        5

Kubernetes        5

Summary

This proposal introduces COSI^[a], the Common Operating System Interface. Inspired by the Container Runtime Interface (CRI), Container Network Interface (CNI) and the Container Storage Interface (CSI), COSI will focus on the configuration of the underlying operating system, providing Protocol Buffer definitions and a gRPC API reference implementation for common system configuration settings such as DNS, network and node filesystems.

Motivation

The tight coupling of the kubelet with the Linux operating system is an ongoing challenge for the Kubernetes platform. As part of normal operation, the kubelet must communicate with the underlying operating system to configure pod networking, request storage, write files to the filesystem, manage processes and various other tasks.

Unfortunately, in most Linux systems the standard contract is often not an API, but the execution of a command line utility. This means that the node must have multiple binaries present and that the kubelet must shell out to execute these binaries, reading back unstructured output and exit codes.

An attempt to document all the kubelet dependences was started in 2016 in issue 26093, Identify, document and maybe remove kubelet dependencies. As of October 2020 this issue is still open. Perhaps the clearest definition of Kubernetes’ Linux operating system dependencies is located in the e2e test suite. These dependencies include Linux Kernel features and many individual binaries.

Another example of this coupling is KEP-2000 Graceful Node Shutdown, where the proposed implementation writes systemd files and sends a signal to logind.  This proposed solution is not applicable to Kubernetes distributions like Talos and k3OS that do not use systemd.

These examples highlight issues in the kubelet -> Linux contract, including:

1. Linux distributions have not developed standard APIs for most system configuration items. The canonical way to configure a Linux system is to invoke a shell command to a specific binary implementation with custom command line arguments.
   
2. Shell-based execution requires building command-line arguments in code and custom parsing of output (for example, see IPTables output parsing) and error handling via exit codes.
   
3. Many system utilities have custom configuration file formats, making validation, drift detection and the development of tooling more error-prone.
   
4. Competing security models (user/group/sudo/etc.) make it difficult to enforce and audit security settings at the operating system level.

5. Remote management is mainly by secure remote shell (SSH), which brings in a new set of security and management concerns.

Container Linux Distributions

A number of Linux distributions have emerged with a focus on containerized workloads. These distributions tend towards a minimal set of features, immutability, and management from a centralized control plane. However, these distributions diverge from each other in terms of system management tools and API endpoints.

Below^[b][c][d][e] is a table summarizing the various methods container-oriented Linux distributions are configured and managed:

COSI attempts to define a standardized configuration and API interface. This has the potential to simplify the kubelet and the development of distributions.

Goals & Non Goals

Goals

Define protocol buffer definitions for configuring operating system components (Network configuration, MOTD, Kernel Modules, sysctl, DNS servers, etc.)

Define a mechanism by which Kubernetes control and node components can communicate with arbitrary operating system configurations (‘osd’), including a plugin architecture that allows different components to be managed..

Develop a reference implementation that provides an API for managing the underlying operating system.

Non-Goals

        Prescribe implementation details of the operating system configuration software.

Possible Solutions

• Common settings for initial node configuration that can be transpiled to different userdata formats.

• gRPC interface^[i] to OS configuration

Advantages to the Community

Given the relative simplicity of container-based operating systems, there is limited ability for differentiation. However we see in these new operating systems the same trends that have plagued Unix and Linux for decades: incompatible APIs and configuration formats, a lack of separation between interface and implementation, and significant effort spent duplicating common functionality.

The Kubernetes community has seen enormous success in defining standard interfaces including the Container Runtime Interface (CRI), Container Network Interface (CNI) and the Container Storage Interface (CSI). The power of this pattern of common interfaces with pluggable backends allows users a single method of control and the development of a broad 3rd party community.

By defining an interface that container orchestration agents such as the kubelet can use, issues like 26093 are eliminated, as are potential incompatibilities in the versions of utilities (find, grep, etc.).

Benefits to the Kubernetes community include simplifying the kubelet, and may allow other methods of node management including control nodes communicating directly with worker nodes.

References

Systemd

• Systemd: Interface Portability and Stability Promise
• Varlink: a protocol for IPC. Includes a good overview of Linux IPC.
• systemd issue #235: sd-bus API documentation is incomplete

IPC

• Bus1
• TTRPC: GRPC for low-memory environments
• NSSS: nsswitch over unix domain sockets

Kubernetes

• Kubeadm initsystem
• Kubelet graceful shutdown

Other Linux Distributions

• SUSE - MicroOS