Versioning and releasing components outside of kubernetes/kubernetes

Lubomir I. Ivanov, neolit123, SIG Cluster Lifecycle, 6th April, 2020

What are these slides about?

• The Kubernetes project has a code organization issue - everything resides in k/k and it is hard to move it out.
• A few proposals are given.
• A decision has to be made on moving forward by defining a policy.
• Stakeholder SIGs for the time being are: Architecture, Release, CLI, Cluster Lifecycle, Testing.
• In the future more SIGs will be stakeholders if we further start splitting k/k.

Why should some components move out?

• Managing everything in the same GitHub repository is noisy and hard to maintain.
• k/k PR tests lack granulary and heuristics - e.g. scalability tests are run on a typo fix. Separate repositories can run the tests that they need.
• Components may bring unwanted dependencies.
• At the time of writing the Code Organization sub-project of SIG Architecture is supporting the move of kubeadm (Cluster Lifecycle) and kubectl (CLI). These are considered clients and not part of the core.

The problem statement

• There is no defined policy (or process) of how external components are going to be released as part of the k8s release artifacts.
• It is not clear if they must: A) follow the k8s release schedule, B) decouple from it, make their own choice on A vs B or do something else.
• While kubectl and kubeadm are client CLI tools, should the same policy apply to the k8s daemons in the future?
• “Staging” (via /staging and the publishing bot) is not considered as a solution here as it is supposed to be a temporary, intermediate step before extraction.

Current state

• All components that are being released in the k8s release process reside as source code in the kubernetes/kubernetes repository.
• All components follow the k8s release schedule.
• The components are built and packaged by Makefile rules in k/k.
• Some external components are already being coupled / managed with a k8s release, but they don’t follow its schedule:
• Kubernetes-cni plugins (bundled in the kubelet deb/rpm)
• Crictl (build/workspace.bzl)
• Etcd server, CoreDNS, other images…
• Other...

​

Pending extraction: kubectl

• Kubectl is present in /staging, but also in /cmd and /pkg.
• Has build rules and e2e tests in k/k.
• Has an issue tracker in k/kubectl, although a number of tickets land in k/k.
• Will be moved to k/kubectl.
• Prefers to stop following the k8s release schedule and versioning.
• Unclear (to the author) stance on the problems at hand.
• Tracking issue (and KEP) for moving kubectl to staging:
• https://github.com/kubernetes/enhancements/issues/1020

Pending extraction: kubeadm

• Resides in /cmd/kubeadm.
• Has build rules and e2e tests in k/k.
• Has an issue tracker in k/kubeadm, although a number of tickets land in k/k.
• Wants to move to k/kubeadm.
• Prefers to continue following the k8s release schedule and versioning.
• Has a “moving out” KEP with ~200 comments, but blocked on “no policy”:
• https://github.com/kubernetes/enhancements/pull/1425
• https://github.com/kubernetes/enhancements/issues/1424
• Has POC tools for branch and tag synchronization with k/k:
• https://github.com/neolit123/kubeadm-test

Proposal A: follow the release schedule

With this approach component C from repository k/c will be versioned exactly like k/k. The release x.yy.zz of k8s will include the component versioned as x.yy.zz.

Pros:

• All components are versioned the same way; less confusion for developers and users.
• All components follow the same schedule. Tests, bugs, features and KEPs are all tracked by the release team for that particular k8s release.
• Testing plan WRT skew remains unchanged.

Proposal A: follow the release schedule

Cons:

• Special tooling for automatic branch and tag synchronization between k/k and other repositories is required.
• Prone to a messy outcome in GitHub downtime scenarios. May require the occasional manual fix.
• The tags for the component repository can be synchronized only after k/k is tagged. This means that component C must be built from the HEAD of its matching “release-x.yy” branch instead of an existing tag.
• Potential for developer mistakes - e.g. pushing a last minute unwanted commit at HEAD of k/c.

Proposal B: decouple from the release schedule

With this approach component C from repository k/c will have its own versioning and release cycle. The release of k8s x.yy.zz will include a stable version of the component.

Pros:

• More freedom for the component developers to plan their work on bug fixes and features.
• Be governed by a true SemVer, as k/k does not really follow true SemVer.

Proposal B: decouple from the release schedule

Cons:

• Confusion for users and contributors WRT version skew of the component with the k8s version. Version matrixes needed.
• Should there be standardized machine readable way for users to fetch what is the supported version and skew of C?
• Unclear policy around release-blocking bugs, KEPs and test-signal.
• When k8s x.yy.0 is released supposedly the latest stable tag from component C will be picked for a release, but what version of C should we pick when x.yy.1 is released? Should there be branch mapping between k/k and k/c?

Proposal C: stop releasing components

With this approach component C from repository k/c will no longer be part of the k8s release.

Pros:

• Negates most cons from proposals A and B.
• Users have the freedom to build their own k8s “distribution”.

Proposal C: stop releasing components

Cons:

• Potential for C to still end up being k8s release blocking.
• Confusion in users what version of component C they are supposed to pick for creating a supported / working k8s cluster against the rests of k/k components. Should there be a machine readable way for users to find this out?
• The component must provide its own release mechanism - e.g. via GitHub releases. Should this be standardized?
• Unclear policy around KEPs and test-signal for component C. Should KEPs still go against k/enhancements?

Do not split the monolith

• This option is considered undesired by the majority of maintainers.
• SIG Architecture and component owning SIGs prefer to start moving components to external repositories, leaving k/k for the “core”.
• The definition of what is “core” may change over time.

Building external components for a k8s release

• Depending on the decision on how to version external components. There are still pending questions around actually building their source code and from where.
• A couple of known options are presented.

Proposal A: building components from k/k

During k/k’s “make release”, source code of external components can be pulled in and built.

Pros:

• A k/k version tag can apply to the build of the external component.
• Packaging remains untouched as long as the output folders remain the same as they are currently.

Cons:

• The build is no longer hermetic and air-gapped.
• The whole k8s build would fail if there errors in external components.

Proposal B: building components from k/release

Currently k/release tooling (Anago) clones k/k and execute “make release” to build the k/k artifacts. During this step external components can also be built.

Pros:

• k/k is not blocked on external component build errors.
• k/k only builds what is in its source tree.

Cons:

• The code for building release tarballs is currently in k/k.
• Difficult to execute due to the complexity of the build tooling. Anago is currently being refactored into a tool called krel.

Other big projects out there

• KDE
• Large collection of projects - desktop environments, frameworks, applications.
• Created in 1996. Uses GitHub as a mirror. Plasma desktop modules are mostly tagged the same way as the “plasma-desktop” tree (currently 5.18.*). The “k*” widgets are tagged the same way as their bundle (currently 20.03.*). The whole project is currently ~56 mil LOC.
• Linux kernel
• Created in 1991. Currently ~28 mil LOC. A monolith repository with strict ABI / API coupling.
• Has no near term plans to change its repository model.
• Qt Framework
• A framework for cross-platform app development.
• Created in 1995. Uses GitHub as a mirror. Uses Git modules. Currently ~3.5 mil LOC in the base module alone. Versions of all modules match the version of the core. The release process is managed by module maintainers with help from tooling.

​

Conclusion

• A new policy must be defined for moving components out of k/k.
• SIG Architecture and SIG Release, should define the policy.
• SIG Architecture should ensure that the resulting org is maintainable.
• SIG Release should ensure the resulting release process is manageable.
• SIG Release should ensure the work of the release team is manageable.
• SIG Testing should provide feedback on all topics related to testing of external components.
• The SIGs with pending extractions can provide feedback but ultimately need to comply with the resulting policy.