1 of 28

Configuring Your Kubernetes Cluster�on the Next Level

Lucas Käldström14th of November 2018 - KubeCon Shanghai

2 of 28

$ whoami

Lucas Käldström, Upper Secondary School Student, �19 years old

CNCF Ambassador, Certified Kubernetes Administrator and Kubernetes SIG Lead

Speaker at KubeCon in Berlin, Austin & Copenhagen

Kubernetes approver and subproject owner, active in the community for ~3 years

Driving luxas labs which currently performs contracting for Weaveworks

A guy that has never attended a computing class

3 of 28

Agenda

  1. Where are we at today?
    1. Current status & limitations
  2. Where do we want to go?
    • ComponentConfig hallway pitch
    • Solutions and conventions
    • Technical deep-dive
  3. Let’s write a ComponentConfig Go application!
  4. The future
    • Roadmap & how you can help

4 of 28

  1. Current status

5 of 28

Current status

  1. Kubernetes releases at a fast pace -- every three months, hard to keep up
  2. As the software is quickly evolving, the configuration knobs are too
    1. => backwards-incompatible changes in configuration needs to be made
  3. The admin interface when configuring Kubernetes components is flags
    • If you try to use a flag removed in new version => exit 1
    • Need to build complex version-dependent flag set logic
  4. Flag names and functionality is a bit inconsistent between components
    • (Currently also many inconsistencies in the underlying config structs)

6 of 28

Current status as of v1.12

  • External ComponentConfig structs are available for components:
    • k8s.io/kubelet (v1beta1)
    • k8s.io/kube-proxy (v1alpha1)
    • k8s.io/kube-scheduler (v1alpha1)
    • k8s.io/kube-controller-manager (v1alpha1)
  • The kubelet, kube-proxy and kube-scheduler can read ComponentConfig from a file using the `--config` flag
    • Only the kubelet has an API version that is beta or higher, so only for the kubelet it is recommended to use ComponentConfig in production
  • Shared types:
    • LeaderElectionConfiguration and DebuggingConfiguration are hosted in k8s.io/apiserver
    • ClientConnectionConfiguration is hosted in k8s.io/apimachinery

7 of 28

  1. Enter ComponentConfig

8 of 28

apiVersion: kubecontrollermanager.config.k8s.io/v1kind: KubeControllerManagerConfigurationcontrollers: csrSigning: clusterSigningCertFile: /some/path namespace: concurrentNamespaceSyncs: 5 nodeLifecycle: enableTaintManager: true

End goal:

$ kube-controller-manager --config config.yaml

9 of 28

Hallway pitch

  1. All components read a config file that follows Kubernetes API conventions*
  2. When upgrading, the new binary can still read the older API structure
  3. Higher-level systems can easily access the config structs via vendoring
  4. Common configuration types are shared between components
  5. The UX for integrating with any Kubernetes-like component gets consistent

*To a reasonable extent. ComponentConfigs aren’t REST resources, so some of the guidelines aren’t applicable

10 of 28

ComponentConfig conventions

API group name: {samplecomponent}.config.k8s.io

API version: Follows K8s standard, e.g. v1alpha1, v1alpha2, v1beta1, v1, v2beta1, v2

Kind name: {SampleComponent}Configuration

k8s.io�├── {component}�│ └── config�│ └── {version}�│ └── types.go�└── kubernetes� └── pkg� └── {component}� └── apis� └── config

├── types.go� ├── scheme� │ └── scheme.go� └── {version}� └── types.go

apiVersion: kubecontrollermanager.config.k8s.io/v1beta1kind: KubeControllerManagerConfigurationcontrollers: csrSigning: clusterSigningCertFile: /some/path namespace: concurrentNamespaceSyncs: 5 nodeLifecycle: enableTaintManager: true

Code structure as of v1.12:

11 of 28

Common / shared config reuse

One common “problem” so far is that the components have been using nearly the same types of fields, but using custom and different schemas. Real-world example on the right.

To mitigate this, we’ve introduced “shared config types” packages in:�- k8s.io/apimachinery/pkg/apis/config/v1alpha1�- k8s.io/apiserver/pkgs/apis/config/v1alpha1

These packages host structs like `LeaderElectionConfiguration` and `ClientConnectionConfiguration`.�This shared set will grow over time.

apiVersion: kubeproxy.config.k8s.io/v1alpha1kind: KubeProxyConfigurationclientConnection: burst: 10 contentType: application/vnd.kubernetes.protobuf kubeconfig: /var/lib/kube-proxy/kubeconfig.conf qps: 5

apiVersion: kubelet.config.k8s.io/v1beta1kind: KubeletConfigurationcontentType: application/vnd.kubernetes.protobufkubeAPIBurst: 10kubeAPIQPS: 5kubeConfig: /etc/kubernetes/kubelet.conf

Real-world example:

12 of 28

External and internal versions

  • An external type is a struct that can be encoded and decoded.
    • It must have JSON tags set on all its fields�
  • An internal type is a struct that the program uses internally in all code
    • It should not have JSON tag set.
    • The internal type’s schema should equal the schema of the latest stable external type.�
  • External types are defaulted
    • After an external type has been decoded from a file, it is automatically defaulted�
  • External types register conversions to the internal type
    • This in order to make is possible for the program to use the config post-decoding�
  • Auto-generated code is created to minimize what needs to be written
    • Auto-generated code is used for conversion, nested defaulting and implementing interfaces

13 of 28

  1. Sample Go application

14 of 28

Guide: Use ComponentConfig for your app

Let’s say you’re building an addon to Kubernetes, and want to it to have the same “look and feel” as the other k8s components. You don’t like to reinvent the wheel and reimplement all the API machinery features, but re-use the Kubernetes API machinery framework.

Here’s a walkthrough of the files to be written for it to work:

  • types.go -- Contains the types for the API group
  • register.go -- Registers the types in a SchemeBuilder.
  • doc.go -- Includes meta tags that act as parameters to code generators
  • + auto-generated code for defaulting, conversions and deepcopy interfaces
  • For external packages only:
    • conversion.go -- Conversion code from external to internal API.
    • defaults.go -- Contains defaulting functions for the structs.

15 of 28

Guide: Directory structure

Here’s an example directory structure.�A couple of things to note:

  • The internal types are in `pkg/apis/config`
    • It does not register any defaulting or conversions.
  • The external types are versioned and defaulted
    • (Ideally) lossless conversions exist between�every external type and the internal type
  • `scheme.go` is in a dedicated package
    • This allows for easy imports and encoding/decoding
  • The internal types should equal the latest stable types
    • The latest stable types’ conversions to internal == no-op

github.com/luxas/sample-config�├── cmd�│ └── sample-config�│ └── main.go�└── pkg� └── apis� └── config

├── doc.go� ├── register.go

├── types.go� ├── scheme� │ └── scheme.go� ├── v1� │ ├── defaults.go

│ ├── doc.go� │ ├── register.go� │ └── types.go� └── v1beta1

├── conversion.go� ├── defaults.go

├── doc.go� ├── register.go� └── types.go

16 of 28

Guide: What we’ll do

  • A Go program that reads the a file with either v1 or v1beta1 config, if `--config` is set, or outputs the default v1 config. How we’ll do this:
    1. Create an internal package in `pkg/apis/config` with the internal MyAppConfiguration struct.
    2. Create two external packages in `pkg/apis/config/{v1,v1beta1}` with external structs
    3. Create defaulting logic for the external types in `defaults.go` files
    4. Create custom conversion logic from v1beta1 -> internal type (== v1)
    5. Create a scheme with the internal and external packages linked
    6. Create a small program in `cmd/sample-config` that reads a file into the internal config, or�uses the v1 defaults

17 of 28

Guide: What we’ll do

v1.MyAppConfiguration{}

$ bin/sample-config --config [config-file]

If [config-file] is set:

If [config-file] is not set:

config.yaml (v1|v1beta1)

Read

Decode

Default

Convert

Default

Convert

Internal type for use in program

Convert to v1

Encode

stdout

Flags

18 of 28

Guide 1/7: The types.go file

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Objecttype MyAppConfiguration struct {� metav1.TypeMeta�� // ClientConnection configures the connection to Kubernetes� ClientConnection apimachineryconfig.ClientConnectionConfiguration� // LeaderElection configures so the component can be HA-deployed� LeaderElection apiserverconfig.LeaderElectionConfiguration� // Server holds configuration settings for the HTTPS server� Server ServerConfiguration�}�type ServerConfiguration struct {� // Default: "0.0.0.0"// +optional� Address string// Default: 10250// +optional� Port uint32// +optional� TLSCertFile string// +optional� TLSPrivateKeyFile string�}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Objecttype MyAppConfiguration struct {� metav1.TypeMeta `json:",inline"`�� // ClientConnection configures the connection to Kubernetes� ClientConnection apimcfgv1.ClientConnectionConfiguration `json:"clientConnection"`// LeaderElection configures so the component can be deployed in HA mode on k8s� LeaderElection apiscfgv1.LeaderElectionConfiguration `json:"leaderElection"`// Default: "0.0.0.0"

// +optional� ServerAddress string `json:"serverAddress"`// Default: 10250// +optional� HTTPSPort uint32 `json:"port"`// TLSConfig holds settings for the TLS configuration� TLSConfig TLSConfig `json:"tlsConfig"`�}��type TLSConfig struct {� // +optional� TLSCertFile string `json:"tlsCertFile"`// +optional� TLSPrivateKeyFile string `json:"tlsPrivateKeyFile"`�}

External v1beta1 with json tags

Internal, has same schema as v1, but no json tags

Any serializable type must implement runtime.Object and embed TypeMeta!

19 of 28

Guide 2/7: The register.go file

Internal package: types & deepcopy

External package: types & deepcopy + defaulting & conversions

package v1beta1��const GroupName = "config.luxaslabs.com"var (� SchemeBuilder = runtime.NewSchemeBuilder(� addKnownTypes,� addDefaultingFuncs,� )� localSchemeBuilder = &SchemeBuilder� AddToScheme = localSchemeBuilder.AddToScheme� // SchemeGroupVersion is'the group & version for this scheme� SchemeGroupVersion = schema.GroupVersion{� Group: GroupName,� Version: "v1beta1",� }�)��func addKnownTypes(scheme *runtime.Scheme) error {� scheme.AddKnownTypes(SchemeGroupVersion, &MyAppConfiguration{})� return nil�}

package config��const GroupName = "config.luxaslabs.com"var (� SchemeBuilder = runtime.NewSchemeBuilder(� addKnownTypes,� )� localSchemeBuilder = &SchemeBuilder� AddToScheme = localSchemeBuilder.AddToScheme� // SchemeGroupVersion is'the group & version for this scheme� SchemeGroupVersion = schema.GroupVersion{� Group: GroupName,� Version: runtime.APIVersionInternal,� }�)��// Adds the list of known types to the given scheme.func addKnownTypes(scheme *runtime.Scheme) error {� scheme.AddKnownTypes(SchemeGroupVersion, &MyAppConfiguration{})� return nil�}

20 of 28

Guide 3/7: The doc.go file

In doc.go, it is specified how the autogenerated code should be generated.

  • `+k8s:deepcopy-gen` should always be `package` for ComponentConfig pkgs.
    • deepcopy-gen makes serializable configs implement runtime.Object.
  • `+k8s:conversion-gen` points to the internal package and optional shared pkgs
    • conversion-gen creates Convert_v1_To_v2 functions and registers those conversions
  • `+k8s:defaulter-gen` should always be `TypeMeta` for ComponentConfig pkgs.
    • defaulter-gen registers defaulting funcs for all types that embed the parameter (TypeMeta).

// +k8s:deepcopy-gen=package// +k8s:conversion-gen=github.com/company/my-app/pkg/apis/config// +k8s:conversion-gen=k8s.io/apimachinery/pkg/apis/config/v1alpha1// +k8s:conversion-gen=k8s.io/apiserver/pkg/apis/config/v1alpha1// +k8s:defaulter-gen=TypeMetapackage v1beta1

// +k8s:deepcopy-gen=packagepackage config

External sample:

Internal sample:

21 of 28

Guide 4/7: The defaults.go file

This file must include the `addDefaultingFuncs(scheme)` function that in turn calls the auto-generated `RegisterDefaults(scheme)` function. The format of the defaulting function is `SetDefaults_{SampleStruct}`.�Defaulting fns from shared pkgs are optional and can be called via `RecommendedDefault{SampleStruct}`.

package v1beta1��func addDefaultingFuncs(scheme *runtime.Scheme) error {� return RegisterDefaults(scheme)�}��func SetDefaults_MyAppConfiguration(obj *MyAppConfiguration) {� if len(obj.ServerAddress) == 0 {� obj.ServerAddress = "0.0.0.0"� }� if obj.HTTPSPort == 0 {� obj.HTTPSPort = 9090� }�� apimachineryconfigv1.RecommendedDefaultClientConnectionConfiguration(&obj.ClientConnection)� apiserverconfigv1.RecommendedDefaultLeaderElectionConfiguration(&obj.LeaderElection)�}

22 of 28

Guide 5/7: The conversions.go file

For straightforward conversions between structs, where the field name and type match, autogenerated code is created automatically. But when more complex changes are made to a new API, you must write the `Convert_{v1}_{SampleStruct}_To_{internal}_{SampleStruct}` function yourself.

Remember to first call the (partial) autogenerated portion of the conversion!

package v1beta1��func Convert_v1beta1_MyAppConfiguration_To_config_MyAppConfiguration(in *MyAppConfiguration, out *config.MyAppConfiguration, s conversion.Scope) error {� if err := autoConvert_v1beta1_MyAppConfiguration_To_config_MyAppConfiguration(in, out, s); err != nil {� return err� }� out.Server.Address = in.ServerAddress� out.Server.Port = in.HTTPSPort� out.Server.TLSCertFile = in.TLSConfig.TLSCertFile� out.Server.TLSPrivateKeyFile = in.TLSConfig.TLSPrivateKeyFile� return nil�}

23 of 28

Guide 6/7: The scheme.go file

A scheme holds all registered types, defaulting and conversion functions centralized in one place. Use the `AddToScheme` functions to register a specific package.

In the scheme package, all known versions are aggregated and registered in the global Scheme variable.

Codecs provide encoding/decoding functions for the types in the scheme

package scheme��var (� // Scheme is the scheme to which all API types are registered.� Scheme = runtime.NewScheme()�� // Codecs provides access to encoding and decoding for the scheme.� Codecs = serializer.NewCodecFactory(Scheme)�)��func init() {� AddToScheme(Scheme)�}��// AddToScheme builds thescheme using all known versions of the API.func AddToScheme(scheme *runtime.Scheme) {� utilruntime.Must(config.AddToScheme(Scheme))� utilruntime.Must(v1beta1.AddToScheme(Scheme))� utilruntime.Must(v1.AddToScheme(Scheme))� utilruntime.Must(scheme.SetVersionPriority(v1.SchemeGroupVersion))�}

24 of 28

Guide 7/7: The hard work pays off!

Now, we can read any file with an external version directly into the internal version the program may use using `runtime.DecodeInto()`.

�The internal version can also be populated from defaults in an external version using Scheme.Default() & .Convert()

�Finally, it’s easy to marshal any object into whatever form using a generic encoder for a specific external version, and `runtime.Encode()`

// decodeFileInto reads a file and decodes the it into an internal typefunc decodeFileInto(filePath string, obj runtime.Object) error {� content, err := ioutil.ReadFile(filePath)� if err != nil { return err }� // Regardless of if the bytes are of the v1 or v1beta1 version,// it will be read successfully and converted into the internal versionreturn runtime.DecodeInto(scheme.Codecs.UniversalDecoder(), content, obj)�}�// populateV1Defaults populates cfg based on v1 defaultsfunc populateV1Defaults(cfg *config.MyAppConfiguration) error {� // Create a new config of some external version,// default it, convert it into the internal version� v1cfg := &v1.MyAppConfiguration{}� scheme.Scheme.Default(v1cfg)� return scheme.Scheme.Convert(v1cfg, cfg, nil)�}

// marshalYAML marshals any ComponentConfig object registered in the scheme for the specific versionfunc marshalYAML(obj runtime.Object, groupVersion schema.GroupVersion) ([]byte, error) {� // yamlEncoder is a generic-purpose encoder to YAML for this scheme� yamlEncoder := json.NewYAMLSerializer(json.DefaultMetaFactory, scheme.Scheme, scheme.Scheme)� // versionSpecificEncoder writes out YAML bytes for exactly this v1beta1 version� versionSpecificEncoder := scheme.Codecs.EncoderForVersion(yamlEncoder, groupVersion)� return runtime.Encode(versionSpecificEncoder, obj)�}

25 of 28

  1. Roadmap

26 of 28

Roadmap

  • v1.11
    • The KEP for this feature was designed, reviewed and approved
    • Internal refactoring for kubelet, kube-proxy, kube-controller-manager and kube-scheduler
    • The k8s.io/{component} repos for the mentioned components were created
  • v1.13
    • Improve unit, API roundtrip, defaulting and validation testing coverage
    • Start the work of graduating the schemas of kube-proxy, kube-controller-manager and kube-scheduler from v1alpha1 to v1beta1
  • v1.14
    • Refactor the API server’s codebase to support reading configuration from a file
    • (Hopefully by this time) Have all components’ API versions be beta or higher

27 of 28

Recap

  1. ComponentConfig is the pattern of storing the config in a file
    1. This allows for “GitOps”-style deployments of Kubernetes components
  2. This effort aims to provide an unified UX for configuring Kubernetes
    • But you can also use these patterns in your Kubernetes extension or any application!
  3. With the k8s ComponentConfig types in `k8s.io/*` repos, it’s possible to create higher-level tools that configure Kubernetes declaratively
  4. Going forward we need component-focused contributors to design a stable (v1beta1 or higher) schema for all Kubernetes components

28 of 28

Thank you!

Get in touch:

@luxas on Github�@kubernetesonarm on Twitter�lucas@luxaslabs.com