Configuring Your Kubernetes Cluster�on the Next Level
Lucas Käldström�14th of November 2018 - KubeCon Shanghai
$ whoami
Lucas Käldström, Upper Secondary School Student, �19 years old
CNCF Ambassador, Certified Kubernetes Administrator and Kubernetes SIG Lead
Speaker at KubeCon in Berlin, Austin & Copenhagen
Kubernetes approver and subproject owner, active in the community for ~3 years
Driving luxas labs which currently performs contracting for Weaveworks
A guy that has never attended a computing class
Agenda
Current status
Current status as of v1.12
apiVersion: kubecontrollermanager.config.k8s.io/v1�kind: KubeControllerManagerConfiguration�controllers:� csrSigning:� clusterSigningCertFile: /some/path� namespace:� concurrentNamespaceSyncs: 5� nodeLifecycle:� enableTaintManager: true |
End goal:
$ kube-controller-manager --config config.yaml |
Hallway pitch
*To a reasonable extent. ComponentConfigs aren’t REST resources, so some of the guidelines aren’t applicable
ComponentConfig conventions
API group name: {samplecomponent}.config.k8s.io
API version: Follows K8s standard, e.g. v1alpha1, v1alpha2, v1beta1, v1, v2beta1, v2
Kind name: {SampleComponent}Configuration
k8s.io�├── {component}�│ └── config�│ └── {version}�│ └── types.go�└── kubernetes� └── pkg� └── {component}� └── apis� └── config ├── types.go� ├── scheme� │ └── scheme.go� └── {version}� └── types.go |
apiVersion: kubecontrollermanager.config.k8s.io/v1beta1�kind: KubeControllerManagerConfiguration�controllers:� csrSigning:� clusterSigningCertFile: /some/path� namespace:� concurrentNamespaceSyncs: 5� nodeLifecycle:� enableTaintManager: true |
Code structure as of v1.12:
Common / shared config reuse
One common “problem” so far is that the components have been using nearly the same types of fields, but using custom and different schemas. Real-world example on the right.
To mitigate this, we’ve introduced “shared config types” packages in:�- k8s.io/apimachinery/pkg/apis/config/v1alpha1�- k8s.io/apiserver/pkgs/apis/config/v1alpha1
These packages host structs like `LeaderElectionConfiguration` and `ClientConnectionConfiguration`.�This shared set will grow over time.
apiVersion: kubeproxy.config.k8s.io/v1alpha1�kind: KubeProxyConfiguration�clientConnection:� burst: 10� contentType: application/vnd.kubernetes.protobuf� kubeconfig: /var/lib/kube-proxy/kubeconfig.conf� qps: 5 |
apiVersion: kubelet.config.k8s.io/v1beta1�kind: KubeletConfiguration�contentType: application/vnd.kubernetes.protobuf�kubeAPIBurst: 10�kubeAPIQPS: 5�kubeConfig: /etc/kubernetes/kubelet.conf |
Real-world example:
External and internal versions
Guide: Use ComponentConfig for your app
Let’s say you’re building an addon to Kubernetes, and want to it to have the same “look and feel” as the other k8s components. You don’t like to reinvent the wheel and reimplement all the API machinery features, but re-use the Kubernetes API machinery framework.
Here’s a walkthrough of the files to be written for it to work:
Guide: Directory structure
Here’s an example directory structure.�A couple of things to note:
github.com/luxas/sample-config�├── cmd�│ └── sample-config�│ └── main.go�└── pkg� └── apis� └── config ├── doc.go� ├── register.go ├── types.go� ├── scheme� │ └── scheme.go� ├── v1� │ ├── defaults.go │ ├── doc.go� │ ├── register.go� │ └── types.go� └── v1beta1 ├── conversion.go� ├── defaults.go ├── doc.go� ├── register.go� └── types.go |
Guide: What we’ll do
Guide: What we’ll do
v1.MyAppConfiguration{}
$ bin/sample-config --config [config-file]
If [config-file] is set:
If [config-file] is not set:
config.yaml (v1|v1beta1)
Read
Decode
Default
Convert
Default
Convert
Internal type for use in program
Convert to v1
Encode
stdout
Flags
Guide 1/7: The types.go file
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object�type MyAppConfiguration struct {� metav1.TypeMeta�� // ClientConnection configures the connection to Kubernetes� ClientConnection apimachineryconfig.ClientConnectionConfiguration� // LeaderElection configures so the component can be HA-deployed� LeaderElection apiserverconfig.LeaderElectionConfiguration� // Server holds configuration settings for the HTTPS server� Server ServerConfiguration�}�type ServerConfiguration struct {� // Default: "0.0.0.0"� // +optional� Address string� // Default: 10250� // +optional� Port uint32� // +optional� TLSCertFile string� // +optional� TLSPrivateKeyFile string�} |
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object�type MyAppConfiguration struct {� metav1.TypeMeta `json:",inline"`�� // ClientConnection configures the connection to Kubernetes� ClientConnection apimcfgv1.ClientConnectionConfiguration `json:"clientConnection"`� // LeaderElection configures so the component can be deployed in HA mode on k8s� LeaderElection apiscfgv1.LeaderElectionConfiguration `json:"leaderElection"`� // Default: "0.0.0.0" // +optional� ServerAddress string `json:"serverAddress"`� // Default: 10250� // +optional� HTTPSPort uint32 `json:"port"`� // TLSConfig holds settings for the TLS configuration� TLSConfig TLSConfig `json:"tlsConfig"`�}��type TLSConfig struct {� // +optional� TLSCertFile string `json:"tlsCertFile"`� // +optional� TLSPrivateKeyFile string `json:"tlsPrivateKeyFile"`�} |
External v1beta1 with json tags
Internal, has same schema as v1, but no json tags
Any serializable type must implement runtime.Object and embed TypeMeta!
Guide 2/7: The register.go file
Internal package: types & deepcopy
External package: types & deepcopy + defaulting & conversions
package v1beta1��const GroupName = "config.luxaslabs.com"�var (� SchemeBuilder = runtime.NewSchemeBuilder(� addKnownTypes,� addDefaultingFuncs,� )� localSchemeBuilder = &SchemeBuilder� AddToScheme = localSchemeBuilder.AddToScheme� // SchemeGroupVersion is'the group & version for this scheme� SchemeGroupVersion = schema.GroupVersion{� Group: GroupName,� Version: "v1beta1",� }�)��func addKnownTypes(scheme *runtime.Scheme) error {� scheme.AddKnownTypes(SchemeGroupVersion, &MyAppConfiguration{})� return nil�} |
package config��const GroupName = "config.luxaslabs.com"�var (� SchemeBuilder = runtime.NewSchemeBuilder(� addKnownTypes,� )� localSchemeBuilder = &SchemeBuilder� AddToScheme = localSchemeBuilder.AddToScheme� // SchemeGroupVersion is'the group & version for this scheme� SchemeGroupVersion = schema.GroupVersion{� Group: GroupName,� Version: runtime.APIVersionInternal,� }�)��// Adds the list of known types to the given scheme.�func addKnownTypes(scheme *runtime.Scheme) error {� scheme.AddKnownTypes(SchemeGroupVersion, &MyAppConfiguration{})� return nil�} |
Guide 3/7: The doc.go file
In doc.go, it is specified how the autogenerated code should be generated.
// +k8s:deepcopy-gen=package�// +k8s:conversion-gen=github.com/company/my-app/pkg/apis/config�// +k8s:conversion-gen=k8s.io/apimachinery/pkg/apis/config/v1alpha1�// +k8s:conversion-gen=k8s.io/apiserver/pkg/apis/config/v1alpha1�// +k8s:defaulter-gen=TypeMeta�package v1beta1 |
// +k8s:deepcopy-gen=package�package config |
External sample:
Internal sample:
Guide 4/7: The defaults.go file
This file must include the `addDefaultingFuncs(scheme)` function that in turn calls the auto-generated `RegisterDefaults(scheme)` function. The format of the defaulting function is `SetDefaults_{SampleStruct}`.�Defaulting fns from shared pkgs are optional and can be called via `RecommendedDefault{SampleStruct}`.
package v1beta1��func addDefaultingFuncs(scheme *runtime.Scheme) error {� return RegisterDefaults(scheme)�}��func SetDefaults_MyAppConfiguration(obj *MyAppConfiguration) {� if len(obj.ServerAddress) == 0 {� obj.ServerAddress = "0.0.0.0"� }� if obj.HTTPSPort == 0 {� obj.HTTPSPort = 9090� }�� apimachineryconfigv1.RecommendedDefaultClientConnectionConfiguration(&obj.ClientConnection)� apiserverconfigv1.RecommendedDefaultLeaderElectionConfiguration(&obj.LeaderElection)�} |
Guide 5/7: The conversions.go file
For straightforward conversions between structs, where the field name and type match, autogenerated code is created automatically. But when more complex changes are made to a new API, you must write the `Convert_{v1}_{SampleStruct}_To_{internal}_{SampleStruct}` function yourself.
Remember to first call the (partial) autogenerated portion of the conversion!
package v1beta1��func Convert_v1beta1_MyAppConfiguration_To_config_MyAppConfiguration(in *MyAppConfiguration, out *config.MyAppConfiguration, s conversion.Scope) error {� if err := autoConvert_v1beta1_MyAppConfiguration_To_config_MyAppConfiguration(in, out, s); err != nil {� return err� }� out.Server.Address = in.ServerAddress� out.Server.Port = in.HTTPSPort� out.Server.TLSCertFile = in.TLSConfig.TLSCertFile� out.Server.TLSPrivateKeyFile = in.TLSConfig.TLSPrivateKeyFile� return nil�} |
Guide 6/7: The scheme.go file
A scheme holds all registered types, defaulting and conversion functions centralized in one place. Use the `AddToScheme` functions to register a specific package.
In the scheme package, all known versions are aggregated and registered in the global Scheme variable.
Codecs provide encoding/decoding functions for the types in the scheme
package scheme��var (� // Scheme is the scheme to which all API types are registered.� Scheme = runtime.NewScheme()�� // Codecs provides access to encoding and decoding for the scheme.� Codecs = serializer.NewCodecFactory(Scheme)�)��func init() {� AddToScheme(Scheme)�}��// AddToScheme builds thescheme using all known versions of the API.�func AddToScheme(scheme *runtime.Scheme) {� utilruntime.Must(config.AddToScheme(Scheme))� utilruntime.Must(v1beta1.AddToScheme(Scheme))� utilruntime.Must(v1.AddToScheme(Scheme))� utilruntime.Must(scheme.SetVersionPriority(v1.SchemeGroupVersion))�} |
Guide 7/7: The hard work pays off!
Now, we can read any file with an external version directly into the internal version the program may use using `runtime.DecodeInto()`.
�The internal version can also be populated from defaults in an external version using Scheme.Default() & .Convert()
�Finally, it’s easy to marshal any object into whatever form using a generic encoder for a specific external version, and `runtime.Encode()`
// decodeFileInto reads a file and decodes the it into an internal type�func decodeFileInto(filePath string, obj runtime.Object) error {� content, err := ioutil.ReadFile(filePath)� if err != nil { return err }� // Regardless of if the bytes are of the v1 or v1beta1 version,� // it will be read successfully and converted into the internal version� return runtime.DecodeInto(scheme.Codecs.UniversalDecoder(), content, obj)�}�// populateV1Defaults populates cfg based on v1 defaults�func populateV1Defaults(cfg *config.MyAppConfiguration) error {� // Create a new config of some external version,� // default it, convert it into the internal version� v1cfg := &v1.MyAppConfiguration{}� scheme.Scheme.Default(v1cfg)� return scheme.Scheme.Convert(v1cfg, cfg, nil)�} // marshalYAML marshals any ComponentConfig object registered in the scheme for the specific version�func marshalYAML(obj runtime.Object, groupVersion schema.GroupVersion) ([]byte, error) {� // yamlEncoder is a generic-purpose encoder to YAML for this scheme� yamlEncoder := json.NewYAMLSerializer(json.DefaultMetaFactory, scheme.Scheme, scheme.Scheme)� // versionSpecificEncoder writes out YAML bytes for exactly this v1beta1 version� versionSpecificEncoder := scheme.Codecs.EncoderForVersion(yamlEncoder, groupVersion)� return runtime.Encode(versionSpecificEncoder, obj)�} |
Roadmap
Recap