3/31/2020:
Reusable kustomize modules
by
Alexey Odinokov
Agenda
Proposal
Understanding the problem
The invitation for discussion
UPD: proposal on filtering
Overview
Airship 2.0 moves forward with the documents that allow to deploy ephemeral node and k8s control plane, e.g. [1], and that means that the number of manifests will grow in the nearest future. So far manifests contain only resources - but the idea is to be able to override some parameters and to make the structure that allows one module or function to use another module.
As of now the main proposal for manifest modules can be found here [2]
It is still in progress and the main purpose of this document is to extend the proposal with demonstrating the possibility to reuse the code (transformers) of one module in another module and to show how this affects the manifests structure.
[1] https://review.opendev.org/#/q/status:open+project:airship/airshipctl+branch:master+topic:initinfra
[2] https://github.com/mattmceuen/replacement-transformer-example
Understanding the problem
In the example [1] each module uses its own VariableCatalogue (e.g. [2]), that is employed by transformer as a data source (e.g. [3]). It would work for the structure functions/site, but we plan to have at least 3-4 levels: functions/composite/type/site
That means that composite will have its own VariableCatalogue and part of its parameters should go to the lower level function’s VariableCatalogue. It would be great to have an ability to apply the changes in the function’s VariableCatalogue by calling the function’s transformer AFTER the composite overridden the function’s VariableCatalogue. See the picture on the next slide.
Understanding the problem
The output of Composite1(Cs1) will contain overridden Fn1-VariableCatalogue, but “Fn1-the rest of the data”
will be with values from the initial Fn1-VariableCatalogue. We want to call Fn1-transformers separately.
Desirable behavior 1
We want something like this. The drawback of this is that in this case we call Fn1 transformers 2 times. For large hierarchies each low level transformer will be called minimum N-1 times, where N is number of layers. There is a better option depicted on the next slide.
Desirable behavior 2
This is better, than the previous slide in the sense that we perform the controlled call of Fn1-tranformers only when we already updated Fn1-VariableCatalogue. Luckily Kustomize allows that configuration to be done, but the module will look a bit tricky. See the next slides for details
Proposal
Kustomize treats files inside directory where kustomize.yaml lays and outside of this directory in the different way. It’s not allowed to call resource/transformer from outside of the directory where kustomize.yaml lays.
BUT it’s possible to call the whole external directory that has its own kustomize.yaml.
If you call it from resources field - it will just add those resources. The interesting thing is that you can call it from generators and resources field as well. And in that case the output of that directory is treated as generator or resources configuration [1].
That allows us to put all Fn1-transformers into a separate folder and add all transformers as resources in its own kustomize yaml.
NOTE: in general it’s an interesting feature, because that means that we can modify transformer’s config with other transformers (potentially do it several times) and this adds more opportunities for us, but it adds more complexity. This feature isn’t studied in details. For now let’s keep it simple :)
Proposal
Similarly we put all static resources and generators (if applicable) into a separate directory to be able to call them from CS1-resources.
In that case the function directory structure will look like:
<function-1>:
Resources:
Kustomize.yaml
Static resources + generators
Transformers:
Kustomize.yaml
transformers
Kustomize doesn’t allow you to have kustomize.yaml in the FN-1 directory, but we can put kustomize.yaml with the following content to the internal directory along with Resources/Transformer directories:
resources:
- ../resources
transformers:
- ../transformers
Proposal
As an example I used ephemeral site from airshipctl/manifests (I took it from Dmitriy’s commit) and modified in order to make transformers work. I used jinjaPlugin instead of replacementTransformer, but in this example they’re interchangeable.
The link can be found here: https://github.com/aodinokov/kustomize-jinjaPlugin/blob/master/manifests4transormers/type/test-bootstrap/ephemeral/whole/kustomization.yaml
Please pay attention to the transformers directory of the ephemeral-site type:
It shows how to override lower function’s VariableCatalogue and call its transformers
Some notes
Note 1: The working transformer’s call sequence of functions/composites/etc can be:
Note 2: It’s NOT necessary to split resources and transformers for the top level components - e.g. the final site can avoid using this.
Some notes
Note 3: The more complicated scheme is possible, e.g. like on the picture. Function 2 is used by several components and their components are getting merged in Type 1. The conflicts within Function 2 is possible, but by explicit changing the order of transformers execution. Of course it’s better to avoid such things as much as possible.
Note 4: in general, we’re not limited by the structure “site/type/composite/function”. We can use one module in another and do more layers theoretically. But the mentioned struct is simple and I think that so far we should keep it unless we get some issue with it.
More notes
Note 6: it would be great to have a special transformer-validator, that would check the scheme for each VariableCatalogue in each component as a first transformer
Very generic note 7 :) : the module scheme reminds helm module structure, and in general Kustomize looks like a multilayered helm chart or set of charts:
UPD 6/24/2020: VariableCatalogue cleanup
apiVersion: v1
kind: Namespace
metadata:
name: foo
$patch: delete
UPD 6/24/2020: Proposal
Basically that means that we can’t cleanup unregistered types, but can cleanup registered, e.g.: v1 ConfigMap, apps/v1 Deployment &etc. But since kustomize doesn’t know anything about airshipit.org/v1alpha1 VariableCatalogue instead of deleting the resource kustomize adds $patch: delete to its body :) We can:
apiVersion: builtin
kind: PatchStrategicMergeTransformer
metadata:
name: notImportantHere-delete
patches: |-
# previous apiVersion: airshipit.org/v1alpha1 kind: VariableCatalogue
apiVersion: v1
kind: ConfigMap
metadata:
name: type-test-bootstrap-ephemeral-catalogue
$patch: delete
And call it after replacement transformers in manifests-example/site/test-workload/target/workload/kustomization.yaml or other places that includes catalogs to remove them after usage in replacement transformers.
UPD 7/25/2020: Use SMP to remove
Starting kustomize 3.8.1 some plugins got rid of apimashinery and now SMP works in the same way for all types of documents.
‘$patch: delete’ doesn’t need to know the resource scheme and works always. That allows us not to delegate the deletion on airshipctl, but do it immediatly in the model as it was proposed. SMP $patch: delete works for VariableCatalogue removal. That means we may avoid using airshipctl for the cleanup step - instead we may add the additional transformed that will remove catalogues.
Here is the link that shows that:
https://github.com/aodinokov/noctl-airship-poc/tree/master/packages/test_delete
Thank you!