Michael Irwin�Engineering Manager�Docker

Creating A Landlord for Multi-tenant K8s Using Flux, Gatekeeper, Helm, and Friends

17 May 2022 I Valencia, Spain

Introduction

• Engineering Manager at Docker
• Was a Docker Captain for ~5 years prior
• Worked at VT 2011-2022
• Lots of software development, cloud, and containers
• Proposed, architected, and led the creation of a “Common Application Platform”
• Find me @mikesir87

Outside of Tech…

Session Outline

1. Goals of the Effort
2. Multi-tenancy 101
3. Creating a Landlord
4. Wrap-up/Q&A

​

All source code is at:�github.com/mikesir87/gitopscon-eu-2022-demo

1. Goals of the Effort

The Common Application Platform

Initiative

Establish an organizational unit that provides application technology infrastructure and shared services for all application development teams.

​

Anticipated Outcomes

• Teams will be able to leverage a single common platform on which to run applications
• Shared application development and deployment services will be managed by a single team

In a Nutshell…

“You Build a Container, We’ll Run It!”

Separation of Concerns

• The platform owns everything “below” the app
• The app team owns everything related to the app itself

Cloud Infrastructure

Cluster Infrastructure

Node/Compute Resources

Application Definitions

Core Cluster Services

Application Services

Application Code

2. Multi-tenancy 101

Definition of a Tenant

• Really… it’s up to the individual application team!
• Don’t dictate how development/change management should operate
• Some examples…
• A tenant per application
• A tenant per environment (dev, staging, prod)
• A tenant per application per environment

What Could Possibly Go Wrong?

• A team makes changes to another team’s workload
• A team accesses the config/secrets of another team
• A team doesn’t specify CPU/memory constraints and consumes them all (noisy neighbor)
• A team defines an Ingress that intercepts another app’s traffic
• Teams create their own NodePort/LoadBalancer Services
• And so on…

The Tenancy Spectrum

Soft

Hard

Virtual Cluster

microVM per pod

Namespace per tenant

Node grouping/ isolation

Additional policy enforcement

Separate RBAC per tenant

Least Secure

Cheapest

Most Secure

Expensive

3. Creating a Landlord

https://unsplash.com/photos/s34TlUTPIf4

The Goals of the Landlord

• The landlord should be able to
• Define all of the tenants
• Customize tenant configuration per individual needs
• Be applied declaratively and idempotently
• Tenants can come and go
• Updates to one tenant should NOT affect any others
• Reapplying a single tenant’s config should cause no negative impacts
• Supports version control/history
• Not require custom programming/maintenance

Why Helm?

• The landlord should be able to
• Define all of the tenants
• Customize tenant configuration per individual needs
• Be applied declaratively and idempotently
• Tenants can come and go
• Updates to one tenant should NOT affect any others
• Reapplying a single tenant’s config should cause no negative impacts
• Supports version control/history
• Not require custom programming/maintenance

First Things First… Namespaces!

Virtual Cluster

microVM per pod

Namespace per tenant

Node grouping/ isolation

Additional policy enforcement

Separate Flux per tenant

Define Namespaces

• Every tenant needs a namespace to define their manifests

tenants:

team-awesome:

team-cats:

team-dogs:

values.sample.yaml

How do tenants deploy stuff?

The Manifest Repo

• With Flux!
• The Platform Team creates and owns a manifest repo
• Makes it easy for us to sync SSH keys needed to clone repos
• The app team is given maintainership of the repo
• Whatever manifests they put there will be deployed!
• They can control the process to update the manifests

What the Landlord Needs to Do

• For each tenant, we need to…
• Create a ServiceAccount in the tenant’s namespace for Flux to use
• Create a GitRepository that fetches the source material
• Create a Kustomization that applies the manifests

Define Flux Config

• Define a GitRepository, Kustomization, and optional Receiver
• Support basic configuration adjustments

tenants:

team-awesome:

flux:

path: ./manifests/...

team-cats:

flux:

path: ./manifests/...

team-dogs:

flux:

path: ./manifests/...

{{- range $name, $tenant := .Values.tenants }}

{{- $flux := $tenant.flux | default dict -}}

apiVersion: source.toolkit.fluxcd.io/v1beta2

kind: GitRepository

metadata:

name: {{ $name }}

namespace: flux-tenant-config

spec:

interval: 30m

url: github.com/mikesir87/{{ $name }}

secretRef:

name: flux-ssh-credentials

ref:

branch: main

---

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2

kind: Kustomization

...

values.sample.yaml

Locking Things Down a Little Bit…

Virtual Cluster

microVM per pod

Namespace per tenant

Node grouping/ isolation

Additional policy enforcement

Separate RBAC per tenant

Using Gatekeeper

• Don’t try to create your own policy engine!
• Gatekeeper will run all state change requests through the policy engine
• Make policies that match your needs!

Define Shared GK Policies

• After deploying Gatekeeper, apply the policies to your tenants
• Might be easiest to start with the policies that apply to all tenants
• Pod Security Standards (PSS)
• NodePort/LoadBalancer blocking

Define Parameterized GK Policies

• Apply policies that are parameterized per tenant

config:

tenantHostPrefix: tenants.local.mikesir87.training

​

tenants:

team-awesome:

flux:

path: ./manifests/team-awesome

team-cats:

domains:

- cats.local.mikesir87.training

team-dogs:

Breaking up workloads!

Virtual Cluster

microVM per pod

Namespace per tenant

Node grouping/ isolation

Additional policy enforcement

Separate RBAC per tenant

Defining Node Pools

• Kubernetes doesn’t have a native definition for node pool
• But, we can use a combo of taints/tolerations and node labels with affinity/selectors!
• Taints - prevent pods from being scheduled accidentally
• Tolerations - allow pods to be scheduled on a node with a taint
• NodeAffinity/NodeSelector - requires a node to have specific labels for pod scheduling

Node Pools Graphically!

…

Node

Pod

toleration:

- platform/node-pool=teamA:NoSchedule

nodeSelector:

platform/node-pool: teamA

toleration:

- platform/node-pool=teamB:NoSchedule

nodeSelector:

platform/node-pool: teamB

Defining Node Pools

• There are a variety of tools, but we recently moved to Karpenter!
• Support config-based pool management (perfect for a landlord)
• Tagging nodes can provide an idea of cost allocation too!

nodePools:

demos:

costCode: demos

tenants:

...

{{- range $name, $poolDetails := .Values.nodePools }}

apiVersion: karpenter.sh/v1alpha5

kind: Provisioner

metadata:

name: {{ $name }}

spec:

taints:

- key: platform/node-pool

value: {{ $name }}

effect: NoSchedule

...

Forcing Pods into Node Pools

• Using Gatekeeper mutations, we can mutate pods and assign pods to nodes without the tenant even knowing about it!

nodePools:

demos:

costCode: demos

​

tenants:

team-awesome:

flux:

path: ./manifests/team-awesome

nodePool: demos

{{- range $name, $tenant := .Values.tenants }}

{{- if $tenant.nodePool }}

apiVersion: mutations.gatekeeper.sh/v1beta1

kind: Assign

metadata:

name: {{ $name }}-nodepool-selector

namespace: gatekeeper-system

spec:

applyTo:

- groups: [""]

kinds: ["Pod"]

versions: ["v1"]

match:

scope: Namespaced

kinds:

- apiGroups: ["*"]

kinds: ["Pod"]

namespaces: ["{{ $name }}"]

location: "spec.nodeSelector"

parameters:

assign:

value:

platform/node-pool: {{ $tenant.nodePool }}

...

Other Things for the Landlord

• Add annotations for log forwarding
• Integrating RBAC with your IdP
• We used Dex and Jetstack’s kube-oidc-proxy
• Dex forwarded to core VT auth systems
• Group membership granted access to your tenant namespaces
• Can potentially grant different access for different needs

tenants:

team-awesome:

flux:

path: ./manifests/team-awesome

nodePool: demos

splunkIndex: awesome-team

rbac:

- groupName: awesome-team.devs

- groupName: awesome-team.ops

exec: true

One last thing!

Deploying the Landlord

• Just use a HelmRelease!
• If the tenants are the values, just provide them!
• Usage tip - use a ConfigMap to provide the values
• A GitRepository/Kustomization that converts the values to a ConfigMap
• By doing so, it allows you to use the values directly instead of having to extract it from the HelmRelease

Looping back… why?!?

• GitOps = tenants treat git repos as source of truth for apps
• Landlord = platform uses Helm/values to define tenant setup
• As tenants come and go, it’s simply updating the values

​

Thanks! Questions?