1 of 23

Creating k8s workshops with Traefik and GitOps

Lucas Käldström - CNCF Ambassador

28th of October, 2020 - Traefik Online Meetup

Image credit: @ashleymcnamara

2 of 23

$ whoami

Lucas Käldström, 2nd-year student at Aalto, 21 yo

​

CNCF Ambassador, Certified Kubernetes Administrator and Kubernetes WG/SIG Lead�

KubeCon Speaker in Berlin, Austin,

Copenhagen, Shanghai, Seattle & San Diego

KubeCon Keynote Speaker in Barcelona�

Former Kubernetes approver and subproject owner, active in the community for 4+ years.�Worked on e.g. SIG Cluster Lifecycle => kubeadm to GA.

�Weave Ignite author

Cloud Native Nordics co-founder & meetup organizer

​

3 of 23

Educators, hands up!

“Tell me and I forget,

teach me and I remember,

involve me and I learn.”�-- Chinese proverb

#CloudNativeNordics

4 of 23

MAP

4

5 of 23

IT’S NOT ONLY ME!

5

Foto: Torstein Lund Eik

@phennex & @kubernetesonarm

#CloudNativeNordics

6 of 23

Agenda

  • Running a workshop? How? Why?
    • Organize interactive workshops to “bridge the gap” to physical Meetups�
  • Technical requirements
    • Solution: workshopctl, a fully-OSS tool to automate all the things!
    • Using Traefik v2 for Load Balancing to save cloud $$$
    • GitOps Architecture
    • Live demo!�
  • Practicalities
    • Economics breakdown
    • How to manage this online? What about online meetups?
    • Do it yourself step-by-step checklist

6

7 of 23

Have you wanted to teach cloud native?

  • Done the “classical way”, in person, leads to
    • Broken/slow WiFi
    • Problems with virtualization
    • Enterprise laptop restrictions�
  • Virtually, this is even harder
    • How do you provide your remote attendants with a smooth environment without them having to spend the first hour on downloading minikube?�
    • How do you keep track of n different 95% similar, but still different environments?�
  • We can do better!

8 of 23

Enter, workshopctl, browser-based goodness

  • VS Code over HTTPS
  • Password-protected
  • Unique k8s cluster for each attendee
  • Running in a Pod :)

​

  • kubectl access
  • docker access
  • helm installed
  • k8s syntax highlight
  • Exercise READMEs
  • Solutions on-demand
  • Full Ingress support

User-specific domain

Pre-populated

tutorials!

Cluster shell with kubectl, docker, helm

K8s syntax highlighting

9 of 23

What? How?

  • Every attendee gets their own cloud-managed Kubernetes cluster
    • They have “root” access, if they break it, they don’t have an environment, but it doesn’t affect others�
  • Hosted easily at your preferred domain, e.g.
    • https://cluster-24.workshops.kubernetesfinland.com�
  • Visual Studio Code in the browser brings users an intuitive experience

10 of 23

Requirements for the solution

  • It should scale for 50-100 different full-blown k8s environments
    • By re-using 90%+ of config, it allows you to have different “cluster personas”
    • It should also be possible to use and customize off-the-shelf Helm charts�
  • It should be resilient, parallelized and automated
    • No more Bash loops running synchronously and failing b/c intermittent network�
  • It should be developer- and attendee-friendly
    • For the dev there should be a easy-to-use, declarative and deterministic CLI
    • For the attendee there should only be one link to click and one password to input

11 of 23

Problem solved!

  • workshopctl bundles together cloud native tech:
    • Traefik v2: Kubernetes Ingress Controller for speed and simplicity�
    • Flux v2 / GitOps Toolkit: GitOps all the things, make changes declaratively�
    • External-DNS: Sync desired DNS records from Ingress objects to the provider�
    • Code Server: The VS Code server-side rendering component�
    • Helm: I don’t fancy hand-writing 1000s of LoC of YAML, I patch pre-packaged bundles�
    • Kubernetes: ...of course

12 of 23

Architecture

Kubernetes Cluster XX in DigitalOcean

Kubernetes Cluster 02 in DigitalOcean

Kubernetes Cluster 01 in DigitalOcean

1xDroplet

$20-80/month

Droplet

DigitalOcean LoadBalancers

​

​

Traefik

Internet

Traefik

Namespace: default

nginx Ingress Rule

nginx Service

nginx Pod 1

nginx Pod 2

nginx Pod 3

nginx Deployment

Traefik Service

Type=LoadBalancer

$10/month

Public IPv4

DO Domains

​

Ingress Rules

traefik.@

@

podinfo.@

UI

<user-def>.@

Let’s Encrypt DNS-01 ACME

external-dns

Deployments/Pods

TXT _acme-challenge.traefik.@

TXT� _acme-challenge.@

TXT _acme-challenge.podinfo.@

code-server

TXT / A DNS records

podinfo

podinfo

podinfo

A �@

A�traefik.@

A�podinfo.@

13 of 23

Why Traefik?

  • Only need to pay for one Cloud LB per cluster
    • Cloud LBs aren’t cheap if you start using them for all your apps�
  • Automatic Let’s Encrypt integration
    • Using the DNS-01 verification scheme for better reliability in dynamic envs like this�
  • Kubernetes-native integrations with Ingress
    • “Plug and play”-experience, reacts dynamically to desired changes�
  • Nice dashboard for getting a holistic view

14 of 23

Traefik UI, API and metrics available!

Just visit traefik.cluster-XX.{my-domain.com} and input password!

15 of 23

workshopctl workflow

  • Standardized, user-friendly and reproducible way of running workshops
  • workshopctl init --clusters 40 --provider digitalocean [...]
    • workshop.yaml created, with customizable and declarative config
  • workshopctl gen
    • Manifests generated => git push
  • workshopctl apply
    • 40 clusters created using DigitalOcean’s managed k8s
    • All environments set up with VS Code accessible on a public domain
  • workshopctl cleanup

15

@luxas / @kubernetesonarm

#CloudNativeNordics

16 of 23

workshopctl directory layout

  • clusters/
    • XX/
      • <chart-1>.yaml
      • <chart-2>.yaml
  • charts/ (optional)
    • chart-1/
      • values{,-override}.yaml
      • templates/
    • chart-2/
  • workshopctl.yaml

@luxas / @kubernetesonarm

#CloudNativeNordics

17 of 23

workshopctl architecture

  • workshopctl gen
    1. For each chart, built-in (core-workshop-infra and podinfo) and in the local directory:
    2. Read external-chart (if exists), and download chart using Helm if so
    3. Read values-override.yaml (if exists), and apply template variables (go templates)
    4. Run built-in and (in the future) external processors functioning like an Unix pipe
    5. Pipe processed values to helm template => processed manifests out
    6. Run built-in and (in the future) external processors functioning like an Unix pipe
    7. Save to clusters/01/<chart>.yaml
    8. Do a git push

@luxas / @kubernetesonarm

#CloudNativeNordics

18 of 23

workshopctl architecture

  • workshopctl apply
    1. For each cluster, create a parallel goroutine
    2. Provision the managed k8s environment in the cloud provider
    3. Setup GitOps Toolkit to sync the given repo
    4. Apply a Secret with cluster-specific secret data
    5. Apply the core-workshop-infra chart to get started asap
    6. Wait for everything to come up

@luxas / @kubernetesonarm

#CloudNativeNordics

19 of 23

What I have learned

  • It’s always DNS, sadly.
    • Propagation times can be REALLY slow at times, and that’s breaking the rest of the system
  • For the above reason, DNS-01 ACME challenges seem more reliable
    • HTTP-01 suffer worse from long DNS propagation issues
  • You might have to increase your quota
    • Contact your cloud provider well in advance and say you need to run XX VMs & networks
  • Sometimes cloud providers have a bad day, have a backup plan

@luxas / @kubernetesonarm

#CloudNativeNordics

20 of 23

Economics Breakdown

Say theoretically you want to organize a�workshop for 40 persons for 4 hours.�

You should at least expect the clusters to be�running for 6 hours in the best case (need to provision a bit before due to DNS).��$40/month for 1 Droplet => $0.060/hr * 6 hours * 40 persons = $14.4�$10/month for 1 LB => $0.015/hr * 6 hours * 40 persons = $3.6

Total: $18 per workshop (45c/person)

​

Traefik makes you need only one LB.�DigitalOcean doesn’t charge for the Kubernetes control plane.

DigitalOcean doesn’t charge for managing the domain records.

vCPUs

RAM

Dedicated

Price/month

2

4

No

$20

4

8

No

$40

2

4

Yes

$40

4

8

Yes

$80

21 of 23

How bring this to an online audience?*

  • Schedule Meetup/Eventbrite event
    • E.g. like this Eventbrite event
    • You might want to collect email addresses, and send out personal passwords/links�
  • Set up Slack, Discord, Teams, Zoom workspaces
    • Make sure you bring at least 1 assistant per 10 persons, maybe more.
    • Use breakout rooms or similar for helping people�
  • Create curriculum
  • Run workshopctl :)

*Disclaimer: I haven’t actually run workshops of this exact sort purely online yet.

22 of 23

Questions?

If you want an example run-through of this workshop, you can find the recording here.

23 of 23

Thank you!

@luxas on Github

@kubernetesonarm on Twitter

lucas@luxaslabs.com