1 of 37

ContainerWeek

April 2019

​

First presentation: Building Kubernetes Clusters, One Piece At A Time

Second presentation: Running Kubernetes Without Nodes

2 of 37

Hi!

  • I'm Jérôme (@jpetazzo on Twitter) 🇫🇷🇺🇸🇩🇪...🇧🇷?
  • Spent 7 years at Docker wearing various hats
  • Before:
    • various open source roles
    • co-founded Enix
  • After:
    • taking some time off
    • k8s and containers consulting and training

@jpetazzo

3 of 37

In-person training does not scale

@jpetazzo

4 of 37

Learning Kubernetes?

  • Free video available on https://container.training/!
  • Recording of a 1-day Kubernetes workshop (in English)
  • YouTube playlist (55 videos)
  • Direct link: VAIIIII

​

@jpetazzo

5 of 37

6 of 37

Two topics today!

​

​

¹Or until the cloud runs out of capacity. 🤷

​

@jpetazzo

7 of 37

Understanding Kubernetes�by building clusters one piece at a time

8 of 37

Why? (1/2)

  • Personal goal: pass the CKA (Certified Kubernetes Administrator) exam
  • The curriculum has a whole section about building/operating clusters
  • I know how to use Kubernetes
  • I kind of know Kubernetes components, but …
  • I don't know how to build a cluster from scratch
  • Solution: Kubernetes The Hard Way

@jpetazzo

9 of 37

Why? (2/2)

  • Kubernetes The Hard Way assumes that you're using GCP�(I wanted to play on a local cluster)
  • If you don't know what you're doing, it's hard to change anything�(I didn't know what I was doing)
  • It uses cfssl to manage the CA and certificates�(I would like to start without certificates)
  • Each component is started with many flags�(What do they all do? I don't know!)

@jpetazzo

10 of 37

How?

  • Build a cluster from scratch
  • Start each component with as few flags as possible
  • Remove anything that is not essential
  • And only then, re-add features one at a time

​

​

Next slide: diagram by Lucas Käldström

​

@jpetazzo

11 of 37

12 of 37

What do we need to do?

  • Download binaries for Docker, etcd, Kubernetes
  • Run etcd
  • Run API server
  • Test! (create NGINX deployment)
  • Does it work?�If it doesn't …�Find out what's missing, fix it, and try again!

@jpetazzo

13 of 37

Time for a live demo! (see here for details)

14 of 37

Takeaways

  • We created a Kubernetes cluster�(only one node; no security; many missing features; but IT WORKS!)
  • Tinkering with that cluster is easier than with other solutions�(full disclosure: this is my subjective opinion, yours may differ)

​

​

​

@jpetazzo

15 of 37

@jpetazzo

16 of 37

Questions?

17 of 37

Learning Kubernetes?

  • I also deliver in-person training!
    • April in Paris (🇫🇷🥖) and Chicago
    • May in Cleveland
    • June in San Jose
  • Can't travel? No problem!
    • Free video available on https://container.training/!
    • Recording of a 1-day Kubernetes workshop (in English)
  • Contact me if you want custom training for your team!

@jpetazzo

18 of 37

Running Kubernetes without nodes

19 of 37

Okay, so we've set up�our Kubernetes cluster.

​

Now what?

​

How many nodes�do we need?

20 of 37

Capacity planning is hard

  • If we provision too much�→ we're wasting money
  • If we provision too little�→ our pods won't run
  • If we use rolling updates, autoscaling, jobs�→ demand varies over time

�

@jpetazzo

21 of 37

Solution 1: think big

  • "Let's put all our apps in one giant cluster. Things will balance each other!"
  • Apps generally need to scale up and down at the same time (traffic patterns)
  • It is better to reduce the blast radius by deploying on smaller clusters�(see "A cascading failure of distributed systems", by Dan Woods)

​

@jpetazzo

22 of 37

Solution 2: auto-scale the cluster

  • Automatically add nodes when more capacity is needed
  • Scales up fairly well
  • Doesn't scale down
  • We have to manually drain and shutdown nodes
  • Note: GKE cluster autoscaler is pretty good�(but you need to learn about PodDisruptionBudgets etc.)

@jpetazzo

23 of 37

Solution 3: don't run nodes

🤔

@jpetazzo

24 of 37

Run containers without running servers

@jpetazzo

25 of 37

AWS Fargate / Azure Container Instances

  • Both are "container-as-a-service"
  • AWS Fargate is a backend for ECS�(not designed to be used with Kubernetes in the first place)
  • ACI allows various use-cases, including:�"elastic bursting with AKS" using Virtual Kubelet
  • Let's have a look at Virtual Kubelet!

​

​

​

@jpetazzo

26 of 37

Virtual Kubelet

  • Open source project (GitHub)
  • "Pretends" to be a node
  • Is actually a proxy for a container service provider
  • When a pod is scheduled to this node:�Virtual Kubelet provisions containers on the service provider

​

@jpetazzo

27 of 37

28 of 37

Virtual Kubelet pros and cons

  • Very active open source project
  • Not all providers support all features
  • Extensive support for Azure
  • However, for AWS …�Virtual-kubelet and the AWS Fargate virtual-kubelet provider are in very early stages of development. DO NOT run them in any Kubernetes production environment or connect to any Fargate production cluster.�😰

@jpetazzo

29 of 37

Elotl Kiyot

  • Kiyot (by Elotl, Inc.) implements the CRI (Container Runtime Interface)
  • For reference, other CRI implementations include:�Docker, CRI-O, containerd …
  • The CRI is a relatively low-level interface to run containers

@jpetazzo

30 of 37

How does Kiyot run pods?

  • Kiyot runs each pod in its own EC2 instance�(by talking to Elotl Milpa)
  • Instances are sized following the pod's resources limit�(from tX.nano with 512 MB, all the way up)
  • Pods use standard EC2 VPC networking�(no need for ENIs or custom CRI plugin)

�(See picture on next slide!)

��

​

@jpetazzo

31 of 37

32 of 37

Time for more live demo! (see here for details)

33 of 37

Conclusions

  • We use exactly the resources that we need (no more, no less)
  • We can scale out infinitely (i.e. as long as our cloud provider has capacity)
  • Pods are visible as regular instances in the AWS console
  • They even have proper tags to identify them easily
  • Starting a pod is a bit slower …�… but much faster than provisioning a new node

�If you want to try it, check Milpa Community Edition

@jpetazzo

34 of 37

One last thing...

  • Technology is cyclic�(example: terminals → personal computers → the web …)
  • One of the initial goals of orchestrators like Mesos was consolidation�(optimize costs and utilization by packing workloads together)
  • Today, it is better to "de-consolidate" because we have:
    • faster cloud APIs
    • per-minute instance billing
    • easy and fast deploy artifacts (container images)

@jpetazzo

35 of 37

Questions?

36 of 37

Would you like to know more?

​

​

@jpetazzo

37 of 37

Thank you!