1 of 42

What does “production ready” really mean for a Kubernetes cluster?

Lucas Käldström - CNCF Ambassador

7th of May, 2019 - Umeå

Image credit: @ashleymcnamara

1

2 of 42

$ whoami

Lucas Käldström, High School Student, 19 years old�

CNCF Ambassador, Certified Kubernetes Administrator and Kubernetes SIG Lead�

KubeCon Speaker in Berlin, Austin,

Copenhagen, Shanghai & Seattle�

Kubernetes Approver and Subproject Owner, active in the community for ~3 years. Got kubeadm to GA.�

Driving luxas labs which currently performs contracting for Weaveworks�

A guy that has never attended a computing class

2

3 of 42

Agenda

Define the buzzwords!

What does “production-ready” mean to you?
What are the requirements for a highly available cluster?

What to think about when securing the cluster

TLS certificates for all components
Enable and set up RBAC (Role Based Access Control)
Attack vectors you might not have thought about before

3

4 of 42

Agenda

Make the cluster highly-available if needed

Do you need it?
How to set up a HA cluster with kubeadm
“Attack vectors” you might not have thought about before

Use the Cluster API for controlling the cluster declaratively

Intro to the Cluster API
How to set up Kubernetes using the Cluster API and upgrade/rollback

4

5 of 42

Agenda

Essential Kubernetes Addons

Container Runtime & Registry
Monitoring the cluster
Centralized Logging & Audit Logging
Out-of-tree Cloud Providers
Ingress Controllers
Persistent Storage with CSI

5

6 of 42

Which layer are you talking about?

Master A

Master N

Node 1

Node N

Kubernetes cluster

Machines

Application A

Application B

App C

App D

App E

Applications

Focusing on�this layer

6

7 of 42

Define what “production-ready” �means to you

Buzzwords all around...

7

8 of 42

“The cluster is production ready�when it is in a good enough shape �for the user to serve real-world traffic”

8

9 of 42

“Your offering is production ready when it�slightly exceeds your customer’s expectations�in a way that allows for business growth”

-- Carter Morgan, Google (@_askcarter)

9

10 of 42

It’s all about tradeoffs (!!)

10

11 of 42

Okay, so what does this mean�in terms of technical work items?

11

12 of 42

Production-ready cluster?

The cluster is reasonably secure�
The cluster components are highly available enough for the user’s needs�
All elements in the cluster are declaratively controlled�
Changes to the cluster state can be safely applied (upgrades/rollbacks)�
The cluster passes as many end-to-end tests as possible

12

13 of 42

Kubernetes’ high-level component architecture

Nodes

Master

Node 3

OS

Container

Runtime

Kubelet

Networking

Node 2

OS

Container

Runtime

Kubelet

Networking

Node 1

OS

Container

Runtime

Kubelet

Networking

API Server (REST API)

Controller Manager

(Controller Loops)

Scheduler

(Bind Pod to Node)

etcd (key-value DB, SSOT)

User

Legend:

13

14 of 42

What about “high availability”?

Instances (>=1) of a component can fail without causing the cluster to fail
Machines (>=1) in the cluster can fail without causing the cluster to fail

More about this in section III.

14

15 of 42

Securing Kubernetes

Things to keep in mind

15

16 of 42

TLS-secured communication everywhere!

Use mutual TLS for all communication
Certificates/identities should be rotatable
Use a separate CA for etcd
Use the Certificates/CSR API, with an external key signer if possible
Encrypt Secrets stored in etcd

16

17 of 42

API Authentication and Authorization

Disable ABAC, Anonymous Authentication and Insecure HTTP access
Enforce the RBAC and Node Authorizers
It’s recommended to delegate user authentication to a 3rd-party service
Enable Advanced Audit Logging

17

18 of 42

Lock down the kubelets in the cluster

Each kubelet should have:

unique client credentials
a serving cert signed by the cluster CA

Disable the readonly port (10255) & public (!) cAdvisor port (4194)
Enforce authn & authz for the main kubelet port (10250)
Enable automatic certificate rotation for the kubelets

18

19 of 42

Be careful with the Dashboard and Helm 2

Don’t give them (or any app!) cluster-admin power; very easy to escalate privileges
The security of the dashboard has improved since v1.7.0

The dashboard now has a login screen and delegates privileges

Specify the exact operations tiller may perform with RBAC
Secure the Helm <-> Tiller communication with TLS certificates

19

20 of 42

Deny by default -- best security practices

Deny-all with RBAC
Deny-all with NetworkPolicy
Set up a restrictive PodSecurityPolicy as the default

20

21 of 42

Setting up a dynamic TLS-secured cluster

Nodes

Control Plane

API Server

Controller Manager

Scheduler

CN=system:kube-controller-manager

CN=system:kube-scheduler

Kubelet: node-1

HTTPS (6443)

Kubelet client

O=system:masters

Self-signed HTTPS (10250)

CN=system:node:node-1

O=system:nodes

Kubelet: node-2 (to be joined)

Self-signed HTTPS (10250)

Bootstrap Token & trusted CA

CN=system:node:node-2

O=system:nodes

CSR Approver

CSR Signer

Legend:

Logs / Exec calls

Normal HTTPS

POST CSR

SAR Webhook

PATCH CSR

node-1 CSR

node-2 CSR

Bootstrap Token

CSR=Certificate Signing Request, SAR=Subject Access Review

21

22 of 42

More information about Kubernetes security

Try out Aqua Security’s kube-bench project
Official docs: Best Practices for Securing a Kubernetes Cluster
Hacking and Hardening Kubernetes Clusters by Example [I] - Brad Geesaman
11 Ways (Not) to Get Hacked on the Kubernetes blog

22

23 of 42

Minimize the points of failure �in the cluster

Proactively avoid disasters

23

24 of 42

kubeadm

Master 1

Master N

Node 1

Node N

kubeadm

Cloud Provider

Load Balancers

Monitoring

Logging

Cluster API Spec

Cluster API

Cluster API Implementation

Addons

Kubernetes API

Bootstrapping

Machines

Infrastructure

= The official tool to bootstrap a minimum viable, best-practice Kubernetes cluster

Layer 2

kubeadm

Layer 3

Addon Operators

Layer 1

Cluster API

24

25 of 42

How achieve HA with kubeadm?

HA etcd cluster

External Load Balancer or DNS-based API server resolving

Master A (kubeadm init)

API Server

Controller Manager

Scheduler

Shared certificates

etcd

Master B (kubeadm init)

API Server

Controller Manager

Scheduler

Shared certificates

Master C (kubeadm init)

API Server

Controller Manager

Scheduler

Shared certificates

Nodes (kubeadm join)

Kubelet 1

Kubelet 2

Kubelet 3

Kubelet 4

Kubelet 5

Do-it-yourself

Set up HA etcd cluster
Copy certificates from master A to B and C
Set up a loadbalancer�in front of the API servers

25

26 of 42

Is this cluster setup highly-available?

No

HA etcd cluster

Master A

API Server

Controller Manager

Scheduler

Shared certificates

etcd

Master B

API Server

Controller Manager

Scheduler

Shared certificates

Master C

API Server

Controller Manager

Scheduler

Shared certificates

Nodes

Kubelet 1

Kubelet 2

Kubelet 3

Kubelet 4

Kubelet 5

Master D

Loadbalancer

Single point of failure :(

26

27 of 42

Other things to keep in mind with a HA cluster

Remember to keep the CoreDNS replicas >= 1, and use Pod anti-affinity�
Some certificates need to be identical across control plane nodes

e.g. the ServiceAccount signing private key for the controller-manager
=> Needs to be rotated for all instances at the same time�

Monitoring the cluster components becomes increasingly more important with a HA cluster that is expected to have a high SLO

You can for example use Prometheus and kube-state-metrics as a starting point�

Do you need a HA cluster?

Is it worth the added cost and complexity?

27

28 of 42

“Monitor it so you know when it fails�before your customers do”

-- Justin Santa Barbara, Google (@justinsb)

28

29 of 42

Declarative cluster control �with the Cluster API

Manage clusters like applications

29

30 of 42

Cluster API

The What and the Why of Cluster API

“To make the management of (X) clusters across (Y) providers simple, secure, and configurable.”�
“How can I manage any number of clusters in a similar fashion to how I manage deployments in Kubernetes?”
“How do I manage other lifecycle events across that infrastructure (upgrades, deletions, etc.)?”
“How can we control all of this via a consistent API across providers?”

30

31 of 42

“GitOps” for your cluster(s)

With Kubernetes we manage our applications declaratively

Why not for the cluster itself?

With the Cluster API, we can declaratively define the desired cluster state

Operator implementations reconcile the state
Use Spec & Status like the rest of k8s
Common management solutions for e.g. upgrades, autoscaling and repair
Allows for “GitOps” workflows

apiVersion: cluster.k8s.io/v1alpha1�kind: MachineDeployment�metadata:� name: my-nodes�spec:� replicas: 3� selector:� matchLabels:� foo: bar� template:� metadata:� labels:� foo: bar� spec:� providerConfig:� value:� apiVersion: "baremetalconfig/v1alpha1"� kind: "BareMetalProviderConfig"� zone: "us-central1-f"� machineType: "n1-standard-1"� image: "ubuntu-1604-lts"� versions:� kubelet: 1.14.2� containerRuntime:� name: containerd� version: 1.2.0

31

32 of 42

Essential Kubernetes Addons

For enhanced insight and functionality

32

33 of 42

Cloud Native Trail Map

Trail Map: l.cncf.io

33

34 of 42

Choose your runtime & registry

Docker is the most common runtime, but you could consider using containerd (Graduated) or cri-o (Incubating) instead for less footprint and attack area.

Also, an internal container image registry might be needed. Harbor can set up a scalable registry for you on Kubernetes.

34

35 of 42

Monitoring the cluster

Now that the cluster is up and running, let’s start monitoring it. As a good starting point, you can use the prometheus-operator Helm Chart.

That gives you a Prometheus instance running in Kubernetes, good preset rules for monitoring (kube-state-metrics), and Grafana dashboards for visualization.

35

36 of 42

Enable Fluent Bit for logging

In order to store container logs for a long period of time, you need to enable a log forwarder from the container runtime to some kind of logging aggregation service like ElasticSearch.

You can use the fluent-bit-kubernetes-logging project as a good starting point for this task. Bonus points for also aggregating the Audit Logs

36

37 of 42

Enable cloud/environment extensions

What’s traditionally called Cloud Providers for Kubernetes; handles Node creation/deletion with the environment, and Type=LoadBalancer Services, and optional other features.

Anyone can create a so-called Cloud Provider integration for their environment. Example to the right.

37

38 of 42

Set up an Ingress controller

In order to expose your Services to the outer world, you need some kind of 3rd-party Ingress Controller.

Ingress Controllers makes your Ingress objects in Kubernetes work. You might want the controller itself to be a Type=LoadBalancer Service.

The ones you could look out for are Traefik, Nginx Ingress, and Contour.

38

39 of 42

Persistent Storage is key

Lastly, you most likely need Persistent Storage for many of your applications. Kubernetes supports the Container Storage Interface (CSI) for providers to implement.

Rook implements various types of clustered storage in a Kubernetes-native way. Alternatively, you can use your cloud provider’s solution.

39

40 of 42

Recap

Identify the needs of your business

How much money and effort do you want to put into HA & security?

High Availability != multiple masters

Multiple masters are a requirement for high availability

Pay attention to the certificate identities for your components

And make sure you lock things down well with RBAC, disable unnecessary ports, etc.

Declarative control over your cluster is better than imperative

The Cluster API and the GitOps models are worth checking out

40

41 of 42

Thank you!

@luxas on Github

@luxas on Kubernetes’ Slack

@kubernetesonarm on Twitter

lucas@luxaslabs.com

41

42 of 42

Related resources (in no particular order)

42