1 of 45

Megan O’Keefe and Shabir Abdul Samadh

Google Cloud

YAML Your Cloud!

Managing Cloud-Hosted Resources from Kubernetes

2 of 45

Hello!

Megan O’Keefe

Developer Relations, Google Cloud

@askmeegs

Shabir Abdul Samadh

Developer Relations, Google Cloud

@shabirmean

3 of 45

On Deck

What is the Kubernetes Resource Model (KRM)?

Why manage cloud-hosted resources from K8s?

How to manage cloud-hosted resources from K8s?

4 of 45

What is the

Kubernetes Resource Model (KRM)?

5 of 45

The Kubernetes API Server

API Server

etcd

cloud-

controller-

manager

kube-

scheduler

kube-

controller-

manager

cloud provider

Control Plane Node

kubelet

kube

proxy

container

runtime

Node

kubelet

kube

proxy

container

runtime

Node

kubelet

kube

proxy

container

runtime

Node

All of Kubernetes’ components centers on the API server.

It’s what the control plane uses as the source of truth - for external tools/clients and internal components

it’s how the Nodes know what containers they need to run,
it’s how any outside actor (you, a CI/CD system) interacts with the cluster

Exposes an HTTP API used by you (the user) and by the other control plane components

Through this we can query and manipulate any resource understood by the cluster

Examples of K8s API clients: kubectl, client-go, the Kubernetes Dashboard

—-------- Skip —-----------

kubelet - watches the API server for pods on that node and makes sure they are running
cAdvisor - collects metrics about pods running on that particular node
kube-proxy - watches the API server for pods/services changes in order to maintain the network up to date
container runtime - responsible for managing container images and running containers on that node

6 of 45

The Declarative Model

Declarative = API objects express intent

Controllers = loop to make your desired state match the actual running state

Source

observe

diff

act

7 of 45

The Kubernetes Resource Model

apiVersion: apps/v1

kind: Deployment

metadata:

name: nginx-deployment

namespace: my-app

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx:1.14.2

args: [“-port=8080”]

ports:

containerPort: 8080

……

status:

……

Metadata

Desired state

Observed state �(added/updated by the K8s system)

So now the next question is how do we tell the API server “what” you want (the desired state)�
We follow the Kubernetes Resource Model (KRM) to define the resource or object we want�
KRM is the data-model/scheme following which the intended state of Kubernetes resources are expressed�
Has three parts

Metadata explaining the object itself (name, namespace, labels etc.)
Desired state spec explaining what state you want the object in
Status field that maintains the current state of the object�

The status field is only updated and added to the definition server side. Client’s and users only specific metadata and desired state�
So what’s cool about this is that as long as we can define resources using this model and have controllers to manage them we can extend the API Server to support all different kinds of entities�

—-------- Skip —-----------

KRM Example: https://www.youtube.com/watch?v=kvdKvcfYrm8&list=PL510POnNVaaZx1QDIn_-77igM1COuHaHV&t=357s

State separation!!! (Desired Vs Observed)
MetaData

API Group and Version
Resource type
Scope (namespace)
Name (handle idempotency of specific unit)
K-V Labels (for user managed groupings and ease)
K-V annotation (for extensions via other systems/components)

Desired state:

Shows user intent
Attribute values

Default values
allocated values
Dynamic values

Observed state:

8 of 45

The Kubernetes Resource Model

The Kubernetes API can also be extended using Custom Resource Definitions (CRDs) with custom controllers.

Controllers that install + upgrade applications inside K8s

are called operators.

9 of 45

Why manage hosted resources with KRM?

10 of 45

Ways to interact with a cloud provider

Cloud Console

1st party command line

Client Libraries

3rd-party tools

eg. Terraform

kubectl

K8s client libraries

Cloud

APIs

Kubernetes API

With different cloud providers there are so many ways to interact with the cloud APIs�
And these tools also differ from cloud provider to cloud provider as well�
Now if you operate Kubernetes as well (be it cloud or self managed) you have an additional API to take care of�
And this also comes with it’s CLI tools and client libraries

Usually an organization running Kubernetes also needs to provision cloud resource - databases, serverless functionality, etc. �
This results in having to deal with different ways of interacting with your resources.

—-------- Skip —-----------

But if you operate Kubernetes, you then have the Kubernetes API to worry about. Often, an organization running Kubernetes also needs to provision cloud resources, such as databases, DNS, data warehousing. As a result, you might have two different ways of interacting with your resources. In some cases, this works out fine - for instance, you use Terraform to provision your Kubernetes clusters, then the kubernetes API to provision resources within that cluster.

11 of 45

K8s + hosted resources is hard.

Different APIs, different resource models + formats

Different paradigms (imperative vs. declarative, client-side vs. server side)

Different authentication (IAM / RBAC) systems

Add multiple cloud providers, and the complexity multiplies.

IAM: Identity and Access Management

RBAC: Role Based Access Control

12 of 45

What if...

Cloud

Console

1st party command line

Client

Libraries

3rd-party tools

eg. Terraform

kubectl

K8s client libraries

Kubernetes

API

Cloud

APIs

13 of 45

The Kubernetes Resource Model

Tightly coupled tool ecosystems

KRM ecosystem

Source: Kubernetes as a Framework for Control Planes by Brian Grant

Decoupled data model

14 of 45

Unifying resources with the K8s API

Git Repo

Cloud

Resources

API Server

In-Cluster Controllers

In-Cluster Resources

Cloud

Controllers

15 of 45

How to manage hosted resources with KRM?

16 of 45

YAML your cloud, in 4 steps!

Choose your tools.
Write YAML.
Figure out where to put the YAML.
Set up guardrails.

17 of 45

First-party cloud controllers

18 of 45

AWS Controllers for Kubernetes (ACK)

A set of open-source Kubernetes controllers to manage hosted AWS resources

Separate controllers per resource type (EC2, S3, etc.)

Can run inside AWS Elastic Kubernetes Service (EKS)

19 of 45

How ACK works (S3)

Install the ACK S3 controller to an EKS cluster with Helm
Give ACK permissions to create S3 resources
Write + apply an S3 bucket YAML
Deploy an app that uses that S3 bucket

20 of 45

EKS

Cymbal

Ads

S3

Bucket

ACK

21 of 45

Demo:

Managing AWS-hosted resources with AWS Controllers for Kubernetes

22 of 45

Google Cloud Config Connector

Like ACK, Config Connector is an in-cluster controller to manage cloud resources (this time, for Google Cloud).

Unlike ACK, Config Connector is a first-party product.

Unlike ACK, Config Connector operates as a single controller for many GCP resources.

23 of 45

How Config Connector Works

Install Config Connector on a GKE cluster
Give Config Connector permissions to create hosted GCP resources
Write Config Connector YAML to create hosted resources
Apply the YAML to the GKE cluster
Config Connector controllers lifecycle the hosted resources

24 of 45

How Config Connector Works

GKE

Config

Connector

25 of 45

How Config Connector Works

apiVersion: redis.cnrm.cloud.google.com/v1beta1

kind: RedisInstance

metadata:

name: redis-cart

namespace: config-connector-resources

spec:

displayName: "Cymbal Shops Redis Cart"

region: us-central1

redisVersion: "REDIS_5_0"

tier: BASIC

memorySizeGb: 1

26 of 45

How Config Connector Works

GKE

Config

Connector

Memorystore

27 of 45

Demo:

Managing Google-Cloud Hosted Resources with Config Connector

28 of 45

Setting guardrails for cloud KRM

“Only cluster admins can create Memorystore instances.”

➡️ Kubernetes RBAC

“Memorystore instances must use Redis v5.”

➡️ ??

29 of 45

OpenPolicyAgent Gatekeeper

A policy “gate” for a Kubernetes cluster.

Runs as an admission controller (before resources can get to the API server)

Can be configured with custom policies on specific resource types, both Kubernetes and custom (Cloud)

30 of 45

Benefits of OPA Gatekeeper

Automate policy enforcement for Cloud resources before the resources get created

Stay in compliance with industry- or geo-specific regulations

Policies defined as K8s CRDs, can be stored in Git + approved, audited

Accelerate developer velocity + feedback loop (can check policies during CI/CD, too!)

31 of 45

How to use Gatekeeper

Write a Constraint Template (abstract policy template)
Write a Constraint (concrete policy)
Apply those resources to the cluster
Gatekeeper will automatically enforce the policy for all incoming resources, blocking out-of-policy resources.

32 of 45

How to use Gatekeeper

GKE

Config

Connector

Memorystore

IAM

Gatekeeper

33 of 45

Demo:

Setting cloud guardrails with

OpenPolicyAgent

34 of 45

K8s workloads + Cloud workloads

GKE

Config

Connector

Memorystore

IAM

Cymbal

Shops

Gatekeeper

35 of 45

... Across multiple clouds...

GKE

Config

Connector

Memorystore

IAM

Cymbal

Shops

Gatekeeper

EKS

Cymbal

Ads

S3

Bucket

ACK

36 of 45

Are Config Connector and ACK enough?

Config Connector allows managing Google Cloud resources and ACK, enables same for Amazon Web Services resources

Still two very different tools to incorporate into the pipeline

Can we consolidate all this into one tool?

One ring popsicle to rule them all…

37 of 45

Crossplane

Open Source Kubernetes add-on; everything defined as KRM

Support for resource management on multiple platforms (GCP, AWS, Azure, Alibaba)

Assemble multiple resources into a composite unit and expose them as single resource definition

Set up policies and guardrails for cloud resources

Crossplane is an Opensource Kubernetes addon that can be simply installed into any Kubernetes cluster

Then you can activate the controllers for various cloud providers they support

With Crossplane installed in our cluster, this cluster becomes the only control plane for everything:

Our kubernetes resources
Cloud resources running in a primary provider
Cloud resources running in another cloud provider we use
Also resources in our on-Prem environments

A key functionality of Crossplane is also allowing you to assemble multiple cloud resources into a single KRM definition

For this talk we will show a basic deployed example of Crossplane which doesn’t use composition and will link to composition types in the resources at the end

Crossplane also has support for setting up policies on the resources that can be provisioned allowing you to do gatekeeping work

—-------- Skip —-----------

Resources can be provisioned and managed by kubectl, GitOps, or any tools that can talk with the Kubernetes API.

To facilitate reuse and sharing of these APIs, Crossplane supports packaging them in a standard OCI image and distributing via any compliant registry.

Developer is presented with the limited set of configuration that they need to tune for their use-case and is not exposed to any of the complexities of the low-level infrastructure below the API.

Access to these APIs is managed with Kubernetes-native RBAC, thus enabling the level of permissioning to be at that level of abstraction.

38 of 45

How Crossplane Works

Extends the K8s API by introducing more CRDs with their own controller

A CRD per cloud provider to maintain access configuration

Enables compositions of cloud resources into higher level abstractions

Source

It extends the Kubernetes API by introducing new Custom Resource Definitions with their own controllers

The configuration for the cloud provider is also its own CRD - we will see an example of it

The diagram at the bottom explains the concepts that create and manage the cloud resources

Resource claim is the KRM definition the user would submit
Resource class is the Kubernetes object that defines how to serve the claim
The resource is the representation of an instance of a cloud resource that serves the claim
You can also pool resources using a resource pool and us that to serve a claim

—-------- Skip —-----------

Enabling infrastructure owners to build custom platform abstractions (APIs) composed of granular resources that allow developer self-service and service catalog use cases
Providing a universal experience for managing infrastructure, resources, and abstractions consistently across all vendors and environments in a uniform way, called the Crossplane Resource Model (XRM)

A control plane that exposes a universal declarative-style API for cloud computing and offers full lifecycle management, orchestration, scheduling, and policy enforcement

Crossplane is a multicloud workload and resource orchestrator and manages workloads (container, serverless, others) and resources they consume (databases, message queues, buckets, data pipelines, and others) across a set of cloud providers or on-premise environments.

39 of 45

Demo:

Unifying multicloud resource management with Crossplane

40 of 45

Demo!

Memorystore

IAM

GKE

Cymbal

Shops

EKS

Cymbal

Ads

S3

Bucket

Crossplane

We will not go through setting it up. I am going to show the bits that actually created this setup

# list the helm deployment

helm list -n mean-crossplane

# list the crossplane specific resoruces

kubectl get all -n mean-crossplane

# have a look at the crossplane providers

kubectl get provider

# see inside the definition of the AWS provider

kubectl describe providerconfig.aws.crossplane.io/aws-config

# list the secrets used

kubectl get secrets -n mean-crossplane

# have a look at what these secrets are

kubectl get secrets/gcp-creds --template={{.data.creds}} -n mean-crossplane | base64 -D

# list the cloud resources created by crossplane

kubectl get bucket.s3.aws.crossplane.io/cymbal-ads-bucket

kubectl get cloudmemorystoreinstance.cache.gcp.crossplane.io/cymbal-memstore

# delete a cloudresource

kubectl delete cloudmemorystoreinstance.cache.gcp.crossplane.io/cymbal-memstore

41 of 45

Wrap-up!

There is are lots of ways to manage cloud infrastructure.

Kubernetes Resource Model (KRM) can act as the common data model to define all infrastructure in a declarative way

Different cloud providers have their own solutions to manage resources as a KRM entity

Efforts such as Crossplane enables a KRM based model with potential to abstract the underlying provider

42 of 45

Things we didn’t cover!

How does your KRM get from Git to your cluster(s)?

Transforming cloud KRM with kustomize / kpt.

RBAC and access control for cloud KRM.

Rego, the OPA Policy language

Other first-party tools, eg. Azure Service Operator

43 of 45

Resources

44 of 45

Try it out!

Slides: bit.ly/yaml-your-cloud

Demo: bit.ly/yaml-your-cloud-demo

45 of 45

Thank You!