1 of 17

2 of 17

Laura Lorenz�Google

Jeremy Olmsted-Thompson�Google

Paul Morie�Apple

SIG Multicluster Intro & Deep Dive

October 24-28, 2021

3 of 17

SIG Multicluster Intro & Deep Dive

Laura Lorenz (lauralorenz)

Paul Morie (pmorie)

Jeremy Olmsted-Thompson (jeremyot)

​

4 of 17

Welcome!

We’ll cover:

  • What this SIG is about
  • Current activity
    • Kubefed Deprecation
    • ClusterSet / Namespace Sameness
    • About API for storing cluster properties such as Cluster ID / ClusterSet membership
    • Multicluster Services API / Multicluster DNS
    • StatefulSetSlices for migrating stateful sets between clusters
  • Deep dive and Demo
    • About API and MCS API in action using AWS CloudMap MCS Controller for K8s
  • How to contribute

​

5 of 17

What this SIG is about

  • What should be the Kubernetes-native way to ….
    • Expose workloads from multiple clusters to each other
    • Share cluster metadata and its place relative to others
    • Generally break down the walls between clusters
  • Touches many different functional areas, but we are still working to identify the best, most durable primitives
  • We want AND need your input!
    • Real user stories and use cases are extremely valuable
    • Tell us what you’re working on!

6 of 17

Our approach

  • Avoid premature standardization
  • Focus on APIs
  • Avoid solving any optional problems
  • Keep multicluster consistent with single cluster
  • Work backwards from specific problems into something bigger, maybe

7 of 17

Current activity

8 of 17

Kubefed - on archival decision

  • Kubefed will be archived
  • Great learnings came from this project but is no longer under active development
  • Archival is not deletion
  • Thank you to everyone who contributed!

9 of 17

ClusterSet

  • ClusterSet represents a pattern of use from the field:
    • A group of clusters governed by a single authority
    • High degree of trust within the set
    • Namespace Sameness applies to clusters in the set
      • Permissions and characteristics are consistent across clusters for a given namespace
      • Namespaces don’t have to exist in every cluster, but behave the same across those in which they do
  • A cluster’s ClusterSet membership is stored in the about.k8s.io/ClusterProperty `clusterset.k8s.io`
    • See next slide …

​

10 of 17

About API: cluster metadata

  • KEP-2149
  • Now available at sigs.k8s.io/about-api
  • Cluster scoped ClusterProperty CRD - name: value
  • Uniquely identify clusters and identify their membership in a ClusterSet, for the lifetime of membership
  • Provides a reference for multi-cluster tooling to build on within a cluster set (e.g. valid DNS label)
  • Now a well-known place to store these or any other cluster properties that might otherwise be ad-hoc annotations on semantically adjacent objects

apiVersion: about.k8s.io/v1

kind: ClusterProperty

metadata:

name: cluster.clusterset.k8s.io

spec:

value: cluster-1

apiVersion: about.k8s.io/v1

kind: ClusterProperty

metadata:

name: clusterset.k8s.io

spec:

value: mycoolclusterset

apiVersion: about.k8s.io/v1

kind: ClusterProperty

metadata:

name: fingerprint.mycoolimplementation.com

spec:

value: '{"major": "1","minor": "18","gitVersion": "v1.18.2","gitCommit": "52c56ce7a8272c798dbc29846288d7cd9fbae032","gitTreeState": "clean","buildDate": "2020-04-30T20:19:45Z","goVersion": "go1.13.9","compiler": "gc","platform": "linux/amd64"}'

11 of 17

MC Services API

  • KEP-1645 and sigs.k8s.io/mcs-api
  • Services are a multi-cluster building block
  • Allows a single service to span and/or be consumed by multiple clusters
  • Focused only on the API and common behavior, leaving room for various implementations
    • Submariner, GKE, Istio, AWS
  • Consumers only ever rely on local data
  • ClusterIP and headless services just work as expected across clusters.

ClusterB

1

3

2

1

2

ClusterA

1

1

2

yellow.test.svc.clusterset.local

blue-1.clusterA.blue.test.svc.clusterset.local

blue-1.clusterB.blue.test.svc.clusterset.local

blue-2.clusterB.blue.test.svc.clusterset.local

blue-3.clusterB.blue.test.svc.clusterset.local

12 of 17

StatefulSetSlices

  • KEP-3335, proposing a mechanism to support the “split brain” of individual StatefulSet controllers during migration of a StatefulSet across clusters
  • Very cool POC leveraging both StatefulSetSlices and MCS
    • see @pwschuurman and @mattschallert showcasing this in their KubeconNA talk

app-0

app-1

app-2

StatefulSet Controller

  • replicas: 2
  • replicaStartOrdinal: 0

StatefulSet Controller

  • replicas: 1
  • replicaStartOrdinal: 2

13 of 17

Additional Topics

  • More sophistication on MC networking
    • Network policy - applying policy uniformly across clusters
    • Multi-network - stitching together clusters on different networks
  • Multicluster controllers / MC leader election
    • what SIG-MC should recommend or implement as a reference
  • Work API
    • Spreading groups of resources to different cluster

14 of 17

About API and MCS API in action with AWS CloudMap MCS Controller for K8s

Major shoutout to @runakash and @astaticvoid

15 of 17

Get involved

16 of 17

We need your input

Share your use cases, problems, and ideas

17 of 17

Session QR Codes will be sent via email before the event

Please scan the QR Code above to leave feedback on this session