Key Technical Themes

for 2019

Brian Grant

SIG Architecture Co-Chair

https://goo.gl/8R66d5

Quality is Job 1

Crossing the chasm

​

https://medium.com/@shivayogiks/what-is-technology-adoption-life-cycle-and-chasm-e07084e7991f

Code and Release Quality

We need:

• Correctness: high test coverage and clear test signal
• No Dark Code
• Completeness: kubernetes/kubernetes/master always in a releasable state
• Predictability: patches only contain critical bug fixes
• Compatibility: APIs continue to behave the same way for existing clients
• Scalability: performance needs to monotonically improve
• Reliability: Kubernetes is rock solid
• Stability: critical APIs need to be GA/stable
• Observability: actionable information to operate, troubleshoot, and improve
• Usability: lower friction, more complete and up-to-date documentation

New SIG Arch Subproject: Code Organization

Long gone are the days when we could revert half-baked features before a release

To improve Kubernetes reliability, we need to control what goes into releases

Need a team to own the problem

• Evolve vendoring, staging, publishing
• Separate independent components into separate repos
• Feature branches for kubernetes/kubernetes and other monorepos
• Think of releases as a distribution, not just a build snapshot

​

Form follows function

Last Year: Feature Usage View

1. Container platform (aka container orchestrator)
2. Declarative API-centric control plane
3. Configuration distribution system
4. Container Infrastructure as a Service (CIaaS)
5. Platform for automating management
6. Services as a Platform (SaaP)
7. Portable cloud abstraction
8. Family of projects
9. Toolkit
10. Ecosystem

Last Year: System View

Governance Layer: Automation and Policy Enforcement (APIs optional and pluggable)

​

Application Layer: Deployment and Routing (APIs required and pluggable)

​

Nucleus: API and Execution (APIs required and not pluggable)

CronJob batch/v2alpha1

Job

batch/v1

Deployment apps/v1

DaemonSet apps/v1

Pod

core/v1

ReplicaSet apps/v1

StatefulSet apps/v1

ReplicationController core/v1

Endpoints

core/v1

Ingress

extensions/v1beta1

Service

core/v1

ConfigMap

core/v1

Secret core/v1

PersistentVolumeClaim

core/v1

StorageClass

storage/v1

ControllerRevision

apps/v1

Event core/v1

LimitRange

core/v1

ValidatingWebHookConfiguration

admissionregistration/v1alpha1

HorizontalPodAutoscaler

autoscaling/v1

APIService

apiregistration/v1beta1

PodDisruptionBudget

policy/v1beta1

PodPreset

settings/v1alpha1

PodSecurityPolicy

extensions/v1beta1

CertificateSigningRequest

certificates/v1beta1

ClusterRole

rbac/v1beta1

ClusterRoleBinding

rbac/v1beta1

LocalSubjectAccessReview

authorization/v1

Namespace

core/v1

Node

core/v1

PersistentVolume

core/v1

ResourceQuota

core/v1

Role

rbac/v1beta1

RoleBinding

rbac/v1beta1

SelfSubjectAccessReview

authorization/v1

ServiceAccount

core/v1

SubjectAccessReview

authorization/v1

NetworkPolicy networking/v1

ComponentStatus

core/v1

PriorityClass scheduling/v1alpha1

ClusterServiceBroker

servicecatalog/v1beta1

ClusterServiceClass

servicecatalog/v1beta1

ClusterServicePlan

servicecatalog/v1beta1

ServiceInstance

servicecatalog/v1beta1

ServiceBinding

servicecatalog/v1beta1

MutatingWebHookConfiguration

admissionregistration/v1alpha1

SelfSubjectRulesReview

authorization/v1

TokenReview

authentication/v1

CustomResourceDefinition

apiextensions/v1beta1

Emergent use cases

• Hybrid/multi-cloud platform
• Regional compute, multi-region service discovery and load balancing
• Kubernetes + service mesh
• Kubernetes + serverless/functions
• Control plane without the data plane
• Resource management platform / operational database
• Controllers for all the things
• Data plane (Kubelet) without the control plane
• Edge clusters (many, small, poorly connected)
• Installer for distributed applications

Emergent use cases

• Hybrid/multi-cloud platform
• Regional compute, multi-region service discovery and load balancing
• Kubernetes + service mesh
• Kubernetes + serverless/functions
• Control plane without the data plane
• Resource management platform / operational database
• Controllers for all the things
• Data plane (Kubelet) without the control plane
• Edge clusters (many, small, poorly connected)
• Installer for distributed applications

Use-case driven layering / factoring

Resource Management Platform

Application Management

Policy Enforcement

Service Mesh Data Plane

Automation

Container Runtime

Network Plugin

Volume Plugin

Execution Data Plane

Control Plane

Infrastructure

Use-case driven layering / factoring

Resource Management Platform

Application Management

Policy Enforcement

Service Mesh Data Plane

Automation

Container Runtime

Network Plugin

Volume Plugin

Execution Data Plane

Control Plane

Infrastructure

Kubernetes, the

Resource Management Platform

30,000 foot view of Kubernetes architecture

Resource Management Platform

100s of projects using the Kubernetes Resource Model:

• Operators/controllers: Applications, Databases, Services, Infrastructure
• Platforms: CI/CD, PaaS, FaaS
• Systems: service mesh, workflow, log management

Optional Authorization

​

Operational Primitives

​

Core Admission Policy

​

Optional Authentication

​

Core Resource Definition

Core Authorization

​

Endpoints

core/v1

ConfigMap

core/v1

Secret core/v1

Event core/v1

ValidatingWebHookConfiguration

admissionregistration/v1beta1

APIService

apiregistration/v1

ClusterRole

rbac/v1

ClusterRoleBinding

rbac/v1

Namespace

core/v1

ResourceQuota

core/v1

Role

rbac/v1

RoleBinding

rbac/v1

ServiceAccount

core/v1

MutatingWebHookConfiguration

admissionregistration/v1beta1

CustomResourceDefinition

apiextensions/v1beta1

Resource Management Platform Details

Lease

coordination.k8s.io/v1beta1

LocalSubjectAccessReview

authorization/v1

SelfSubjectAccessReview

authorization/v1

SubjectAccessReview

authorization/v1

SelfSubjectRulesReview

authorization/v1

TokenReview

authentication/v1

Also:

• Non-resource APIs
• discovery APIs, OpenAPI
• component status, healthz
• Authorization, authentication, audit hooks
• Controllers, such as namespace lifecycle and GC

​

Making the RMP more consistent, generic, dynamic

• Finish CRD
• Finish server-side “apply”
• Uniform completion/failure determination
• Improved support for consistency and sequencing
• Controller SDK(s)
• Rename Cluster everywhere
• Make ~all resources namespaced
• Make all hooks dynamic
• Dynamic CLI
• ...

Kubernetes, the

Policy Framework

Need clear, consistent direction for policy in K8s

• Policies impose permissions, quotas, constraints, requirements, defaults, etc. on other resources
• What patterns should we adopt going forward?
• Built in vs extensions
• Extension using DSLs vs APIs
• Domain-specific (scheduling policy) vs resource-specific (pod restriction)
• Conventions across policy types: whitelists, blacklists, profiles, defaults, etc.
• Cluster-level vs namespace-level
• Policies vs component flags
• How do we provide policy defaults?

​

​

​

Kubernetes, the

Service Mesh

Why Service Mesh

• Service meshes are evolving fast
• Service mesh provides
• Service discovery
• Load balancing and routing
• Observability
• Identity
• Policy enforcement
• Kubernetes does, too!
• Causes users confusion
• Its primitives also have a number of problems
• For example, multi-cluster load balancing is a common need
• Service was partly inspired by Google’s global load balancer, GSLB
• Apiserver permission escalation bug
• Time to rethink

Endpoints

• Endpoints [sic] is for endpoint tracking
• Peer discovery, including for the control plane
• Monitoring targets, including for component registration for the control plane
• Load-balancing targets, including for multi-cluster
• Authenticating proxy/port-forwarding targets, for debugging from outside the cluster
• Endpoints needs a reboot
• Sharding and cache friendliness for scalability
• Cruft / smells (e.g., tolerate unready endpoints)
• Non-pod endpoints
• Independence from Service
• Facilitate aggregation across clusters
• The equivalent of Endpoints (BNS) is where Borg set its boundary
• Borg writes them to another system (Chubby), with a scalable caching layer

Service and Ingress

• L4/L7 split is awkward
• Service has issues
• Mix of features and concepts (e.g., LoadBalancer)
• Subtle and obscure features, such as headless and imported services
• Not fully portable (LoadBalancer again)
• Also tied to (internal) DNS and service environment variables
• Has synchronous IP allocation logic in apiserver
• Not designed for aggregation across multiple Endpoints sets
• Ingress has even more issues
• Not included by default
• Not GA
• Not remotely portable
• Not universally adopted
• Also need an internal L7 load balancing solution -- “HTTP is the new TCP”
• Not integrated with external DNS, CDN, certificate management, or pod identity
• Has no status for consumers

​

Should Kubernetes have a Service Mesh?

• And, if so, what should it look like?
• Or should we reduce scope now that there are alternatives?
• Can we feasibly deprecate / replace major system functionality?
• And how would that impact K8s networking? application portability?

Infrastructure

Abstraction and Orchestration

Abstracting infrastructure

• Consistency and workload portability are key concerns
• Decouple concerns of application operators, cluster admins, and providers
• Example: PersistentVolumeClaim, PersistentVolume, StorageClass
• Application-driven dynamic provisioning and configuration
• Example: dynamic volume provisioning
• Provider-specific implementations should eventually be out of tree
• We need to handle topology consistently across the system
• Nodes
• Volumes
• Workload controllers
• Scheduler
• Services

Infrastructure orchestration

• API-based cluster managers
• SIG Cluster Lifecycle: Cluster API
• Ecosystem: OpenShift Cluster Operator, Gardener, Kubermatic, Giant Swarm
• Enabling users to not care about nodes
• Nodes as a cluster admin concern, especially in multi-tenant clusters
• Node auto-scaling and auto-provisioning
• Other CaaS platforms as backends
• Manage all the things
• External DNS
• Certificates
• Other IaaS resources

​