K8S 2017 Planning:�Features and Roadmap

Aparna Sinha�Ihor Dvoretskyi

Kubernetes 2017 release timeline

Process for today - 25 mins

• Room is divided into ‘“Stations” - please choose which ones you want to develop roadmap for
• Each “Station” has a starter list of themes for 2017 - please read, discuss and add to it
• Fill out email template (from Ihor)

2017 Themes

• Scale the project (tablestakes)

​

• Reference architectures for application workflows

​

• Secure multi-tenancy with service catalog

​

• Production ready cluster lifecycle

​

• Multi-cloud support for AWS / Azure / on premises

​

​

Stations for 2017 planning

Cluster life cycle

Federation

UI

API Machinery

Apps

Auth

Autoscaling

AWS

Cluster Ops

Contributor Experience

Docs

Scalability

Instrumentation

Network

Node

OpenStack

Scheduling

Service Catalog

Storage

Testing

Windows

Bare Metal

Confidential & Proprietary

Google Cloud Platform

5

Starter Themes - Apps

Consumer Point of View

- should we converge template systems

- recommended approach to packaging?

- Docker user experience is easier due to opinionation

- template authors have to maintain various packages

​

CI/CD - Non Goal (each org can delegate to SDLC team), no feature needed in K8S

• Question - how about a more blessed Docker Image registry? Incubator/recommendation, maybe kubectl integration

​

Source to image, Source to deploy workflow;

2017 goal: Well documented example patterns, recommended workflows for how to get an app from dev to production - home-grown/DIY, OTS integrations, CI/CD, PaaS(es)/vertically integrated stacks

Mike Ahrens, Eric Tune

Starter Themes - Scalability & Federation

Scalability -

• More careful def of scalability, More production like, natural apps to benchmark a cluster
• Move to at least minimal HA config for scalability testing
• SLOs
• Using the CNCF cluster as a bare metal environment
• Debuggability and Introspection (project wide)
• (project wide not scaling) Opentracing tracepoints into the kube system? Help with debug
• Another trace could be logs in k8s components (controller, sched to kubelet), need way to package data...

Bob Wise

Starter Themes - Node, Auth, Network

• Network:
• internal L7.
• Hybrid application (in k8s and not)
• Hierarchical network policy (admins specify who can do what / not)
• Moving beyond filtering towards quality of service
• Node: feature discovery,
• P0 Container Runtime strategy - CRI, OCI, Rkt, etc.
• P1 GPU support, NUMA, CPU sets
• Node - Disk, image and log management, Checkpointing

Jordan Liggitt

Starter Themes - Node, Auth, Network

• Auth:
• P0: Management of secrets - encryption and delegation to Vault
• P0/P1: App to App auth, App to ext service auth.. Credentials for things outside k8s. spiffe.io proposal (Joe Beda)
• Network policy - enhance / replace to give control over egress, express IP/CIDR based constraints. Egress policy needed in a MT service environment to ensure you have consumer to producer traffic (needed for service catalog)
• Platform to participate in the auth (x is allowed to talk to y service)
• Metadata for pods - should we scope it down / gate so pods only get info that’s appropriate (IAM for pods)
• User experience - improve login
• User space load balancing - need better story…
• RBAC to beta

​

Jordan Liggitt

Starter Themes - AWS, OpenStack, Vsphere, BareMetal

1. Reference architectures
2. Conformance tests runnable across cloud providers in deterministic time
3. AWS set up reliable
4. Adding new cloud provider in K8S install should be (1 eng quarter)
5. Integrated solutions for (key mgmt, IAM) for multiple clouds
6. K8S understands concept of group of nodes - enabling spot fleet...
1. Reason using autoscaling groups is so we get QoS on AWS. Fastest way to get instances running even if you don’t scale them
7. Priority - AWS, Azure, OpenStack, Vsphere

David Aronchick

Cluster Lifecycle, Cluster Ops - “production quality lifecycle”

P0 Upgrades - Scheduled downtime

• Upgrade control plane in maintenance window, roll back
• Large clusters have many maintenance windows
• Self hosting integration with kubeadm; P1- upgrade stateful apps

P0 HA - how to address N-nodes in an HA cluster

• (e.g., w/o DNS distribute multiple IP addresses tracked by clients, or use a load balancer or other ideas) - implications on complexity of setup and reliability…

P1: Split KubeAdm into phases

• (skip a phase - bootstrap, TLS, cluster discovery, verified prereq - make independently available. Console output should be descriptive and self serve).
• Kops and other provisioning systems to use kubeadm toolbox.

P2: Conformance & Interop tests

• Provider DNS integration (on -premise… any infra for LB and DNS config)
• node level - depends on kubelet version. Cluster conformance test - make it easy

P0/P1: Docs - on ops, upgrades, manual install, explain what’s happening but give tools to automate. Improve process for ensuring docs are created early

Joe Beda, Piotr

Starter Themes - Autoscaling & Instrumentation

• Custom metrics for HPA
• Ensure all k8s components are monitored - all should have metrics end points, and tools to export
• HPA and cluster autoscaler should work well with federated clusters
• Simplify deployment of 3rd party monitoring
• HPA API adapter (allows custom metrics w/ 3rd party tools, dependency on Node)
• Guide on how to integrate 3rd party monitoring
• Intro Metrics server and deprecate Heapster
• Introduce Infrastore - database repo of infra info. Adhoc analytics / queries.

​

• Logging - collection from other sources

Piotr, Google

Service catalog

• Alpha, V1, MVP - Q1 2017
• Register broker
• Provision services
• Bind to them
• Openservice broker API - implement controllers
• K8S core: ease consumption of services / inject pods with credentials and config and remove automatic svc injection

​

Starter Themes - API Machinery, Contrib Exp.

Extensions

• ThirdPartyResource
• Admission-control hooks
• External cloudproviders
• Kubectl extensions

Cleanup

• Initializers and finalizers
• Move resource printers to the server

Clients

• Apiserver federation

• Policies: maintainers policy, contributor ladder, allowed licenses, etc.
• Onboarding process
• Github issues and stackoverflow questions under control
• Reduce average review time to under 1 day, and merge code the day it is approved
• Reduce build time to <5 minutes

Starter Themes - Apps & Service Catalog

• Application development
• CI/CD
• Developer experience
• PaaS (es)
• Workloads
• StatefulSet
• 50 charts for top apps
• Batch, streaming, workflow, cron
• Kubernetes is as well supported as Mesos and YARN for Spark, Kafka, Cassandra
• DaemonSet updates

Starter Themes - Scalability & Federation

Scalability

• 5000 nodes

Federation

• Complete and GA
• Upgradable
• HA
• Private services
• Use cases addressed include scaling, multi-zone, hybrid, at least one other cloud provider

​

Starter Themes - Node, Auth, Network

​

• Auth - identity minting, secrets, multi-tenancy, RBAC by default
• App-to-system auth[nz]�Multi-tenancy
• Security�- Root-in-cluster to cooperative multi-tenancy to secure multi-tenancy�- Certificate rotation�- Secret management via Vault
• Network -

Starter Themes - AWS, Bare Metal, OpenStack

Cloud provider strategy - cloud provider SIGs (AWS, OpenStack, BareMetal)

• Parity of Kubernetes features across cloud providers, conformance
• Reference architectures with rationale behind deployment choices, guidance
• Reference architectures?

Starter Themes - Cluster Lifecycle, Cluster Ops

Operator experience

• Top 5 Linux distros and top 3 cloud providers
• Upgrade support
• HA support
• Leak-free teardown
• Kubeadm GA
• Unification of kops and kubeadm
• Make “the hard way” easy (minimize kubeadm magic)
• Cluster directory deprecated

Known gaps to fix in 2017

• Scaling the project (codebase, community)
• Improve the docs - SIG docs
• Reference architecture