1 of 23

Cluster �& Native Service Management

  • Michael Ma

linkedin.com/in/michael-kema

2 of 23

Cloud Computing

  • Commodity Hardware
    • Low Cost
    • Failure is NORM
  • Heterogenous Hardware
    • Support Various Workload
    • Compute and Storage Decoupled
  • Massive Scale
    • Resiliency
    • Cost Efficiency
  • Multi Geo-Location
    • Resiliency
    • Locality

​

​

​

​

​

Cloud computing[1] is the on-demand availability of computer system resources, especially data storage (cloud storage) and computing power, without direct active management by the user.

3 of 23

Data Center Topology

4 of 23

What is Machine?

A machine is a physical system using power to apply forces and control movement to perform an action. 

5 of 23

What is a cluster?

  • A Set of Nodes
    • Bare Metal
    • VM
  • Cluster Management System
    • Control Plane
    • Data Plane
  • Functionalities
    • Service Discovery(CP/DP)
    • Inventory Management(CP)
    • Allocation(CP)
    • Failure Detection(CP/DP)
    • Healing(CP)
    • Deployment(CP/DP)
    • Policy Management(CP)
    • Node Lifecycle Management(CP)
    • Workload Lifecycle Management(CP/DP)

A computer cluster is a set of computers that work together so that they can be viewed as a single system. Unlike grid computers, computer clusters have each node set to perform the same task, controlled and scheduled by software.

6 of 23

How is cluster management different?

  • Highly Available?
  • Scalable?
  • Fault Tolerance?
  • Distributed?
  • Entities to Manage?

7 of 23

What is native service?

  • Fault Tolerance
  • Multi-Tenancy
    • Container Based
      • Namespace Isolation
    • Declare Resource Demand
      • Resource Isolation

​

​

​

​

The page "Cloud native service" does not exist. You can create a draft and submit it for review, but consider checking the search results below to see whether the topic is already covered.

8 of 23

Cluster and Native Service Management System

  • Borg
    • Google’s Home Grown
    • Job and Service Management
  • Autopilot
    • Microsoft’s Home Grown
    • Service Management
  • Kubernetes
    • Invented by Google
    • OSS

9 of 23

Concepts

​

BORG

AUTOPILOT

KUBERNETES

Management Scope

Cell

Cluster

Cluster

Control Plane

BorgMaster

Autopilot

KubeMaster

Data Plane

Borglet

Client Services

Kubelet

Workload

Job

Machine Type

ReplicaSet

Workload Instance

Task(single container)

Machine

Pod(multi-container)

Node

Node

Physical Machine

Node

Tenant

N/A

Environment

Namespace

10 of 23

Interface

  • Job Description
    • Borg - BCL(declarative configuration language)
    • Autopilot
    • K8S - YAML

11 of 23

Workload Lifecycle

Borg

Kubernetes

​

  • Waiting
  • Running
  • Terminated

Autopilot

12 of 23

Availability Constraint

  • Borg
    • Number of Task Disruptions
  • Autopilot
    • Failing Limit
  • Kubernetes
    • Pod Disruption Budget

13 of 23

Priority and Quota

  • Priority
    • Borg
      • a positive integer – smaller -> lower priority
      • Bands: monitoring/production/batch/best-effort
      • No in-band preemption
    • Autopilot
    • K8S
  • Quota
    • Resource Reservation
    • Part of Admission Control

14 of 23

Service Discovery

  • Borg
    • Borg Name Service
    • Cell name/job name/task number
    • Host name/port in Chubby
  • Autopilot
    • DNS
    • Host File
    • Local Discovery File
  • K8S
    • Service Type
      • ClusterIP
      • NodePort
      • LoadBalancer
      • ExternalName
    • Discovering
      • Environment Variable
      • DNS
    • Headless Service
      • DNS

15 of 23

Workload Failure Detection

  • Borg
    • HTTP based Health Check URL
  • Autopilot
    • Watchdog
      • OK/Error/Warning
    • KV based Health Store
  • K8S
    • Liveness Probe

16 of 23

Node Failure Detection

  • Borg
    • Borglet Poll
  • Autopilot
    • Watchdog
  • K8S
    • Node Problem Detector
      • Run as DaemonSet
      • Monitor
        • System Logs
        • System Stats
        • Custom Plugin
        • Health Checker
    • Node Controller
      • Probe Node

​

​

17 of 23

Healing

  • Autopilot
    • History and Error Based
    • Actions
      • Reboot
      • Reimage
      • Replace

18 of 23

Monitoring

  • Borg
    • Infrastore
  • Autopilot
    • Collection Service
    • Cockpit
  • K8S

19 of 23

Architecture – Control Plane

  • Borg
    • Borgmaster
      • 5 Replicas
      • Paxos based Replication/Persistence
      • Leader Election
      • Communication with Borglet
    • Scheduler
      • Feasibility Checking
      • Scoring
      • Best Fit vs Worst Fit

20 of 23

Architecture – Control Plane

  • Autopilot
    • Device Manager
      • Strong Consistency
      • Replicated using Paxos
      • Pull vs Push
    • Satellite Services
      • Poll State from Device Manager
      • Deployment Service
      • Watchdog Service
      • Repair Service
      • Provisioning Service
      • Collection Service
      • Cockpit

​

21 of 23

Architecture – Control Plane

  • K8S
    • API Server
    • Cluster Management Service
      • Replication Controller
      • Node Controller
        • CIDR Assignment
        • Inventory Reconciliation
        • Node Health Monitoring
    • ETCD
      • Key – object path
      • Value – binary representation of the object

22 of 23

Architecture – Control Plane Scalability

  • Borg
    • Decouple Functionalities
      • Separate Scheduler based on Snapshot State
        • Workload Specific Scheduler
        • Score Caching
        • Equivalence Class
        • Random Selection
      • Borglet Probe
      • Read-only API
      • Sharded Across Replicas
  • Autopilot

23 of 23

Architecture – Data Plane

  • Borg
    • Borglet
      • Borgmaster Polls Borglet
      • Link Shard for Partition/Aggregation/Pruning
      • Cgroup based Performance Isolation
      • Port Shared
  • Autopilot
    • FileSync
    • Application Manager
    • Local Watchdog
  • K8S
    • Kubelet