1 of 18

KubeVela - The Modern App Delivery System in Alibaba

Mar. 2021, CM@WS Meeting 10

Cloud Native App Platform @AlibabaCloud

Last Updated: Oct 2021

2 of 18

What is KubeVela?

KubeVela

Render

Orchestrate

Deploy

Kubernetes Clusters

Clouds

IoT/Edge

CI Pipeline

Jenkins

...

CI

CD

An easy-to-use yet highly extensible app delivery system targets on today's hybrid, multi-cloud environments.

3 of 18

Introduction

  • The core component behind web-scale app platforms of Alibaba.
  • Focus on consistent & extensible app delivery workflow at scale.
  • The solution of how Alibaba is adopting GitOps and IaC.
    • Lesson learned: Do NOT do GitOps + IaC at a scale w/o KubeVela ^ ^.
  • It’s open source and a CNCF project:

4 of 18

Highlights

  • Application Centric
    • Consistent yet higher level abstraction to capture the full deployment of microservices on top of hybrid environments.
    • No infrastructure level concerns, simply deploy.
  • Fully Programmable
    • The abstractions, capabilities and even the deployment workflow itself in KubeVela are by design customizable/extensible via CUE templating language.
    • No restrictions, natively programmable.
  • Infrastructure Agnostic
    • KubeVela is an application delivery control plane, not a runtime plugin.
      • No invasion into user’s infrastructure.
      • Can deploy and manage any workload type on any infrastructure per your needs.
        • e.g. containers, cloud services, databases, or even virtual machine instances.

5 of 18

Architecture

User declares app deployment intents via Open Application Model (OAM).

KubeVela will then render OAM into a DAG of artifacts, operational behaviors and workflow steps, with leverage of CUE.

When this DAG is executed, the corresponding app delivery workflow will happen.

Open Application Model (OAM)

Component A

Component B

Step 1

Workflow

Policy 2

Component C

Kubernetes Clusters

Clouds

IoT/Edge

Trait 1

Trait 2

CI Pipeline

Jenkins

Argo

...

DAG

6 of 18

Use KubeVela

App Team

  1. Declare an application deployment plan in a single file.
  2. $ kubectl apply -f app.yaml
  3. Done!

Deployment plan is the ONLY concept app team needs to learn in KubeVela, and as a declarative API, it works seamlessly with any CI/CD or GItOps tools.

7 of 18

The Deployment Plan

Components to deploy

​

e.g. a Helm chart, a Kustomize pkg, a Cloud Formation template, a Terraform module … literally anything!

Workflow of the deployment

​

e.g. blue-green deploy, progressive traffic shifting, manual approval … any pipeline style delivery steps!

Traits for Day 2 operations

​

e.g. ingress/route rules, auto-scaling rules … operational behaviors attached to components!

The higher level abstraction to model a full application deployment on top of hybrid infrastructures.

Policies to enforce

​

e.g. security scopes, overrides, healthy check policies, firewall rules … any policies to enforce before deployment happen!

8 of 18

Programmable

In KubeVela internal, an application deployment process is expressed as a DAG of components, traits, policies and steps in CUE. Hence, the Application entity is simply a “UI” for this DAG, it will be rendered into CUE code and then executed by control plane Kubernetes.

This also makes it possible for you to customize KubeVela with CUE at any time to meet diverse requirements on features or abstractions from different users. We call this power as “the programmable platform”.

DAG

CI Pipeline

Jenkins

Control Plane Cluster

(As Execution Engine)

9 of 18

可编程式应用交付

在 KubeVela 内核中，任何应用交付需求最终都会被表达为一个基于 CUE 模板语言的有向无环图（DAG）。所以 Application 对象实际上只是这个 DAG 面向用户侧的一种 UI 而已：一旦提交，它就会被 KubeVela 渲染成上述 CUE 代码然后使用 Kubernetes 来执行这个 DAG。

这也使得 KubeVela 能够允许你通过 CUE 模板语言任意定制和修改整个系统的行为和抽象，以便让 KubeVela 暴露的功能满足不同用户的多样化需求。我们称这种强大的可扩展性为：可编程式应用交付平台。

DAG

CI Pipeline

Jenkins

Control Plane Cluster

(As Execution Engine)

10 of 18

X-Definitions: The Interfaces for Programming KubeVela

  • Extend/customize KubeVela with CUE
    • ComponentDefinition
      • Helm, Kustomize, Terraform, AWS CF, AlibabaCloud ROS ...
    • TraitDefinition
      • Canary, Autoscaler, Route ...
    • PolicyDefinition
      • Security Group, Healthy Group ...
    • WorkflowDefinition
      • Blue-green, Traffic Shifting ...
  • Why CUE?
    • can glue EVERYTHING into a DAG.
      • k8s, non-k8s, IaC modules, cloud API ...
    • highly efficient in data templating and passing.
    • Configuration as Data at scale.
    • Golang friendly.

11 of 18

Why KubeVela replies on a control-plane K8s?

  • IaC (i.e. CUE) is perfect in agility, reusability and extensibility for building app delivery workflow
  • But IaC also brings configuration drift, difficulty in maintaining, tracing and auditing
    • They can not be fixed by “GitOps”
    • They are nightmares in web-scale deployment ...

Application

​

my-app.yaml

X-Definitions

​

helm.cue

rds.cue

vpc.cue

container

State of infrastructure drift 2021

RDS instance

VPC instance

composition

Hence, we use a control plane K8s as the IaC Execution Engine to achieve:

  • Automation
  • Idempotence
  • Convergence
  • Determinism

They are all critical needs for large-scale app delivery.

Control Plane Cluster

(As Execution Engine)

12 of 18

So, does Alibaba adopt GitOps?

  • It’s not practical to fully rely on Git to drive the real-world delivery workflow.
  • Modeling everything (environments, clusters, deployment patches, pipeline steps, rollout strategies etc) fully in Git layout is challenging.
  • Web-scale makes above issues even worse.
    • What if I have 10 target environments and 3000+ apps?
  • However, KubeVela as the app delivery control plane helped us to achieve all benefits of GitOps with easy approach.

Source: The Fundamental Flaws of GitOps – A Statistical Analysis

13 of 18

Our pipeline

For given app, it now only needs 1 manifest to describe the deployment plan (even across environments), and KubeVela will take over the rest delivery workflow with full automation and deterministic.

CI Pipeline

Jenkins

Artifact Registry

Src Code Repo

KubeVela

Runtime K8s Cluster A

Cloud Services

Runtime K8s Cluster B

Control Plane CLuster

Config Repo

Auto or manually configure

Pull Artifacts

CI

CD

Application YAML

Subscription Channel

KubeVela will set up subscription channel for runtime environments to pull app metadata and sync.

GitOps

sync app.yaml

Push app.yaml

(legacy mode)

14 of 18

We endorses a team-centric workflow

  • Platform Team
    • maintaining components, traits, policy, workflow definitions as CUE templates
  • App Team
    • choosing templates, assembling them into an Application deployment plan!

Application

template

template

template

RDS

Pod

Nginx

Pod

$ vela up

15 of 18

More features yet to come ...

  • Monitoring and logging traits.
  • Nex-gen CLI and GUI console.
  • Integrations with existing CI/CD tools (Jenkins, Argo,Tekton ...).
  • Vertical use cases such as AI/Big Data/Edge app and workflow hosting.

16 of 18

Alibaba’s Application Infrastructure

Payment System

Cart System

Buy System

Serverless App Service

Unified App Infrastructure (KubeVela)

We

Runtime Cluster 1

Runtime Cluster 2

Other Clouds/Infrastructures

External Customers

Platform Team

internal groups

cloud product

~1M containers, ~10K deploys/day

17 of 18

Further Reading: Open Application Model - The Unified yet Extensible Abstraction for App Delivery

​

Traits

Scaling

Rollout

Traffic

Metrics

Route

​

Workload Definition

Database

Helm Chart

Job

StatefulSet

Deployment

Component Definition

Component Definition

Application

AWS RDS

Web Service

  • Web Service + Route
  • Database

Component Definition

Backend Worker

Application

  • Backend Worker

18 of 18

Thank You!

Further readings