Quark Hibernate Container

Yulin Sun, Shaobao Feng

Serverless Container Requirement/Challenge

1. Low latency On-demand user request handling:
1. Cold startup
2. Warm startup
2. High density deployment
• Low memory
• CPU usage
3. Challenge:
• Memory: Warm-up container consume same memory as running service
• CPU: Warm-up container might consume CPU when there is no user request

Hibernate Container

• Start hibernate after application finished initialization
• Application memory is swapped out to disk
• Startup
• Triggered by user request (http) or control plan request
• Load memory from disk on-demand by page fault
• Resource consumption
• Memory: only sandbox memory
• CPU: No CPU
• Disk: Application memory swap file
• Startup latency: Low
• ​

Demo/Performance

• Resident Set Size (RSS)
• Private Mem
• Shared Mem
• Memory usage with N instances

N x Private Mem + 1 x Shared Mem

​

Node.js deployment density limit

Machine Spec

• CPU: 100 CPU core
• Memory: 300 GB

Application

• Memory: 128 MB
• CPU: 0.5 core

​

Running Container

• Count : 100 / 0.5 = 200
• Memory: 200 * 0.2 GB = 40 GB

Maximum Hibernate Container

• Count: (300 - 40) * 1000 / 5 ~= 52K containers

​

Next step work

1. IO improvement
1. Batch pages loading
2. NVMe Direct IO
2. Checkpoint / Delta pages set hibernate
3. WASM Node.js

Goal: Not replacement of warm startup but another option

M x Warm startup + N Hibernate

​

A New Serverless Application Deploy Mode

kubectl create -f app.yaml

A Knative AutoScale Example

apiVersion: serving.knative.dev/v1

kind: Service

metadata:

name: autoscale-go

namespace: default

spec:

template:

metadata:

annotations:

autoscaling.knative.dev/class: kpa.autoscaling.knative.dev

autoscaling.knative.dev/metric: concurrency

autoscaling.knative.dev/target: "10"

autoscaling.knative.dev/min-scale: "0"

autoscaling.knative.dev/max-scale: "100"

autoscaling.knative.dev/max-hibernete: "18"

autoscaling.knative.dev/min-hibernete: "3"

spec:

containers:

- image: gcr.io/knative-samples/autoscale-go:0.1

Manual Hibernating of pods

• kubectl hibernate pod-xxxx
• kubectl wakeup pod-xxxx

The Challenge of kubernetes

• Thousands of pods on one node, when all nodes restart simultaneously , the list API calls to kube-apiserver is too heavy.
• The hibernated pod will release the CPU/Mem resource, how do the scheduler schedule pods based on the real time resource usage?
• The hibernated pod may be awaken any time, how to make sure of no OOM on nodes? (based on water mark?)

Init

Warm

up

Running

Hibernate

Cold Start

User Request

Request Finish

SIGSTOP

SIGCONT

User Request

Front End

Controller

Worker Node

Node Agent

Container

Container

…

Request

Control Msg

Control data

Msg

Bus

Guest Applications

Page Tables

Bitmap Allocator

QKernel

Swapping Mgr

Mem Reclaim Mgr

QVisor

Host Linux Kernel

Swap File

REAP File

Full

Swap Out

Page Fault

Swap-in

REAP

Batch Swap-In

Init

Warm

Running

Hibernate

① Cold Start

②User Request

③Request Finish

④ SIGSTOP

⑤SIGCONT

⑦ User Request

Wake-up

⑨ SIGSTOP

Hibernate Running

⑧ Request Finish

⑥ User Request

①②③④⑤⑥⑦⑧⑨

4MB Mem Block

Control

Page

4KB

Page#1

4KB Page#1023

Control Page

Next

Free Page Bitmap

Refcnt Array

AtomicU16#0

AtomicU16#1

AtomicU16#2

AtomicU16#1023

4KB

Page#2

…

…

L2 Bitmap Array

u64#0

u64#1

Bit 0 ~ 63

Bit 64 ~ 127

u64#15

…

Bit 960 ~ 1023

Bit#0

Bit#1

Bit#15

Bit#63

…

L1 Bitmap (u64)

Free Page Bitmap

…

Bitmap Page Allocator

Head

Tail

…

Linux Process

Linux Container Applications

Linux Guest Kernel

QEMU / Firecracker …

Linux Container (cgroup, namespaces)

Linux Container Applications

QKernel

QVisor

Memory Management

Process Management

Network Stack

Virtual File System

Linux Host Kernel

KVM

Guest Applications

Guest Kernel

Virtual Machine Monitor (VMM)

Quark

Linux Virtual Machine

Host Kernel

System Call Virtualization Layer

Linux System Call

Quark System Call

TSoR Cluster

External

RDMA Conn

Node#2

Node#4

Node#3

Node#1

…

Cluster Node

RDMA Service

TSoR Gateway

Quark Pod#1

…

RDMA NIC

TCP Egress

TCP Ingress

TCP Ingress

TCP Egress

Orchestration Control Plane

Cluster Orchestration System

TSoR Client

Quark Pod#N

TSoR Client

…

Kubernetes Cluster

Node1

Pod#2

Pod#1

Node 2

Pod#3

External

…

①②③④⑤⑥

①

②

③

④

RDMA Service

RDMA Srv Client Mgr

SQ

CQ

Shared

Data Buffs

…

TSoR Gateway

TSOR Client

Shared

Data Buffs

Guest User Space

Guest Kernel Space

RDMA Channel

RDMA Service

RDMA Connection Mgr

RDMA Service

​

RDMA Connection Mgr

RDMA Connection

RDMA Channel (Control)

RDMA Channel (Data) #2

RDMA Channel (Data) #1

…

Write Ring Buffer

Write Ring Buffer

read Ring Buffer

read Ring Buffer

Node#1

Node#2

TSoR Gateway

Gateway Control Plane Agent

TSoR Client

Ingress TCP Layer

Egress TCP Layer

External

TCP Ingress Traffic

TCP Egress Traffic

RDMA Service

Orchestration Control Plane

Quark Pod

TSoR Ingress Traffic

TSoR Egress Traffic

RDMA NIC

Cluster Node#1 (192.168.0.0/24)

RDMA Service

…

TSoR (Ingress) Gateway

Client4

Cluster Node#2

(192.168.1.0/24)

RDMA Conn#1

RDMA Conn#2

TSoR Route Table

RDMA Conn#3

Pod#3 192.168.1.5

Cluster Node#3

(192.168.2.0/24)

Pod#4 192.168.2.8

Cluster Node#4

(192.168.3.0/24)

Pod#5 192.168.3.6

TSoR Cluster IP Table

External

TSoR Ingress Gateway Table

External IP

202.21.11.5

…

26

​

​

TCP Socket

Container

write(int socket, void *buf, ssize_t N))

​

​

TCP Socket

Container

read(int socket, void *buf, ssize_t N))

Write Buffer

Read Buffer

SysRead

Submit Queue

Push Request

Pop Request

RDMA QP

SQ

CQ

Handle Request

Complete Queue

RDMA QP

Handle Request

SQ

CQ

RDMA Write Immediate

TSoR_cli

share region

rdma_svc

rdma_svc

share region

TSoR_cli

Node

Sandbox

Process

TCP Conn

Node

Sandbox

Process

TCP Conn

• Scalability
• Multi-tenant

http://projectid.xxx/

API Gateway

API Gateway

…

Function Dispatcher

Function Dispatcher

…

Container#1

Container#2

Resource Manager

Node Agent

Event Queue

Ingress

Dispatcher

Region 1

Region 2

Pod A

Pod B

Pod B

Egress

Ingress

192.168.0.1:80

192.168.2.3:80

192.168.5.2:128

Pod B

Pod B

EIP 202.1.2.3:90

192.168.0.1:8080

192.168.82.3:8080

Region 3

Ingress

Pod B

Pod B

EIP 68.5.4.3:507

192.168.0.1:8080

192.168.82.3:8080

Egress Mapping

Ingress Mapping

Service Definition

Service Definition

Global K8S Mgr

Adaptor C

S3/Redis

Egress

Ingress

DNS:

PodB.local → 10.5.6.7

​

DNS

PodB.local → 10.5.8.5

Object Store

VPC

Pod A