1 of 93

Network Service Mesh

Deep Dive

2 of 93

Housekeeping

Reach this slide deck via this QR Code

Reach Network Service Mesh KubeconNA 2018 Events Page

  • Lots of talks, demos, happy hour
  • Links to slides, and other collateral
  • Get involved!

3 of 93

Network Service Mesh Deep Dive (This Deck)

4 of 93

The Problem

5 of 93

Things like this:

Sarah

Corporate Intranet

Sarah’s Pod

L2/L3 connection

K8s interface

Security goes here...

6 of 93

Or This...

Network Service

secure-intranet-connectivity

Firewall

Pod

Sarah’s Pod

L2/L3 connection

VPN Gateway

Pod

L2/L3 connection

7 of 93

The Solution

8 of 93

Network Service Mesh Intro

9 of 93

When a NetworkServiceEndpoint exposes a Network Service, it can attach ‘Labels’ (key=value pairs). We call these Destination Labels.

In this example “Firewall Pod” would have DestinationLabel “app=firewall” and ”VPN Gateway Pod” would have label “app=vpn-gateway”.

​

Network Service

secure-intranet-connectivity

Firewall

Pod

Sarah’s Pod

VPN Gateway

Pod

app=firewall

app=vpn-gateway

10 of 93

When a Pod requests an L2/L3 Connection to a NetworkService, it can attach ‘Labels’ (key=value pairs). to that request. We call these SourceLabels. So in this example, when the Firewall Pod asks for an L2/L3 connection to the secure-intranet-connectivity Network Service, it uses SourceLabel “app=firewall”.

​

Network Service

secure-intranet-connectivity

Firewall

Pod

Sarah’s Pod

VPN Gateway

Pod

app=firewall

app=vpn-gateway

L2/L3 connection

app=firewall

11 of 93

Network Service

secure-intranet-connectivity

Firewall

Pod

Sarah’s Pod

VPN Gateway

Pod

app=firewall

app=vpn-gateway

L2/L3 connection

app=firewall

kind: NetworkService

apiVersion: V1

metadata:

name: secure-intranet-connectivity

spec:

payload: IP

matches:

- match:

sourceSelector:

app:firewall

route:

- destination:

destinationSelector:

app:vpn-gateway

​

12 of 93

... It should route them to a Network Service Endpoint with DestinationLabel ’app=firewall’

​

​

Network Service

secure-intranet-connectivity

Firewall

Pod

Sarah’s Pod

VPN Gateway

Pod

app=firewall

app=vpn-gateway

L2/L3 connection

app=firewall

kind: NetworkService

apiVersion: V1

metadata:

name: secure-intranet-connectivity

spec:

payload: IP

matches:

- match:

sourceSelector:

App:firewall

route:

- destination:

destinationSelector:

app:vpn-gateway

​

13 of 93

We add an ‘empty’ match which matches any connection request...

​

​

Firewall

Pod

Sarah’s Pod

VPN Gateway

Pod

app=firewall

app=vpn-gateway

L2/L3 connection

app=firewall

kind: NetworkService

apiVersion: V1

metadata:

name: secure-intranet-connectivity

spec:

payload: IP

matches:

- match:

sourceSelector:

app:firewall

route:

- destination:

destinationSelector:

App:vpn-gateway

- match:

route:

- destination:

destinationSelector:

app:firewall

​

Network Service

secure-intranet-connectivity

L2/L3 connection

14 of 93

And route them to Network Service Endpoints with DestinationLabel ’app=firewall’.

​

Firewall

Pod

Sarah’s Pod

VPN Gateway

Pod

app=firewall

app=vpn-gateway

L2/L3 connection

app=firewall

kind: NetworkService

apiVersion: V1

metadata:

name: secure-intranet-connectivity

spec:

payload: IP

matches:

- match:

sourceSelector:

app:firewall

route:

- destination:

destinationSelector:

App:vpn-gateway

- match:

route:

- destination:

destinationSelector:

app:firewall

​

Network Service

secure-intranet-connectivity

L2/L3 connection

15 of 93

Building Abstraction

16 of 93

Two Ways to Abstract Problems

Implementation Focused

Developer Focused

Cloud 1.0

Cloud-native

17 of 93

Two Ways to Abstract *these* Problems

Subnet1

Subnet2

Implementation Focused

Developer Focused

  • Previous implementation was:
    • Interface
    • Subnets
  • Add to the k8s API:
    • vInterfaces
    • vSubnets

Pod

k8s int

vint1

vint2

  • What does the Developer *really* want?

Pod

k8s int

Something that functionally does the thing needed when sent packets

L2/L3 connection

CONNECTIVITY TO corporate Intranet

Connect to another CNF

Allow POD to TALK to RADIO Network

GuaranteeD LATENCY/BANDWIDTH

​

LOAD BalancING

protecting from THREATS

CONNECTIVITY to ISOLATED Resources

18 of 93

What Developers Want:

PROTECTION FROM THREATS

GuaranteeD LATENCY/BANDWIDTH

ProxyING

LOAD BalancING

CONNECTIVITY to ISOLATED Resources

The Service Developers a developer may want for their L2/L3 traffic

​

  • Interface/Subnet/Network to are implementation details
    • What matters is the *Services* your L2/L3 payloads should receive

19 of 93

Three Easy Pieces

20 of 93

Network Service

21 of 93

Network Service

Network Service Endpoint

22 of 93

Network Service

​

​

Network Service Endpoint

Network Service Client Pod

L2/L3 connection

23 of 93

The Abstract Components

24 of 93

Network Service Registry Domain

Network Service Registry

Registry of:

  • NetworkServices
  • NetworkServiceEndpoints
  • NetworkServiceManagers
    • (more later on this)

25 of 93

Network Service Registry Domain

​

message NetworkService {

string name = 1;

string payload = 2;

repeated Match matches = 3;

}

​

Network Service Registry

26 of 93

Network Service Registry Domain

message NetworkService {

string name = 1;

string payload = 2;

repeated Match matches = 3;

}

​

message NetworkServiceEndpoint {

string network_service_name = 1;

string payload = 2;

string network_service_manager_name = 3;

string endpoint_name = 4;

map<string, string> labels = 5;

string state = 6;

}

​

Network Service Registry

27 of 93

Network Service Registry Domain

message NetworkService {

string name = 1;

string payload = 2;

repeated Match matches = 3;

}

​

message NetworkServiceEndpoint {

string network_service_name = 1;

string payload = 2;

string network_service_manager_name = 3;

string endpoint_name = 4;

map<string, string> labels = 5;

string state = 6;

}

​

message NetworkServiceManager {

string name = 1;

string url = 2;

google.protobuf.Timestamp last_seen = 3;

string state = 4;

}

​

Network Service Registry

28 of 93

Network Service Registry Domain

Network Service Registry

Network Service Manager Domain

Network Service Manager (NSM)

29 of 93

Network Service Registry Domain

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Endpoint (NSE)

Network Service Endpoint (NSE)

...

Network Service Registry

30 of 93

Network Service Registry Domain

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Client (NSC)

Network Service Client (NSC)

...

Network Service Endpoint (NSE)

Network Service Endpoint (NSE)

...

Network Service Registry

31 of 93

Network Service Registry Domain

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Mesh Dataplane (NSMD)

Network Service Client (NSC)

Network Service Client (NSC)

...

Network Service Endpoint (NSE)

Network Service Endpoint (NSE)

...

Network Service Registry

32 of 93

Network Service Registry Domain

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Mesh Dataplane (NSMD)

Network Service Client (NSC)

Network Service Client (NSC)

...

Network Service Endpoint (NSE)

Network Service Endpoint (NSE)

...

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Mesh Dataplane (NSMD)

Network Service Client (NSC)

Network Service Client (NSC)

...

Network Service Endpoint (NSE)

Network Service Endpoint (NSE)

...

...

Network Service Registry

33 of 93

Network Service Registry Domain

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Registry

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Manager Domain

Network Service Manager (NSM)

NetworkServiceManagers Interact With the Network Service Registry for Discovery

34 of 93

Network Service Registry Domain

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Registry

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Manager Domain

Network Service Manager (NSM)

NetworkServiceManagers Interact Peer to Peer to Establish L2/L3 Connections

35 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

36 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

37 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

Network Service Endpoint (NSE)

(Pod)

...

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

...

Network Service Endpoint (NSE)

(Pod)

Network Service Endpoint (NSE)

(Pod)

38 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

Network Service Client (NSC)

(Pod)

...

Network Service Endpoint (NSE)

(Pod)

Network Service Endpoint (NSE)

(Pod)

...

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

...

Network Service Client (NSC)

(Pod)

...

Network Service Endpoint (NSE)

(Pod)

Network Service Endpoint (NSE)

(Pod)

39 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

All Purple APIs shown are GRPC

NetworkServiceRegistry

message NSERegistration {

NetworkService network_service =1;

NetworkServiceManager network_service_manager =2;

NetworkServiceEndpoint networkservice_endpoint = 3;

}

�

service NetworkServiceRegistry {

rpc RegisterNSE (NSERegistration) returns (NSERegistration);

rpc RemoveNSE (RemoveNSERequest) returns (google.protobuf.Empty);

}

​

40 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

All Purple APIs shown are GRPC

NetworkServiceRegistry

message NSERegistration {

NetworkService network_service =1;

NetworkServiceManager network_service_manager =2;

NetworkServiceEndpoint networkservice_endpoint = 3;

}

�

service NetworkServiceRegistry {

rpc RegisterNSE (NSERegistration) returns (NSERegistration);

rpc RemoveNSE (RemoveNSERequest) returns (google.protobuf.Empty);

}

​

41 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

local.NetworkService

All Purple APIs shown are GRPC

message NetworkServiceRequest {

local.connection.Connection connection = 1;

repeated local.connection.Mechanism mechanism_preferences = 2;

}

�

service NetworkService {

rpc Request(NetworkServiceRequest) returns (local.connection.Connection);

rpc Close(local.connection.Connection) returns (google.protobuf.Empty);

}

​

42 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

local.NetworkService

All Purple APIs shown are GRPC

message Connection {

string id = 1;

string network_service = 2;

Mechanism mechanism = 3;

map<string,string> context = 4;

map<string,string> labels = 5;

}

message Mechanism {

MechanismType type = 1;

map<string,string> parameters = 2;

}

​

​

​

​

43 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

local.NetworkService

All Purple APIs shown are GRPC

message Connection {

string id = 1;

string network_service = 2;

Mechanism mechanism = 3;

map<string,string> context = 4;

map<string,string> labels = 5;

}

message Mechanism {

MechanismType type = 1;

map<string,string> parameters = 2;

}

​

​

​

​

enum MechanismType {

DEFAULT_INTERFACE = 0;

KERNEL_INTERFACE = 1;

VHOST_INTERFACE = 2;

MEM_INTERFACE = 3;

SRIOV_INTERFACE = 4;

HW_INTERFACE = 5;

}

44 of 93

Kubernetes Cluster

Node

Node

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Manager (NSM)

(Daemonset)

​

remote.NetworkService

message NetworkServiceRequest {

remote.connection.Connection connection = 1;

repeated remote.connection.Mechanism mechanism_preferences = 2;

}

�

service NetworkService {

rpc Request(NetworkServiceRequest) returns (remote.connection.Connection);

rpc Close(remote.connection.Connection) returns (google.protobuf.Empty);

}

​

All Purple APIs shown are GRPC

45 of 93

Kubernetes Cluster

Node

Node

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Manager (NSM)

(Daemonset)

​

remote.NetworkService

message Mechanism {

MechanismType type = 1;

map<string,string> parameters = 2;

}

message Connection {

string id = 1;

string network_service = 2;

Mechanism mechanism = 3;

map<string,string> context = 4;

map<string,string> labels = 5;

string source_network_service_manager_name = 6;

string destination_network_service_manager_name = 7;

string network_service_endpoint_name = 8;

}

​

All Purple APIs shown are GRPC

46 of 93

Kubernetes Cluster

Node

Node

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Manager (NSM)

(Daemonset)

​

remote.NetworkService

enum MechanismType {

NONE = 0;

VXLAN = 1;

VXLAN_GPE = 2;

GRE = 3;

SRV6 = 4;

MPLSoEthernet = 5;

MPLSoGRE = 6;

MPLSoUDP = 7;

}

​

​

All Purple APIs shown are GRPC

47 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

local.NetworkService

remote.NetworkService

NetworkServiceRegistry

All Purple APIs shown are GRPC

48 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

local.NetworkService

local.NetworkService

remote..NetworkService

NetworkServiceRegistry

All Purple APIs shown are GRPC

49 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Dataplane

DataplaneRegistrar

All Purple APIs shown are GRPC

message MechanismUpdate {

repeated remote.connection.Mechanism remote_mechanisms = 1;

repeated local.connection.Mechanism local_mechanisms = 2;

}

service Dataplane {

rpc Request (crossconnect.CrossConnect) returns (crossconnect.CrossConnect);

rpc Close (crossconnect.CrossConnect) returns (google.protobuf.Empty);

rpc MonitorMechanisms(google.protobuf.Empty) returns (stream MechanismUpdate);

}

​

50 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Dataplane

DataplaneRegistrar

All Purple APIs shown are GRPC

message CrossConnect {

string id = 1;

string payload = 2;

oneof source {

local.connection.Connection local_source = 3;

remote.connection.Connection remote_source = 4;

}

oneof destination {

local.connection.Connection local_destination = 5;

remote.connection.Connection remote_destination = 6;

}

}

​

​

51 of 93

NSC/NSE Topology View

NSC

NSE

Legend

CrossConnect

local.Connection

remote.Connection

Underlay

Packet

Encaped Packet

52 of 93

NSC/NSE Topology View

NSC

NSE

NSC

Dataplane

Node1

Cluster Topology View

NSE

Dataplane

Node2

Legend

CrossConnect

local.Connection

remote.Connection

Underlay

Packet

Encaped Packet

53 of 93

NSC

Dataplane

Node1

Cluster Topology View

NSE

Dataplane

Node2

Step1: Packet originates in NSC

Legend

CrossConnect

local.Connection

remote.Connection

Underlay

Packet

Encaped Packet

54 of 93

NSC

Dataplane

Node1

Cluster Topology View

NSE

Dataplane

Node2

Step2: Packet sent along NSC’s local.Connection

Legend

CrossConnect

local.Connection

remote.Connection

Underlay

Packet

Encaped Packet

55 of 93

NSC

Dataplane

Legend

CrossConnect

local.Connection

remote.Connection

Node1

Cluster Topology View

NSE

Dataplane

Underlay

Node2

Packet

Step3: Packet CrossConnected by Dataplane to remote.Connection

Encaped Packet

56 of 93

NSC

Dataplane

Node1

Cluster Topology View

NSE

Dataplane

Node2

Step4: Packet tunnel encapped for remote.Connection

Legend

CrossConnect

local.Connection

remote.Connection

Underlay

Packet

Encaped Packet

57 of 93

NSC

Dataplane

Node1

Cluster Topology View

NSE

Dataplane

Node2

Step5: Tunnel encapped packet goes over Underlay

Legend

CrossConnect

local.Connection

remote.Connection

Underlay

Packet

Encaped Packet

58 of 93

NSC

Dataplane

Node1

Cluster Topology View

NSE

Dataplane

Node2

Step6: Tunnel encapped packet arrives at Node2’s Dataplane, and its tunnel encap is removed

Legend

CrossConnect

local.Connection

remote.Connection

Underlay

Packet

Encaped Packet

59 of 93

NSC

Dataplane

Legend

CrossConnect

local.Connection

remote.Connection

Node1

Cluster Topology View

NSE

Dataplane

Underlay

Node2

Packet

Step7: Packet is CrossConnected by Node2’s Dataplane to NSE’s local.Connection

60 of 93

NSC

Dataplane

Legend

CrossConnect

local.Connection

remote.Connection

Node1

Cluster Topology View

NSE

Dataplane

Underlay

Node2

Packet

Step8: Node2’s Dataplane puts Packet onto NSE’s local.Connection

61 of 93

NSC

Dataplane

Legend

CrossConnect

local.Connection

remote.Connection

Node1

Cluster Topology View

NSE

Dataplane

Underlay

Node2

Packet

Step9: Packet arrives at NSE

62 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

local.NetworkService

Dataplane

local.NetworkService

remote.NetworkService

Dataplane

DataplaneRegistrar

DataplaneRegistrar

NetworkServiceRegistry

All Purple APIs shown are GRPC

63 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

MonitorCrossConnect

local.MonitorConnection

local.MonitorConnection

remote.MonitorConnection

MonitorCrossConnect

All Purple APIs shown are GRPC

64 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

MonitorCrossConnect

All Purple APIs shown are GRPC

menum CrossConnectEventType {

INITIAL_STATE_TRANSFER = 0;

UPDATE = 1;

DELETE = 2;

}

�

message CrossConnectEvent {

CrossConnectEventType type = 1;

map<string,CrossConnect> cross_connects = 2;

}

service MonitorCrossConnect {

rpc MonitorCrossConnects(google.protobuf.Empty) returns (stream crossconnect.CrossConnectEvent);

}

​

​

​

65 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

local.MonitorConnections

All Purple APIs shown are GRPC

enum ConnectionEventType {

INITIAL_STATE_TRANSFER = 0;

UPDATE = 1;

DELETE = 2;

}

message ConnectionEvent {

ConnectionEventType type = 1;

map<string,Connection> connections = 2;

}

service MonitorConnection {

rpc MonitorConnections(google.protobuf.Empty) returns (stream ConnectionEvent);

}

​

​

​

​

​

66 of 93

Kubernetes Cluster

Node

Node

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Manager (NSM)

(Daemonset)

​

remote.MonitorConnections

enum ConnectionEventType {

INITIAL_STATE_TRANSFER = 0;

UPDATE = 1;

DELETE = 2;

}

message ConnectionEvent {

ConnectionEventType type = 1;

map<string,Connection> connections = 2;

}

message MonitorScopeSelector {

string network_service_manager_name = 1;

}

service MonitorConnection {

rpc MonitorConnections(MonitorScopeSelector) returns (stream ConnectionEvent);

}

​

​

All Purple APIs shown are GRPC

67 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

All Purple APIs shown are GRPC

enum ConnectionEventType {

INITIAL_STATE_TRANSFER = 0;

UPDATE = 1;

DELETE = 2;

}

message ConnectionEvent {

ConnectionEventType type = 1;

map<string,Connection> connections = 2;

}

service MonitorConnection {

rpc MonitorConnections(google.protobuf.Empty) returns (stream ConnectionEvent);

}

​

local.MonitorConnections

68 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

MonitorCrossConnect

local.MonitorConnection

local.MonitorConnection

remote.MonitorConnection

MonitorCrossConnect

All Purple APIs shown are GRPC

69 of 93

Pods Die

70 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

remote.Connections

local.Connections

CrossConnect

CrossConnect

local.Connections

71 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

MonitorCrossConnect

local.MonitorConnection

local.MonitorConnection

remote.MonitorConnection

MonitorCrossConnect

All Purple APIs shown are GRPC

Skydive

MonitorCrossConnect

72 of 93

Auto Healing

Firewall

Pod

Sarah’s Pod

VPN Gateway

Pod

app=firewall

app=vpn-gateway

L2/L3 connection

app=firewall

Network Service

secure-intranet-connectivity

L2/L3 connection

VPN Gateway

Pod

app=vpn-gateway

73 of 93

Auto Healing

Firewall

Pod

Sarah’s Pod

app=firewall

app=firewall

Network Service

secure-intranet-connectivity

L2/L3 connection

VPN Gateway

Pod

app=vpn-gateway

74 of 93

Auto Healing

Firewall

Pod

Sarah’s Pod

app=firewall

Network Service

secure-intranet-connectivity

L2/L3 connection

VPN Gateway

Pod

L2/L3 connection

app=firewall

app=vpn-gateway

75 of 93

Three Easy Pieces

+

=

Six Easy Pieces

Three Implementation Details

76 of 93

Three Easy Pieces

Network Service

​

​

Network Service Endpoint

Network Service Client Pod

L2/L3 connection

77 of 93

  • Network Service Managers

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Manager Domain

Network Service Manager (NSM)

78 of 93

  • Cross Connects

= {

SourceConnection,

DestinationConnection,

}

79 of 93

  • Monitoring

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Endpoint (NSE)

(Pod)

...

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

remote.Connections

local.Connections

CrossConnect

CrossConnect

local.Connections

80 of 93

Everything is Just:

Request

Close

Monitor

81 of 93

To K8s… and Beyond

82 of 93

Network Service Registry Domain

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Mesh Dataplane (NSMD)

Network Service Client (NSC)

Network Service Client (NSC)

...

Network Service Endpoint (NSE)

Network Service Endpoint (NSE)

...

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Mesh Dataplane (NSMD)

Network Service Client (NSC)

Network Service Client (NSC)

...

Network Service Endpoint (NSE)

Network Service Endpoint (NSE)

...

...

Network Service Registry

83 of 93

Kubernetes Cluster

Node(Network Service Manager Domain)

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

Network Service Client (NSC)

(Pod)

...

Network Service Endpoint (NSE)

(Pod)

Network Service Endpoint (NSE)

(Pod)

...

...

Kubernetes API Server

(Network Service Registry via CRDs)

Network Service Manager (NSM)

(Daemonset)

​

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

...

Network Service Client (NSC)

(Pod)

...

Network Service Endpoint (NSE)

(Pod)

Network Service Endpoint (NSE)

(Pod)

84 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

...

Network Service Client (NSC)

(Pod)

...

Network Service Endpoint (NSE)

(Pod)

Network Service Endpoint (NSE)

(Pod)

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Mesh Dataplane (NSMD)

Network Service Client (NSC)

Network Service Client (NSC)

...

Network Service Endpoint (NSE)

Network Service Endpoint (NSE)

...

Register Network Service Endpoints

85 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Node(Network Service Manager Domain)

Network Service Manager (NSM)

(Daemonset)

​

Network Service Mesh Dataplane (NSMD)

(kernel/vswitch)

Network Service Client (NSC)

(Pod)

...

Network Service Client (NSC)

(Pod)

...

Network Service Endpoint (NSE)

(Pod)

Network Service Endpoint (NSE)

(Pod)

Network Service Manager Domain

Network Service Manager (NSM)

Network Service Mesh Dataplane (NSMD)

Network Service Client (NSC)

Network Service Client (NSC)

...

Network Service Endpoint (NSE)

Network Service Endpoint (NSE)

...

remote.NetworkService

86 of 93

Cluster1

Node1

nsmd1

Cluster2

Node2

nsmd2

eNSM2

eNSM1

VIM

VM

VM

VM

87 of 93

Proxy Network Service Manager (pNSM)

(with great power, comes great responsibility)

88 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Proxy Network Service Manager (pNSM)

Not a Real Network Service Endpoint (NSE)

Node1

Network Service Manager (NSM1)

Dataplane

Node2

Network Service Manager (NSM2)

Dataplane

app=physnw

89 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Proxy Network Service Manager (pNSM)

Not a Real Network Service Endpoint (NSE)

Node1

Network Service Manager (NSM1)

Dataplane

Node2

Network Service Manager (NSM2)

Dataplane

app=physnw

kind: NetworkService

apiVersion: V1

metadata:

name: some-ns

spec:

payload: IP

matches:

- match:

route:

- destination:

destinationSelector:

app:physnw

...

​

90 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Proxy Network Service Manager (pNSM)

Not a Real Network Service Endpoint (NSE)

Node1

Network Service Manager (NSM1)

Dataplane

Node2

Network Service Manager (NSM2)

Dataplane

app=physnw

kind: NetworkService

apiVersion: V1

metadata:

name: some-ns

spec:

payload: IP

matches:

- match:

route:

- destination:

destinationSelector:

app:physnw

...

​

remote.NetworkService

91 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Proxy Network Service Manager (pNSM)

Not a Real Network Service Endpoint (NSE)

Node1

Network Service Manager (NSM1)

Dataplane

Node2

Network Service Manager (NSM2)

Dataplane

app=physnw

kind: NetworkService

apiVersion: V1

metadata:

name: some-ns

spec:

payload: IP

matches:

- match:

route:

- destination:

destinationSelector:

app:physnw

...

​

remote.NetworkService

remote.NetworkService

92 of 93

Kubernetes Cluster

Kubernetes API Server

(Network Service Registry via CRDs)

Proxy Network Service Manager (pNSM)

Not a Real Network Service Endpoint (NSE)

Node1

Network Service Manager (NSM1)

Dataplane

Node2

Network Service Manager (NSM2)

Dataplane

app=physnw

kind: NetworkService

apiVersion: V1

metadata:

name: some-ns

spec:

payload: IP

matches:

- match:

route:

- destination:

destinationSelector:

app:physnw

...

​

remote.NetworkService

remote.NetworkService

L2/L3 connection

93 of 93

Q&A