Network Service Mesh
Deep Dive
Housekeeping
Reach this slide deck via this QR Code
Reach Network Service Mesh KubeconNA 2018 Events Page
Network Service Mesh Deep Dive (This Deck)
The Problem
Things like this:
Sarah
Corporate Intranet
Sarah’s Pod
L2/L3 connection
K8s interface
Security goes here...
Or This...
Network Service
secure-intranet-connectivity
Firewall
Pod
Sarah’s Pod
L2/L3 connection
VPN Gateway
Pod
L2/L3 connection
The Solution
Network Service Mesh Intro
When a NetworkServiceEndpoint exposes a Network Service, it can attach ‘Labels’ (key=value pairs). We call these Destination Labels.
In this example “Firewall Pod” would have DestinationLabel “app=firewall” and ”VPN Gateway Pod” would have label “app=vpn-gateway”.
Network Service
secure-intranet-connectivity
Firewall
Pod
Sarah’s Pod
VPN Gateway
Pod
app=firewall
app=vpn-gateway
When a Pod requests an L2/L3 Connection to a NetworkService, it can attach ‘Labels’ (key=value pairs). to that request. We call these SourceLabels. So in this example, when the Firewall Pod asks for an L2/L3 connection to the secure-intranet-connectivity Network Service, it uses SourceLabel “app=firewall”.
Network Service
secure-intranet-connectivity
Firewall
Pod
Sarah’s Pod
VPN Gateway
Pod
app=firewall
app=vpn-gateway
L2/L3 connection
app=firewall
Network Service
secure-intranet-connectivity
Firewall
Pod
Sarah’s Pod
VPN Gateway
Pod
app=firewall
app=vpn-gateway
L2/L3 connection
app=firewall
kind: NetworkService
apiVersion: V1
metadata:
name: secure-intranet-connectivity
spec:
payload: IP
matches:
- match:
sourceSelector:
app:firewall
route:
- destination:
destinationSelector:
app:vpn-gateway
... It should route them to a Network Service Endpoint with DestinationLabel ’app=firewall’
Network Service
secure-intranet-connectivity
Firewall
Pod
Sarah’s Pod
VPN Gateway
Pod
app=firewall
app=vpn-gateway
L2/L3 connection
app=firewall
kind: NetworkService
apiVersion: V1
metadata:
name: secure-intranet-connectivity
spec:
payload: IP
matches:
- match:
sourceSelector:
App:firewall
route:
- destination:
destinationSelector:
app:vpn-gateway
We add an ‘empty’ match which matches any connection request...
Firewall
Pod
Sarah’s Pod
VPN Gateway
Pod
app=firewall
app=vpn-gateway
L2/L3 connection
app=firewall
kind: NetworkService
apiVersion: V1
metadata:
name: secure-intranet-connectivity
spec:
payload: IP
matches:
- match:
sourceSelector:
app:firewall
route:
- destination:
destinationSelector:
App:vpn-gateway
- match:
route:
- destination:
destinationSelector:
app:firewall
Network Service
secure-intranet-connectivity
L2/L3 connection
And route them to Network Service Endpoints with DestinationLabel ’app=firewall’.
Firewall
Pod
Sarah’s Pod
VPN Gateway
Pod
app=firewall
app=vpn-gateway
L2/L3 connection
app=firewall
kind: NetworkService
apiVersion: V1
metadata:
name: secure-intranet-connectivity
spec:
payload: IP
matches:
- match:
sourceSelector:
app:firewall
route:
- destination:
destinationSelector:
App:vpn-gateway
- match:
route:
- destination:
destinationSelector:
app:firewall
Network Service
secure-intranet-connectivity
L2/L3 connection
Building Abstraction
Two Ways to Abstract Problems
Implementation Focused
Developer Focused
Cloud 1.0
Cloud-native
Two Ways to Abstract *these* Problems
Subnet1
Subnet2
Implementation Focused
Developer Focused
Pod
k8s int
vint1
vint2
Pod
k8s int
Something that functionally does the thing needed when sent packets
L2/L3 connection
CONNECTIVITY TO corporate Intranet
Connect to another CNF
Allow POD to TALK to RADIO Network
GuaranteeD LATENCY/BANDWIDTH
LOAD BalancING
protecting from THREATS
CONNECTIVITY to ISOLATED Resources
What Developers Want:
PROTECTION FROM THREATS
GuaranteeD LATENCY/BANDWIDTH
ProxyING
LOAD BalancING
CONNECTIVITY to ISOLATED Resources
The Service Developers a developer may want for their L2/L3 traffic
Three Easy Pieces
Network Service
Network Service
Network Service Endpoint
Network Service
Network Service Endpoint
Network Service Client Pod
L2/L3 connection
The Abstract Components
Network Service Registry Domain
Network Service Registry
Registry of:
Network Service Registry Domain
message NetworkService {
string name = 1;
string payload = 2;
repeated Match matches = 3;
}
Network Service Registry
Network Service Registry Domain
message NetworkService {
string name = 1;
string payload = 2;
repeated Match matches = 3;
}
message NetworkServiceEndpoint {
string network_service_name = 1;
string payload = 2;
string network_service_manager_name = 3;
string endpoint_name = 4;
map<string, string> labels = 5;
string state = 6;
}
Network Service Registry
Network Service Registry Domain
message NetworkService {
string name = 1;
string payload = 2;
repeated Match matches = 3;
}
message NetworkServiceEndpoint {
string network_service_name = 1;
string payload = 2;
string network_service_manager_name = 3;
string endpoint_name = 4;
map<string, string> labels = 5;
string state = 6;
}
message NetworkServiceManager {
string name = 1;
string url = 2;
google.protobuf.Timestamp last_seen = 3;
string state = 4;
}
Network Service Registry
Network Service Registry Domain
Network Service Registry
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Registry Domain
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Endpoint (NSE)
Network Service Endpoint (NSE)
...
Network Service Registry
Network Service Registry Domain
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Client (NSC)
Network Service Client (NSC)
...
Network Service Endpoint (NSE)
Network Service Endpoint (NSE)
...
Network Service Registry
Network Service Registry Domain
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Mesh Dataplane (NSMD)
Network Service Client (NSC)
Network Service Client (NSC)
...
Network Service Endpoint (NSE)
Network Service Endpoint (NSE)
...
Network Service Registry
Network Service Registry Domain
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Mesh Dataplane (NSMD)
Network Service Client (NSC)
Network Service Client (NSC)
...
Network Service Endpoint (NSE)
Network Service Endpoint (NSE)
...
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Mesh Dataplane (NSMD)
Network Service Client (NSC)
Network Service Client (NSC)
...
Network Service Endpoint (NSE)
Network Service Endpoint (NSE)
...
...
Network Service Registry
Network Service Registry Domain
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Registry
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Manager Domain
Network Service Manager (NSM)
NetworkServiceManagers Interact With the Network Service Registry for Discovery
Network Service Registry Domain
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Registry
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Manager Domain
Network Service Manager (NSM)
NetworkServiceManagers Interact Peer to Peer to Establish L2/L3 Connections
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
Network Service Endpoint (NSE)
(Pod)
...
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
...
Network Service Endpoint (NSE)
(Pod)
Network Service Endpoint (NSE)
(Pod)
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
Network Service Client (NSC)
(Pod)
...
Network Service Endpoint (NSE)
(Pod)
Network Service Endpoint (NSE)
(Pod)
...
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
...
Network Service Client (NSC)
(Pod)
...
Network Service Endpoint (NSE)
(Pod)
Network Service Endpoint (NSE)
(Pod)
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
All Purple APIs shown are GRPC
NetworkServiceRegistry
message NSERegistration {
NetworkService network_service =1;
NetworkServiceManager network_service_manager =2;
NetworkServiceEndpoint networkservice_endpoint = 3;
}
�
service NetworkServiceRegistry {
rpc RegisterNSE (NSERegistration) returns (NSERegistration);
rpc RemoveNSE (RemoveNSERequest) returns (google.protobuf.Empty);
}
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
All Purple APIs shown are GRPC
NetworkServiceRegistry
message NSERegistration {
NetworkService network_service =1;
NetworkServiceManager network_service_manager =2;
NetworkServiceEndpoint networkservice_endpoint = 3;
}
�
service NetworkServiceRegistry {
rpc RegisterNSE (NSERegistration) returns (NSERegistration);
rpc RemoveNSE (RemoveNSERequest) returns (google.protobuf.Empty);
}
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
local.NetworkService
All Purple APIs shown are GRPC
message NetworkServiceRequest {
local.connection.Connection connection = 1;
repeated local.connection.Mechanism mechanism_preferences = 2;
}
�
service NetworkService {
rpc Request(NetworkServiceRequest) returns (local.connection.Connection);
rpc Close(local.connection.Connection) returns (google.protobuf.Empty);
}
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
local.NetworkService
All Purple APIs shown are GRPC
message Connection {
string id = 1;
string network_service = 2;
Mechanism mechanism = 3;
map<string,string> context = 4;
map<string,string> labels = 5;
}
message Mechanism {
MechanismType type = 1;
map<string,string> parameters = 2;
}
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
local.NetworkService
All Purple APIs shown are GRPC
message Connection {
string id = 1;
string network_service = 2;
Mechanism mechanism = 3;
map<string,string> context = 4;
map<string,string> labels = 5;
}
message Mechanism {
MechanismType type = 1;
map<string,string> parameters = 2;
}
enum MechanismType {
DEFAULT_INTERFACE = 0;
KERNEL_INTERFACE = 1;
VHOST_INTERFACE = 2;
MEM_INTERFACE = 3;
SRIOV_INTERFACE = 4;
HW_INTERFACE = 5;
}
Kubernetes Cluster
Node
Node
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Network Service Manager (NSM)
(Daemonset)
remote.NetworkService
message NetworkServiceRequest {
remote.connection.Connection connection = 1;
repeated remote.connection.Mechanism mechanism_preferences = 2;
}
�
service NetworkService {
rpc Request(NetworkServiceRequest) returns (remote.connection.Connection);
rpc Close(remote.connection.Connection) returns (google.protobuf.Empty);
}
All Purple APIs shown are GRPC
Kubernetes Cluster
Node
Node
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Network Service Manager (NSM)
(Daemonset)
remote.NetworkService
message Mechanism {
MechanismType type = 1;
map<string,string> parameters = 2;
}
message Connection {
string id = 1;
string network_service = 2;
Mechanism mechanism = 3;
map<string,string> context = 4;
map<string,string> labels = 5;
string source_network_service_manager_name = 6;
string destination_network_service_manager_name = 7;
string network_service_endpoint_name = 8;
}
All Purple APIs shown are GRPC
Kubernetes Cluster
Node
Node
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Network Service Manager (NSM)
(Daemonset)
remote.NetworkService
enum MechanismType {
NONE = 0;
VXLAN = 1;
VXLAN_GPE = 2;
GRE = 3;
SRV6 = 4;
MPLSoEthernet = 5;
MPLSoGRE = 6;
MPLSoUDP = 7;
}
All Purple APIs shown are GRPC
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
local.NetworkService
remote.NetworkService
NetworkServiceRegistry
All Purple APIs shown are GRPC
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
local.NetworkService
local.NetworkService
remote..NetworkService
NetworkServiceRegistry
All Purple APIs shown are GRPC
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Dataplane
DataplaneRegistrar
All Purple APIs shown are GRPC
message MechanismUpdate {
repeated remote.connection.Mechanism remote_mechanisms = 1;
repeated local.connection.Mechanism local_mechanisms = 2;
}
service Dataplane {
rpc Request (crossconnect.CrossConnect) returns (crossconnect.CrossConnect);
rpc Close (crossconnect.CrossConnect) returns (google.protobuf.Empty);
rpc MonitorMechanisms(google.protobuf.Empty) returns (stream MechanismUpdate);
}
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Dataplane
DataplaneRegistrar
All Purple APIs shown are GRPC
message CrossConnect {
string id = 1;
string payload = 2;
oneof source {
local.connection.Connection local_source = 3;
remote.connection.Connection remote_source = 4;
}
oneof destination {
local.connection.Connection local_destination = 5;
remote.connection.Connection remote_destination = 6;
}
}
NSC/NSE Topology View
NSC
NSE
Legend
CrossConnect
local.Connection
remote.Connection
Underlay
Packet
Encaped Packet
NSC/NSE Topology View
NSC
NSE
NSC
Dataplane
Node1
Cluster Topology View
NSE
Dataplane
Node2
Legend
CrossConnect
local.Connection
remote.Connection
Underlay
Packet
Encaped Packet
NSC
Dataplane
Node1
Cluster Topology View
NSE
Dataplane
Node2
Step1: Packet originates in NSC
Legend
CrossConnect
local.Connection
remote.Connection
Underlay
Packet
Encaped Packet
NSC
Dataplane
Node1
Cluster Topology View
NSE
Dataplane
Node2
Step2: Packet sent along NSC’s local.Connection
Legend
CrossConnect
local.Connection
remote.Connection
Underlay
Packet
Encaped Packet
NSC
Dataplane
Legend
CrossConnect
local.Connection
remote.Connection
Node1
Cluster Topology View
NSE
Dataplane
Underlay
Node2
Packet
Step3: Packet CrossConnected by Dataplane to remote.Connection
Encaped Packet
NSC
Dataplane
Node1
Cluster Topology View
NSE
Dataplane
Node2
Step4: Packet tunnel encapped for remote.Connection
Legend
CrossConnect
local.Connection
remote.Connection
Underlay
Packet
Encaped Packet
NSC
Dataplane
Node1
Cluster Topology View
NSE
Dataplane
Node2
Step5: Tunnel encapped packet goes over Underlay
Legend
CrossConnect
local.Connection
remote.Connection
Underlay
Packet
Encaped Packet
NSC
Dataplane
Node1
Cluster Topology View
NSE
Dataplane
Node2
Step6: Tunnel encapped packet arrives at Node2’s Dataplane, and its tunnel encap is removed
Legend
CrossConnect
local.Connection
remote.Connection
Underlay
Packet
Encaped Packet
NSC
Dataplane
Legend
CrossConnect
local.Connection
remote.Connection
Node1
Cluster Topology View
NSE
Dataplane
Underlay
Node2
Packet
Step7: Packet is CrossConnected by Node2’s Dataplane to NSE’s local.Connection
NSC
Dataplane
Legend
CrossConnect
local.Connection
remote.Connection
Node1
Cluster Topology View
NSE
Dataplane
Underlay
Node2
Packet
Step8: Node2’s Dataplane puts Packet onto NSE’s local.Connection
NSC
Dataplane
Legend
CrossConnect
local.Connection
remote.Connection
Node1
Cluster Topology View
NSE
Dataplane
Underlay
Node2
Packet
Step9: Packet arrives at NSE
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
local.NetworkService
Dataplane
local.NetworkService
remote.NetworkService
Dataplane
DataplaneRegistrar
DataplaneRegistrar
NetworkServiceRegistry
All Purple APIs shown are GRPC
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
MonitorCrossConnect
local.MonitorConnection
local.MonitorConnection
remote.MonitorConnection
MonitorCrossConnect
All Purple APIs shown are GRPC
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
MonitorCrossConnect
All Purple APIs shown are GRPC
menum CrossConnectEventType {
INITIAL_STATE_TRANSFER = 0;
UPDATE = 1;
DELETE = 2;
}
�
message CrossConnectEvent {
CrossConnectEventType type = 1;
map<string,CrossConnect> cross_connects = 2;
}
service MonitorCrossConnect {
rpc MonitorCrossConnects(google.protobuf.Empty) returns (stream crossconnect.CrossConnectEvent);
}
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
local.MonitorConnections
All Purple APIs shown are GRPC
enum ConnectionEventType {
INITIAL_STATE_TRANSFER = 0;
UPDATE = 1;
DELETE = 2;
}
message ConnectionEvent {
ConnectionEventType type = 1;
map<string,Connection> connections = 2;
}
service MonitorConnection {
rpc MonitorConnections(google.protobuf.Empty) returns (stream ConnectionEvent);
}
Kubernetes Cluster
Node
Node
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Network Service Manager (NSM)
(Daemonset)
remote.MonitorConnections
enum ConnectionEventType {
INITIAL_STATE_TRANSFER = 0;
UPDATE = 1;
DELETE = 2;
}
message ConnectionEvent {
ConnectionEventType type = 1;
map<string,Connection> connections = 2;
}
message MonitorScopeSelector {
string network_service_manager_name = 1;
}
service MonitorConnection {
rpc MonitorConnections(MonitorScopeSelector) returns (stream ConnectionEvent);
}
All Purple APIs shown are GRPC
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
All Purple APIs shown are GRPC
enum ConnectionEventType {
INITIAL_STATE_TRANSFER = 0;
UPDATE = 1;
DELETE = 2;
}
message ConnectionEvent {
ConnectionEventType type = 1;
map<string,Connection> connections = 2;
}
service MonitorConnection {
rpc MonitorConnections(google.protobuf.Empty) returns (stream ConnectionEvent);
}
local.MonitorConnections
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
MonitorCrossConnect
local.MonitorConnection
local.MonitorConnection
remote.MonitorConnection
MonitorCrossConnect
All Purple APIs shown are GRPC
Pods Die
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
remote.Connections
local.Connections
CrossConnect
CrossConnect
local.Connections
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
MonitorCrossConnect
local.MonitorConnection
local.MonitorConnection
remote.MonitorConnection
MonitorCrossConnect
All Purple APIs shown are GRPC
Skydive
MonitorCrossConnect
Auto Healing
Firewall
Pod
Sarah’s Pod
VPN Gateway
Pod
app=firewall
app=vpn-gateway
L2/L3 connection
app=firewall
Network Service
secure-intranet-connectivity
L2/L3 connection
VPN Gateway
Pod
app=vpn-gateway
Auto Healing
Firewall
Pod
Sarah’s Pod
app=firewall
app=firewall
Network Service
secure-intranet-connectivity
L2/L3 connection
VPN Gateway
Pod
app=vpn-gateway
Auto Healing
Firewall
Pod
Sarah’s Pod
app=firewall
Network Service
secure-intranet-connectivity
L2/L3 connection
VPN Gateway
Pod
L2/L3 connection
app=firewall
app=vpn-gateway
Three Easy Pieces
+
=
Six Easy Pieces
Three Implementation Details
Three Easy Pieces
Network Service
Network Service Endpoint
Network Service Client Pod
L2/L3 connection
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Manager Domain
Network Service Manager (NSM)
= {
SourceConnection,
DestinationConnection,
}
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Endpoint (NSE)
(Pod)
...
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
remote.Connections
local.Connections
CrossConnect
CrossConnect
local.Connections
Everything is Just:
Request
Close
Monitor
To K8s… and Beyond
Network Service Registry Domain
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Mesh Dataplane (NSMD)
Network Service Client (NSC)
Network Service Client (NSC)
...
Network Service Endpoint (NSE)
Network Service Endpoint (NSE)
...
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Mesh Dataplane (NSMD)
Network Service Client (NSC)
Network Service Client (NSC)
...
Network Service Endpoint (NSE)
Network Service Endpoint (NSE)
...
...
Network Service Registry
Kubernetes Cluster
Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
Network Service Client (NSC)
(Pod)
...
Network Service Endpoint (NSE)
(Pod)
Network Service Endpoint (NSE)
(Pod)
...
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network Service Manager (NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
...
Network Service Client (NSC)
(Pod)
...
Network Service Endpoint (NSE)
(Pod)
Network Service Endpoint (NSE)
(Pod)
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
...
Network Service Client (NSC)
(Pod)
...
Network Service Endpoint (NSE)
(Pod)
Network Service Endpoint (NSE)
(Pod)
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Mesh Dataplane (NSMD)
Network Service Client (NSC)
Network Service Client (NSC)
...
Network Service Endpoint (NSE)
Network Service Endpoint (NSE)
...
Register Network Service Endpoints
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Node(Network Service Manager Domain)
Network Service Manager (NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
Network Service Client (NSC)
(Pod)
...
Network Service Client (NSC)
(Pod)
...
Network Service Endpoint (NSE)
(Pod)
Network Service Endpoint (NSE)
(Pod)
Network Service Manager Domain
Network Service Manager (NSM)
Network Service Mesh Dataplane (NSMD)
Network Service Client (NSC)
Network Service Client (NSC)
...
Network Service Endpoint (NSE)
Network Service Endpoint (NSE)
...
remote.NetworkService
Cluster1
Node1
nsmd1
Cluster2
Node2
nsmd2
eNSM2
eNSM1
VIM
VM
VM
VM
Proxy Network Service Manager (pNSM)
(with great power, comes great responsibility)
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Proxy Network Service Manager (pNSM)
Not a Real Network Service Endpoint (NSE)
Node1
Network Service Manager (NSM1)
Dataplane
Node2
Network Service Manager (NSM2)
Dataplane
app=physnw
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Proxy Network Service Manager (pNSM)
Not a Real Network Service Endpoint (NSE)
Node1
Network Service Manager (NSM1)
Dataplane
Node2
Network Service Manager (NSM2)
Dataplane
app=physnw
kind: NetworkService
apiVersion: V1
metadata:
name: some-ns
spec:
payload: IP
matches:
- match:
route:
- destination:
destinationSelector:
app:physnw
...
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Proxy Network Service Manager (pNSM)
Not a Real Network Service Endpoint (NSE)
Node1
Network Service Manager (NSM1)
Dataplane
Node2
Network Service Manager (NSM2)
Dataplane
app=physnw
kind: NetworkService
apiVersion: V1
metadata:
name: some-ns
spec:
payload: IP
matches:
- match:
route:
- destination:
destinationSelector:
app:physnw
...
remote.NetworkService
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Proxy Network Service Manager (pNSM)
Not a Real Network Service Endpoint (NSE)
Node1
Network Service Manager (NSM1)
Dataplane
Node2
Network Service Manager (NSM2)
Dataplane
app=physnw
kind: NetworkService
apiVersion: V1
metadata:
name: some-ns
spec:
payload: IP
matches:
- match:
route:
- destination:
destinationSelector:
app:physnw
...
remote.NetworkService
remote.NetworkService
Kubernetes Cluster
Kubernetes API Server
(Network Service Registry via CRDs)
Proxy Network Service Manager (pNSM)
Not a Real Network Service Endpoint (NSE)
Node1
Network Service Manager (NSM1)
Dataplane
Node2
Network Service Manager (NSM2)
Dataplane
app=physnw
kind: NetworkService
apiVersion: V1
metadata:
name: some-ns
spec:
payload: IP
matches:
- match:
route:
- destination:
destinationSelector:
app:physnw
...
remote.NetworkService
remote.NetworkService
L2/L3 connection
Q&A