1 of 48

CNP: User stories + Samples

NOTE: This powerpoint is meant to be viewed in accordance with the following google doc for completeness

2 of 48

User Stories:

The following is a list of the main user stories CNP is being designed to cover.

All Diagram sources can be found here

3 of 48

Global Assumptions

Focus of this Cluster Network Policy (CNP) API is on admin-controlled East-West network policies and network level multi-tenancy, not on North-South controls

What is a Tenant ?

A group of (1 or more) namespaces surrounded by a hard deny (ingress+egress) boundary
A cluster admin will typically use CNPs to enforce inter-tenant boundaries and delegate intra-tenant network policies to the namespace level user
Delegate : CNP chooses to skip/ ignore certain traffic patterns to be handled by normal K8s Network policies at the namespace level

4 of 48

Use cases for Cluster Network Policy

Strict Deny (non overridable)

Isolate Pods carrying sensitive data from Namespace named “sensitive-ns ” from all other Namespaces.

Strict Allow (non overridable)

Allow traffic to/from kube-dns Pods from kube-system Namespace and allow system monitoring namespace.

Strict Deny but allow exceptions

Strictly deny inter-tenant traffic but strictly allow traffic from system Namespace monitoring and kube-dns Pods

Strict Deny but delegate (tenants discretion)

Strictly deny inter-tenant traffic but delegate public Service (backed by Pods labeled “app: svc-pub ” from Namespace “bar-ns-1”)

Default disposition

What happens when delegated intra-ns traffic is not handled by Namespace owner. Default deny or allow?
Default rules for the cluster when no CNP or K8s NP rules apply

Cluster external: All or nothing for external traffic
Extensibility: Consider a future IPBlock selector, loggingPolicy
All inclusive policy

Deny inter-tenant and sensitive-ns traffic but strictly allow traffic to coreDNS Pods and system monitoring namespace AND delegate traffic to public service to developers.

5 of 48

Solutions considered

non-explicit priority

Empower, Deny, Allow
Empower, Allow, Deny

priority

6 of 48

Notes on current vendor Cluster network policies

Antrea:

Supports ClusterNetworkPolicy
Priority numbering per policy. Ordered rules within a policy
3 action types: Allow, Deny, Reject

Calico:

Supports GlobalNetworkPolicy
Based on priority numbering per policy instance and YAML listing order based priority within a policy instance
4 action types: Allow, Deny, Log, Pass

Cilium

Supports CiliumClusterwideNetworkPolicy
Two types: Allow policies and Deny policies (Deny takes precedence)
No priority numbering

Others

VMware NSX, Cisco ACI, Juniper, AWS/ GCP IaaS
All support some form of cluster scoped container network filtering or IaaS network ACLs
Most use some variation of priority numbering based approach (possibly in combination with security groups etc)

7 of 48

A: “Non-explicit priority”

B: “Priority Ordering”

No requirement to align with current vendor CNI implementations of Global/ Cluster Network Policy or network admin ACL approaches
3 Action Types: Empower, Deny, Allow
Priority is overloaded into Action Type:

Empower > Deny > Allow

All rules aggregated by Action type across policy instances (e.g. all Empower rules, all Deny rules, all Allow rules across all CNP policy instances)
Continued … (see next slide)

Some vendor CNI implementations of CNP use this approach (Calico, Antrea, Cisco, Juniper as well as network ACL implementations for IaaS/ public clouds) hence review what already works/ common operations practice & maybe retain some aspects if it helps with migration for vendor CNI implementations/ deployments in the field
Action Types: Deny, Allow, Pass
Priority/ order is encoded in a separate field separately from Action Type
Explicit ordering of rules across CNPs & within CNPs
Continued … (see next slide)

These are not necessarily mutually exclusive approaches … can have solutions that combine some useful aspects of each (plus variations)

8 of 48

A: “Non-explicit priority”

B: “Priority Ordering”

Contd .. from prior slide

Delegation is supported via Empower action and also by “fall through” from CNP to normal NetPol policies.
Empower also performs “Deny exception”
Aligns with netpol syntax .. (Aggregation across policies, order independence) but future extensibility to be investigated (ex. new actions require new priority, limiting flexibility)
Yaml syntax usability ? see examples on next slides

Contd from prior slide

Delegation is supported by “pass” action & “fall through” from CNP to normal NetPol policies
Real world feedback: Works but sometimes issue with inserting new rules/ creating priority holes ..
Extensible in future if other match or action types are added (e.g. IPBlock with overlapping CIDRs, packet logging actions)
Yaml syntax usability ? (see examples on next slides)

These are not necessarily mutually exclusive approaches … can have solutions that combine some useful aspects of each (plus variations)

9 of 48

User Story Samples

10 of 48

Notes for non-explicit Priority based samples

11 of 48

Notes for Priority based samples

Rule priority resolution:

Two rules in two separate CNP instances are prioritized by explicit priority value of the CNP
Two rules within the same CNP instance are prioritized by yaml listing order within the CNP

The Priority field is an integer value that can range from 0 to 1000

A higher value implies higher priority for the CNP
The priority value 0 is special in that it results in the CNP’s rules being evaluated after any network policies that apply to the same objects

If two CNPs with the same priority have a union in their match conditions the user will be warned on policy application, and behavior will be indeterministic. Recommended to add multiple rules into a single CNP in such cases.
The action field can have three possible values, allow, deny, or pass
The pass action delegates enforcement to standard network policy

If pass is used and no network policy rule applies the traffic will default to standard cluster behavior(allow)

Additional kubectl tooling:

Kubectl plugin to list CNPs in order of CNP priority for convenience in readability

Yaml syntax for CNP rule type and direction align with K8s NetPol yaml convention

12 of 48

1a. Isolate Pods carrying sensitive data from Namespace named “sensitive-ns” from all other Namespaces.

13 of 48

1a

foo-ns-1

foo-ns-2

bar-ns-1

kube-system

sensitive-ns

foo-pod

bar-pod

sensitive

kube-dns

svc-pub

Non-overridable DENY

Dropped traffic

14 of 48

1a: Non-explicit Priority (Empower, Deny, Allow)

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: strict-deny�spec:� appliedTo: // applies to Pods carrying sensitive data in namespace “Sensitive-ns”� namespaceSelector: matchLabels: kubernetes.io/metadata.name: sensitive-ns podSelector: matchLabels: type: sensitive� ingress:� - from: - namespaceSelector: {} action: Deny egress:� - to: - namespaceSelector: {} action: Deny

15 of 48

1a: Explicit Priority Ordering

// Deny for Sensitive namespace, must be highest priority in cluster to �// ensure sensitive ns is adequately isolated�apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: strict-deny�spec:� priority: 100� appliedTo:� // applies to sensitive-ns� namespaceSelector:� matchLabels:� kubernetes.io/metadata.name: sensitive-ns� ingress:� // Rules to strictly deny traffic from all other Namespaces� - from:� - namespaceSelector: {}� action: Deny egress: - to: - namespaceSelector: {} action: Deny

16 of 48

2a. Allow traffic to/from kube-dns Pods from kube-system Namespace and allow system monitoring namespace.

17 of 48

2a

Allowed traffic

foo-ns-1

foo-ns-2

bar-ns-1

kube-system

foo-pod

bar-pod

kube-dns

svc-pub

Monitoring-ns

18 of 48

2a: Non-explicit Priority (Empower, Deny, Allow)

// make sure all pods kube-system and the Monitoring ns can talk to all �// other namespaces // there should be no Deny policy for this traffic apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: allow-system�spec:� appliedTo: // applies to all Namespaces� namespaceSelector: {}� ingress:� - action: Allow from: // allow from system namespaces - namespaceSelector: matchExpressions: - {Key: kubernetes.io/metadata.name, Operator: In, Values: [“kube-system”, “monitoring-ns”]}

// allow from kube-dns pods - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system podSelector: matchLabels: app: kube-dns egress:� - action: Allow to: // allow to system namespaces - namespaceSelector: matchLabels: kubernetes.io/metadata.name: monitoring-ns // allow to kube-dns pods - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system podSelector: matchLabels: app: kube-dns

19 of 48

2a: Non-explicit Priority (Empower, Deny, Allow)

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: allow-to-all�spec:� appliedTo: // applies to system namespaces� namespaceSelector: matchLabels: kubernetes.io/metadata.name: monitoring-ns� ingress:� - action: Allow from: // allow from all namespaces - namespaceSelector: {} egress:� - action: Allow to: // allow to all namespaces - namespaceSelector: {}

20 of 48

2a: Explicit Priority ordering

// make sure all pods kube-system and the Monitoring ns can talk to all �// other namespaces // must be highest priority to ensure traffic is always allowed�apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: allow-system�spec:� priority: 100� appliedTo:� // applies to all Namespaces� namespaceSelector: {}� ingress:� // Allow from kube-dns pods and monitoring pods� - from:� - namespaceSelector:� matchLabels:� kubernetes.io/metadata.name: kube-system� podSelector:� matchLabels: app: kube-dns� action: allow

- namespaceSelector: matchExpressions: - {Key: kubernetes.io/metadata.name, Operator: In, Values: [“kube-system”, “monitoring-ns”]} egress:� // Allow to kube-dns pods and monitoring pods� - to:� - namespaceSelector:� matchLabels:� kubernetes.io/metadata.name: kube-system� podSelector:� matchLabels: app:kube-dns� action: allow - namespaceSelector: matchExpressions: - {Key: kubernetes.io/metadata.name, Operator: In, Values: [“kube-system”, “monitoring-ns”]}

21 of 48

3a. Strictly deny inter-tenant traffic but strictly allow traffic from system Namespace monitoring and kube-dns Pods

22 of 48

Please see additional Document for YAML Samples

23 of 48

4a. Strictly deny inter-tenant traffic but delegate public Service (backed by Pods labeled “app: svc-pub” from Namespace “bar-ns-1”)

24 of 48

4a

Tenant Bar Namespace

foo-ns-1

foo-ns-2

bar-ns-1

foo-pod

bar-pod

svc-pub

Tenant Foo Namespaces

Tenant Bar Namespaces

overridable DENY

Non-overridable DENY

“Delegate”: Exceptions to Deny. Tenants discretion to allow/deny traffic

Allowed traffic

“Delegate”: Exceptions to Deny. Tenants discretion to allow/deny traffic

Allowed traffic

Dropped traffic

25 of 48

4a

foo-ns-1

foo-ns-2

bar-ns-1

foo-pod

bar-pod

svc-pub

overridable DENY

Non-overridable DENY

“Delegate”: Exceptions to Deny. Tenants discretion to allow/deny traffic

Allowed traffic

“Delegate”: Exceptions to Deny. Tenants discretion to allow/deny traffic

Allowed traffic

Dropped traffic

26 of 48

4a: Non-explicit Priority (Empower, Deny, Allow)

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: delegate-svc-pub�spec: appliedTo: // applies to all tenant Namespaces� namespaceSelector: matchExpressions: - {Key: tenant, Operator: Exists} ingress:� - action: Empower from: // skip from public svc - namespaceSelector: matchLabels: kubernetes.io/metadata.name: bar podSelector: matchLabels: app: svc-pub

// skip intra-tenant - namespaces: scope: SameLabels labels: [“tenant”] - action: Deny from: // drop from all other namespaces. This can be replaced by using NotSameLabels Namespaces scope. - namespaceSelector: {} egress:� - action: Empower to: // skip to public svc - namespaceSelector: matchLabels: kubernetes.io/metadata.name: bar podSelector: matchLabels: app: svc-pub // skip intra-tenant - namespaces: scope: SameLabels labels: [“tenant”] - action: Deny to: // drop to all other namespaces. This can be replaced by using NotSameLabels Namespaces scope. - namespaceSelector: {}

27 of 48

4a: Non-explicit Priority (Empower, Deny, Allow) cont

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: allow-svc-pub�spec:� appliedTo: // applies to svc-pub Pods� namespaceSelector: matchLabels: kubernetes.io/metadata.name: bar podSelector: matchLabels: app: svc-pub� ingress:� - action: Empower from: // allow from all namespaces - namespaceSelector: {} egress:� - action: Empower to: // allow to all namespaces - namespaceSelector: {}

28 of 48

4a: Explicit Priority ordering

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: inter-tenant-deny�spec:� appliedTo:� namespaceSelector: matchLabels: type: tenant� priority: 10 Ingress: // Rules listed in order of priority // Explicitly skip/ delegate traffic from pub-svc - from: namespaceSelector: matchLabels: tenant: bar podSelector: matchLabels: app: svc-pub action: Pass

// Deny traffic from other tenant namespaces (tenant label not same) // traffic from same-tenant falls through delegated to netpol - from: namespaces: scope: NotSameLabels labels: [“tenant”] action: Deny Egress: - to: namespaceSelector: matchLabels: tenant: bar podSelector: matchLabels: app: svc-pub action: Pass

29 of 48

4a: Explicit Priority ordering cont.

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: allow-svc-pub�spec:� appliedTo: // applies to svc-pub Pods� namespaceSelector: matchLabels: tenant: bar podSelector: matchLabels: app: svc-pub� priority: 10 ingress:� - from: // allow from all namespaces namespaceSelector: {} action: pass egress:� - to: // allow to all namespaces namespaceSelector: {} action: pass

30 of 48

5a. What happens when delegated intra-ns traffic is not handled by Namespace owner. Default deny or allow?

31 of 48

5a

Tenant Bar Namespace

foo-ns-1

foo-ns-2

bar-ns-1

foo-pod

bar-pod

svc-pub

Tenant Foo Namespaces

Tenant Bar Namespace

overridable DENY

“Delegate”: Exceptions to Deny. Tenants discretion to allow/deny traffic

Dropped traffic

32 of 48

5a: Non-explicit Priority (Empower, Deny, Allow)

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: delegate-intra-tenant�spec:� appliedTo: // applies to all tenant Namespaces� namespaceSelector: matchExpressions: - {Key: tenant, Operator: Exists}� ingress:� - action: Empower from: // skip intra-tenant - namespaces: scope: SameLabels labels: [“tenant”] - action: Deny from: // drop from all other namespaces. This can be replaced by using NotSameLabels Namespaces scope. - namespaceSelector: {}

egress:� - action: Empower to: // skip intra-tenant - namespaces: scope: SameLabels labels: [“tenant”] - action: Deny to: // drop to all other namespaces. This can be replaced by using NotSameLabels Namespaces scope. - namespaceSelector: {}

33 of 48

5a: Non-explicit Priority (Empower, Deny, Allow) cont.

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterDefaultNetworkPolicy�metadata:� name: default-deny�spec:� appliedTo: // applies to all tenant Namespaces� namespaceSelector: matchExpressions: - {Key: tenant, Operator: Exists}� ingress: egress:

34 of 48

5a: Explicit Priority ordering

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: inter-tenant-deny�spec:� appliedTo:� namespaceSelector: matchLabels: type: tenant� priority: 10 Ingress: // Deny traffic from other tenant namespaces (tenant label not same) // traffic from same-tenant falls through delegated to netpol - from: namespaces: scope: NotSameLabels labels: tenant action: Deny Egress: - to: namespaces: scope: NotSameLabels labels: tenant action: Deny

35 of 48

5a: Explicit Priority ordering cont.

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: default-overridable-by-tenant �spec:� appliedTo:� namespaceSelector: matchLabels: type: tenant // Default policy hence priority 0 priority: 0 Ingress: - from: namespaces: scope: SameLabels labels: tenant action: Deny Egress: - to: namespaces: scope: SameLabels labels: tenant action: Deny

36 of 48

5b. Default deny rules to for the cluster when no CNP or K8s NP rules apply

37 of 48

5b: Non-explicit priority (Empower, Deny, Allow)

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterDefaultNetworkPolicy�metadata:� name: default-cluster-policy�spec:� appliedTo: // applies to all tenant Namespaces� namespaceSelector: matchLabels: type: tenant� ingress: egress:

38 of 48

5b: Explicit Priority ordering

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: default-overridable-by-tenant �spec:� appliedTo:� namespaceSelector: matchLabels: type: tenant // Default policy hence priority 0 priority: 0 Ingress: - from: namespaces: scope: SameLabels labels: tenant action: Deny Egress: - to: namespaces: scope: SameLabels labels: tenant action: Deny

39 of 48

7. Extensibility: Consider a future IPBlock selector, loggingPolicy

40 of 48

7: Extensibility: IPBlock selector in future

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: cnp-with-ipblock�spec:� appliedTo: // applies to all tenant Namespaces� namespaceSelector: matchLabels: type: tenant� egress:� - action: Empower // Rules to skip public Svc from strict deny rules to: - podSelector: matchLabels: app: pub-svc - ipBlock: cidr: 10.0.10.0/24 - ipBlock: cidr: 10.0.50.0/24

// Rules to strictly deny traffic to all other Namespaces - action: Deny to: - namespaceSelector: {} - ipBlock: cidr: 10.0.0.0/16 - action: Allow to: - ipBlock: cidr: 10.0.10.0/24

41 of 48

8. Deny inter-tenant and sensitive-ns traffic but strictly allow traffic to coreDNS Pods and system monitoring namespace AND delegate traffic to public service to developers.

42 of 48

8a

Tenant Bar Namespace

foo-ns-1

foo-ns-2

bar-ns-1

foo-pod

bar-pod

svc-pub

Tenant Foo Namespaces

Tenant Bar Namespaces

kube-system

sensitive-ns

sensitive

kube-dns

Monitoring-ns

overridable DENY

Non-overridable DENY

“Delegate”: Exceptions to Deny. Tenants discretion to allow/deny traffic

Allowed traffic

“Delegate”: Exceptions to Deny. Tenants discretion to allow/deny traffic

Allowed traffic

Dropped traffic

43 of 48

Please see additional Document for YAML Samples

44 of 48

Additional Notes

45 of 48

Notes on alternate simplified model

Instead of “Empower”s current definition,

Maybe have an Action called “Skip/ Delegate” which simply skips all remaining Cluster network policies completely instead of behaving like “Deny exception” as currently defined
Or
if we really want a “Deny exception” then call it that and place it under the Deny section … not “Empower”

If we really want to overload priority with action despite its issues, then

Accept that we will not have future actions in this CRD and
Make “Allow” to be higher priority than “Deny” since that seems to work better as the examples have shown.

Further simplification:

Do not even have explicit “Allow” action, only “Skip” and “Deny”.
Everything else is delegated

Explicit priority solution can also add “Skip/ Delegate” as an action type but without overloading priority into the action type.

46 of 48

Precedence model

47 of 48

6: Cluster external: All or none

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: deny-all�spec:� appliedTo: // applies to all tenant Namespaces� namespaceSelector: matchLabels: type: tenant� ingress: action: Deny egress: action: Deny

apiVersion: netpol.networking.k8s.io/v1alpha1�kind: ClusterNetworkPolicy�metadata:� name: allow-all�spec:� appliedTo: // applies to all tenant Namespaces� namespaceSelector: matchLabels: type: tenant� ingress: action: Allow egress: action: Allow

48 of 48

6. Cluster external: All or nothing for external traffic