1 of 52

How to Secure OpenShift Environments and What Happens If You Don´t

Jan Harrie, ERNW GmbH, Germany

Slides: https://cutt.ly/HeNpUtD

@NodyTweet

2 of 52

$ whoami – Jan Harrie

  • Security Consultant @ERNW GmbH
  • Former Security Analyst/Pentester/WebApp-Monkey/Social-Engineer
  • M.Sc. IT-Security TU Darmstadt

​

  • Research // Interests:
    • K8s on-prem solutions
    • Cluster extensions
    • Gardening

@NodyTweet

3 of 52

Agenda

  1. OpenShift & Kubernetes – Introduction & Differences�
  2. Cluster Threats�
  3. (In-)Security of Clusters�
  4. Conclusion & Future Work

@NodyTweet

4 of 52

OpenShift & Kubernetes

Introduction & Differences

@NodyTweet

5 of 52

Introduction OpenShift

  • (On-Premise) Container Execution Platform from RedHat
  • First Release 05/2011
  • Current Stable Release: 4.2 (11/2019)
  • Host Operation System is RedHat Enterprise Linux and Container Linux from CoreOS
  • Since Version 3 with K8s under the hood
  • Since Version 4 Based on CRI-O, previously Docker
  • OKD Community Version, e.g., CentOS
    • Current Stable Release (10/2018): v3.11
    • Builds on K8s 1.11

@NodyTweet

6 of 52

OpenShift vs. K8s – Differences

Kubernetes

  • Role Based Access Control
  • Namespaces
  • Resource Limits
  • Security Context
  • Network Policies
  • Pod Security Policies

@NodyTweet

7 of 52

OpenShift vs. K8s – Differences

Kubernetes

  • Role Based Access Control
  • Namespaces
  • Resource Limits
  • Security Context
  • Network Policies
  • Pod Security Policies

OpenShift

  • Image Streams
  • Application Catalogue
  • User Management
  • Templates
  • Revision History
  • Security Context Constraints

@NodyTweet

8 of 52

Cluster Threats

@NodyTweet

9 of 52

What kind of threat model exist

for a cluster?

@NodyTweet

10 of 52

Cluster Threats

​

External Attacker

  • (Only) Access to Offered Services
  • No API Access
  • No Cluster-Insights Knowledge
  • Maybe public knowledge from DockerHub and Quay or GitHub

ETCD

Master

Worker

Pod

@NodyTweet

11 of 52

Cluster Threats

Internal Attacker

  • API Access
  • Control over Images and Deployments
  • Access to Code Repositories
  • Internal Cluster Knowledge

ETCD

Master

Worker

Pod

@NodyTweet

12 of 52

Internal are External Attackers one Step ahead

​

ETCD

Master

Worker

Pod

@NodyTweet

13 of 52

Internal are External Attackers one Step ahead

​

Internal Attacker

  • API Access
  • Control over Images and Deployments
  • Access to Code Repositories
  • “Cluster Internal Knowledge”

ETCD

Master

Worker

Pod

@NodyTweet

14 of 52

(In-)Security of Clusters

@NodyTweet

15 of 52

Source: KubeCon NA 2017 by Brad Geesaman [7]

@NodyTweet

16 of 52

Source: KubeCon NA 2017 by Brad Geesaman [7]

@NodyTweet

17 of 52

(In-)Security of Clusters

User Management

​

Network Security

​

A dive into Security Context Constraints (SCC’s)

​

@NodyTweet

18 of 52

(In-)Security of Clusters

User Management

​

Network Security

​

A dive into Security Context Constraints (SCC’s)

​

@NodyTweet

19 of 52

User Management in OpenShift

  • OpenShift offers integration into multiple Identity Provider (IdP)
    • E.g., HTPasswd, Keystone, LDAP authentication, Basic authentication (remote), Request header, GitHub, GitLab, Google, OpenID connect ; one IdP configureable
    • Implicit: mappingMethod: claim, Explicit: mappingMethod: lookup
  • Identities are Mapped to User in the Cluster
    • Identities are bases on the IdP, while a User is an Objects in the Cluster
  • Users can be organized in Groups
    • LDAP sync and manual assignment possible
  • "True User Removal" only possible in the IdP
    • Manual deleted Users and Identities are re-created on next login.

@NodyTweet

20 of 52

User Management in OpenShift

  • OpenShift offers integration into multiple Identity Provider (IdP)
    • E.g., HTPasswd, Keystone, LDAP authentication, Basic authentication (remote), Request header, GitHub, GitLab, Google, OpenID connect ; one IdP configureable
    • Implicit: mappingMethod: claim, Explicit: mappingMethod: lookup
  • Identities are Mapped to User in the Cluster
    • Identities are bases on the IdP, while a User is an Objects in the Cluster
  • Users can be organized in Groups
    • LDAP sync and manual assignment possible
  • "True User Removal" only possible in the IdP
    • Manual deleted Users and Identities are re-created on next login.

@NodyTweet

21 of 52

User Management in OpenShift

  • OpenShift offers integration into multiple Identity Provider (IdP)
    • E.g., HTPasswd, Keystone, LDAP authentication, Basic authentication (remote), Request header, GitHub, GitLab, Google, OpenID connect ; one IdP configureable
    • Implicit: mappingMethod: claim, Explicit: mappingMethod: lookup
  • Identities are Mapped to User in the Cluster
    • Identities are bases on the IdP, while a User is an Objects in the Cluster
  • Users can be organized in Groups
    • LDAP sync and manual assignment possible
  • "True User Removal" only possible in the IdP
    • Manual deleted Users and Identities are re-created on next login.

@NodyTweet

22 of 52

User Management in OpenShift

  • OpenShift offers integration into multiple Identity Provider (IdP)
    • E.g., HTPasswd, Keystone, LDAP authentication, Basic authentication (remote), Request header, GitHub, GitLab, Google, OpenID connect ; one IdP configureable
    • Implicit: mappingMethod: claim, Explicit: mappingMethod: lookup
  • Identities are Mapped to User in the Cluster
    • Identities are bases on the IdP, while a User is an Objects in the Cluster
  • Users can be organized in Groups
    • LDAP sync and manual assignment possible
  • "True User Removal" only possible in the IdP
    • Manual deleted Users and Identities are re-created on next login.

@NodyTweet

23 of 52

Role Based Access Control

A lot of default cluster-roles are shipped with OpenShift

  • Introduction of new roles is recommended rather then adjustment
  • Modification may lead to broken functionality

Authenticated User:

  • Implicit association with virtual group system:authenticated // system:authenticated:oauth��
  • What does this mean?

@NodyTweet

24 of 52

Role Based Access Control

A lot of default cluster-roles are shipped with OpenShift

  • Introduction of new roles is recommended rather then adjustment
  • Modification may lead to broken functionality

Authenticated User:

  • Implicit association with virtual group system:authenticated // system:authenticated:oauth��
  • What does this mean?

@NodyTweet

25 of 52

Role Based Access Control

Demo 1: https://asciinema.org/a/281016

@NodyTweet

26 of 52

What can probably go wrong?

If IdP is wrong configured:

  • Users can deploy workload in the cluster

and

Inspection of resolv.conf of the Pods:

$ cat /etc/resolv.conf�nameserver 172.30.0.2�search user1-p0.svc.cluster.local svc.cluster.local cluster.local�options ndots:5

@NodyTweet

27 of 52

What can probably go wrong?

If IdP is wrong configured:

  • Users can deploy workload in the cluster

and

  • Inspection of resolv.conf of the Pods:

$ cat /etc/resolv.conf�nameserver 172.30.0.2�search user1-p0.svc.cluster.local svc.cluster.local cluster.local�options ndots:5

@NodyTweet

28 of 52

What can probably go wrong?

Demo 2: https://asciinema.org/a/281704

@NodyTweet

29 of 52

Mitigation Strategy

  • Patch the Cluster Role:

$ oc adm policy remove-cluster-role-from-group self-provisioner system:authenticated�clusterrolebinding.rbac.authorization.k8s.io/self-provisioners patched��$ oc login -u user1�$ oc new-project user1-p1�Error from server (Forbidden): You may not request a new project via this API.

  • Define DNS policy per Pod [12]

​

​

@NodyTweet

30 of 52

(In-)Security of Clusters

User Management

​

Network Security

​

A dive into Security Context Constraints (SCC’s)

​

@NodyTweet

31 of 52

Network Security

Software Defined Networking build on Open vSwitch

​

​

  • Master-Nodes do not participate in the Cluster Network
  • Each Node gets its own Class-C network for the Pods assigned
  • Overlay communication via VXLAN
  • Integration of other Hosts into the cluster network by:
    • Host as an OpenShift node
    • Creating a VXLAN tunnel

Three plugins available:

  • Open vSwitch Subnet
  • Open vSwitch Multitenant
  • Open vSwitch Networkpolicy

@NodyTweet

32 of 52

Network Security – Open vSwitch Subnet

Configuration of Open vSwitch Subnet is not recommended

  • Cross project communication is possible

@NodyTweet

33 of 52

Network Security – Open vSwitch Subnet

Demo 3: https://asciinema.org/a/280323

@NodyTweet

34 of 52

Network Security – Open vSwitch Multitenant

Setup Plugin Open vSwitch Multitenannt to “prevent” cross-project communication

  • Each Project get is own Virtual Network ID (VNID)
  • Communication between different projects prohibit.
  • Projects can be joined together

BUT !

  • Separation on Namespace-Level
  • Projects with VNID 0 are more privileged
  • The project default has VNID 0

Side reference: TR19 – VXLAN Security or Injection [8]

@NodyTweet

35 of 52

Network Security – Open vSwitch Multitenant

Setup Plugin Open vSwitch Multitenannt to “prevent” cross-project communication

  • Each Project get is own Virtual Network ID (VNID)
  • Communication between different projects prohibit.
  • Projects can be joined together

BUT !

  • Separation on Namespace-Level
  • Projects with VNID 0 are more privileged
  • The project default has VNID 0

Side reference: TR19 – VXLAN Security or Injection [8]

@NodyTweet

36 of 52

Network Security – Open vSwitch Networkpolicy

Alternatively: stick to Open vSwitch Networkpolicy which allows you to deploy NetworkPolicies, and bock all ingress traffic [9] and add explicit whitelistings.

​

​

​

Further more, the plugin allows White- an Black-Listing on Layer3 [10] with CIDR notation or DNS

Configuration of Egress IP’s and Egress Proxies is possible [11]

​

kind: NetworkPolicy

metadata:

name: default-deny

spec:

podSelector: {}

policyTypes:

- Ingress

https://twitter.com/JackKleeman/status/1190354757308862468

@NodyTweet

37 of 52

(In-)Security of Clusters

User Management

​

Network Security

​

A dive into Security Context Constraints (SCC’s)

@NodyTweet

38 of 52

Security Context Constraints

  • Introduced by release 3.0 (05/2015)�
  • Secure Context Constraints (SCC's) is for Pods what RBAC is for the SAs�
  • Restrict execution of Pods�
  • Created by Cluster Administrator and assigned to Service Account�
  • Default SCC is 'restricted'

@NodyTweet

39 of 52

Security Context Constraints

Predefined Profiles

$ oc get scc�NAME PRIV CAPS SELINUX RUNASUSER [...]�anyuid false [] MustRunAs RunAsAny [...]�hostaccess false [] MustRunAs MustRunAsRange [...]�hostmount-anyuid false [] MustRunAs RunAsAny [...]�hostnetwork false [] MustRunAs MustRunAsRange [...]�nonroot false [] MustRunAs MustRunAsNonRoot [...]�privileged true [*] RunAsAny RunAsAny [...]�restricted false [] MustRunAs MustRunAsRange [...]

​

​

@NodyTweet

40 of 52

Security Context Constraints

Predefined Profiles – that allow privileged

$ oc get scc�NAME PRIV CAPS SELINUX RUNASUSER [...]�anyuid false [] MustRunAs RunAsAny [...]�hostaccess false [] MustRunAs MustRunAsRange [...]�hostmount-anyuid false [] MustRunAs RunAsAny [...]�hostnetwork false [] MustRunAs MustRunAsRange [...]�nonroot false [] MustRunAs MustRunAsNonRoot [...]�privileged true [*] RunAsAny RunAsAny [...]�restricted false [] MustRunAs MustRunAsRange [...]

​

​

@NodyTweet

41 of 52

Security Context Constraints

​

Demo 4: https://asciinema.org/a/280383

@NodyTweet

42 of 52

Security Context Constraints

Predefined Profiles – that allow hostPath, hostIPC, hostPID

$ oc get scc�NAME PRIV CAPS SELINUX RUNASUSER [...]�anyuid false [] MustRunAs RunAsAny [...]�hostaccess false [] MustRunAs MustRunAsRange [...]�hostmount-anyuid false [] MustRunAs RunAsAny [...]�hostnetwork false [] MustRunAs MustRunAsRange [...]�nonroot false [] MustRunAs MustRunAsNonRoot [...]�privileged true [*] RunAsAny RunAsAny [...]�restricted false [] MustRunAs MustRunAsRange [...]

​

​

@NodyTweet

43 of 52

Security Context Constraints

​

Demo 5: https://asciinema.org/a/280388

@NodyTweet

44 of 52

Security Context Constraints

Predefined Profiles – that allow root in container

$ oc get scc�NAME PRIV CAPS SELINUX RUNASUSER [...]�anyuid false [] MustRunAs RunAsAny [...]�hostaccess false [] MustRunAs MustRunAsRange [...]�hostmount-anyuid false [] MustRunAs RunAsAny [...]�hostnetwork false [] MustRunAs MustRunAsRange [...]�nonroot false [] MustRunAs MustRunAsNonRoot [...]�privileged true [*] RunAsAny RunAsAny [...]�restricted false [] MustRunAs MustRunAsRange [...]

​

​

@NodyTweet

45 of 52

Security Context Constraints

Predefined Profiles

$ oc get scc�NAME PRIV CAPS SELINUX RUNASUSER [...]�anyuid false [] MustRunAs RunAsAny [...]�hostaccess false [] MustRunAs MustRunAsRange [...]�hostmount-anyuid false [] MustRunAs RunAsAny [...]�hostnetwork false [] MustRunAs MustRunAsRange [...]�nonroot false [] MustRunAs MustRunAsNonRoot [...]�privileged true [*] RunAsAny RunAsAny [...]�restricted false [] MustRunAs MustRunAsRange [...]

​

​

@NodyTweet

46 of 52

Security Context Constraints

Predefined Profiles

$ oc get scc�NAME PRIV CAPS SELINUX RUNASUSER [...]�anyuid false [] MustRunAs RunAsAny [...]�hostaccess false [] MustRunAs MustRunAsRange [...]�hostmount-anyuid false [] MustRunAs RunAsAny [...]�hostnetwork false [] MustRunAs MustRunAsRange [...]�nonroot false [] MustRunAs MustRunAsNonRoot [...]�privileged true [*] RunAsAny RunAsAny [...]�restricted false [] MustRunAs MustRunAsRange [...]

​

​

@NodyTweet

47 of 52

Security Context Constraints – Summary

  • Do not use existing Security Context Constraints except:
    • restricted
    • nonroot�
  • Integration of SELinux is great benefit�
  • Create dedicated SCC’s with least privilege principle if necessary

@NodyTweet

48 of 52

Security Context Constraints – Summary

Demo 6: https://asciinema.org/a/280392

@NodyTweet

49 of 52

Conclusion & Future Work

@NodyTweet

50 of 52

Conclusion & Future Work

  • OpenShift raises the bar by it’s defaults, but must be further adjusted
  • Quick releases with feature extension/adjustment challenges the security research
  • Multiple components are dedicated developed by RedHat and are not spread for the community
  • OpenShift 4.2 is already available and components have been refactored and, new features and concepts are available

​

​

@NodyTweet

51 of 52

Thanks for your Attention

and take care of your defaults ;)

Slides: https://cutt.ly/HeNpUtD

@NodyTweet

52 of 52

References

[1] https://blog.openshift.com/introducing-red-hat-openshift-4-2-developers-get-an-expanded-and-improved-toolbox/�[2] https://twitter.com/bradgeesaman/status/1188541320626921473�[3] https://blog.openshift.com/red-hat-chose-kubernetes-openshift/�[4]https://kubernetes.io/blog/2015/04/borg-predecessor-to-kubernetes/�[5] https://ai.google/research/pubs/pub43438�[6] https://twitter.com/iancoldwater/status/1174793280011325456�[7] https://goo.gl/TNRxtd�[8] https://troopers.de/downloads/troopers19/TROOPERS19_AR_VXLAN_Security.pdf�[9] https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-deny-all-ingress-traffic�[10] https://docs.openshift.com/container-platform/3.11/admin_guide/managing_networking.html#admin-guide-limit-pod-access-egress�[11] https://docs.openshift.com/container-platform/3.11/admin_guide/managing_networking.html#admin-guide-limit-pod-access-egress-router�[12] https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy�[13] https://cookbook.openshift.org/users-and-role-based-access-control/how-can-i-enable-an-image-to-run-as-a-set-user-id.html�

@NodyTweet