1 of 51

Command and KubeCTL

Real-World Kubernetes Security for Pentesters

2 of 51

Hello,

Thanks you for your interest in these slides. Please feel free to share this deck internally with your teams or to friends. I built this deck in my free time so if you do, please consider tagging me on Twitter (@antitree) or dropping me an email (me@antitree.com) -- I’m also interested in having a conversation about securing Kubernetes and so feel free to provide your thoughts.

~Mark

3 of 51

My goal today:

Discuss Kubernetes case studies

Based on lots of assessments and findings
Common misconfigurations and flaws

Walk through tools and attack chain

Show what tools are the most useful
Walk through what attack vectors exist

Discuss Components

Talk about why each of these problems exist and how to learn more

4 of 51

Technical Director and consultant at NCC Group

Created the Container Practice within the company

Perform assessments of

Container environments
DevOps and Orchestration
Container breakout exercises

#ROC Hacker Community:

Rochester 2600 meetup group
Security B-Sides Rochester conference

Mark Manning // @antitree

me

5 of 51

In the beginning there was isolation and it was good!

IDEALLY: a container is:

A single process
Running in an isolated environment
Restricted to only perform its one and only task

Containers are a “security opportunity”

Containers could create a security boundary
Even by default Docker provides decent defenses from a Break Out
But often these are disabled...

6 of 51

And then came Kubernetes...

If you want to run containers in production you (at least) need:

AuthN/Z Controls
Networking Management
Storage Management

�Kubernetes provides:

A way to run a “cluster” of containers
Runtime management of containers
Automatic scalability across a cluster
Plugins, APIs, and extensible features to glue things together

7 of 51

8 of 51

10.10.10.10/24

10.10.10.20/24

10.10.10.30/24

10.10.20.0/16

.1

.2

.3

.4

9 of 51

Pods can run multiple containers

A Node is a host in the cluster

Some services will use a “sidecar” to perform inter-pod networking

A k8s “Namespace” logically separates workloads

The API is the kernel of the cluster and handles all operations within it

10 of 51

The Threat Model is up for interpretation

Operators define their own threat model and security boundaries
We can highlight security issues but the impact depends on how it’s used

Misconfigurations and flaws are subtle and require expertise

Kubernetes (Cloud, Devops,etc.) updates and changes FAST (defaults)
Learning the nuances of these systems is an investment
If teams have k8s experts, they often aren’t security experts (and vis versa)

Humans are still the best at doing security reviews of complex systems

Challenges of Pentesting K8s

11 of 51

Randomly Generated Case Studies

Ticketing is our Jam

We commit ourselves to cloud fineness!

The power maniacs!

How does their kubernetes infrastructure respond to the same exploit scenario

12 of 51

Power Maverick Threat Model

Mission Statement: A power company that sells a product to do power analysis on-prem inside your own infrastructure. Provide their customers with a Kubernetes cluster.

Security expectations:

Everything should be functional
Power Maverick should not create new vulnerabilities that affect their customer’s environment

13 of 51

Power Maverick Threat Model

Security controls:

Leverages that latest Kubernetes defaults
Everything deployed into the default namespace
Cluster deployed into a dedicated VPC
Helm 2 used to deploy services

./helm2 \� --host=tiller-deploy.kube-system.svc.cluster.local \� install antitree/brick

Results:

Attackers able to compromise a Pod are able to steal power data analytics for the customer
Business accepts the risk of this scenario
Low cost to maintain

14 of 51

CyberShmoo Threat Model

Mission Statement: A business designed solely to sell Shmoocon tickets. Multi-cloud architecture with teams of SREs and a need for geographically diverse availability.

Security expectations:

It must be able to sell tickets under high load
High value target for highly skilled hackers who want tickets so security is a priority
Limit the risk of insider attacks

15 of 51

Security controls:

Direct access to the clusters are abstracted away with Terraform and Spinnaker
Node Pools segregate workloads
High risk operations performed on hypervisor based systems

Results:

High barrier of entry for attackers
A Pod compromise means impact is localized to the Node Pool
Expensive to maintain

16 of 51

eKlowd Threat Model

Mission Statement: eKlowd is a geographically diverse startup. It has many developer groups that are self-managed. Each group deploys their application to centralized infrastructure.

Security expectations:

Each group must be able to control their own deployments to production
It is the responsibility of each group to follow proper security controls

17 of 51

eKlowd Threat Model

Security controls:

Each group gets a Namespace
Roles define that groups can only deploy to their own Namespace
A Pod Security Policy protects against a malicious developer compromising a Node

Results:

… let’s see

18 of 51

Attack Chain

Find a service to exploit

Exploit it

Steal its tokens

Curl the API to find endpoints

Kubectl with compromised token

Try to deploy pod blocked

Net scan for other services

Socat tunnels to access pods

Exploit Pod across namespaces

Takeover the host

Deploy malicious privileged Pod

Krew with access_matrix

Bypass PSP

Port Forward to deployed pod

...

19 of 51

20 of 51

Service Account Tokens

Authentication is performed by a “service account token” and always exists under:� /var/run/secrets/kubernetes.io/serviceaccount/token

Most Pods have permission to access the KubeAPI because:

A pod might be interested in information about itself within the context of the cluster
Your service needs to access another service within the cluster (e.g. Vault)
You aren’t sure what will happen if you remove it so you accept the defaults

21 of 51

22 of 51

Attack Chain

Find a service to exploit

Exploit it

Steal its tokens

Curl the API to find endpoints

Kubectl with compromised token

Try to deploy pod blocked

Net scan for other services

Socat tunnels to access pods

Exploit Pod across namespaces

Takeover the host

Deploy malicious privileged Pod

Krew with access_matrix

Bypass PSP

Port Forward to deployed pod

...

23 of 51

24 of 51

Kubectl For Attacks

Kubectl can:

Create Pods
Delete Pods
Run Services
Forward Ports
Tunnel Connections
… anything

First tool you want to get working:

Curl bash the binary into the Pod
Try to use it remotely from your machine

25 of 51

Now that you have a token, what do you do?

ServiceAccount tokens have built in constraints
Usually does not mean you can take over the cluster

What can I do in the container:

Curl bash binaries?
Syscall restrictions?

conmachi

amicontained

What can I do in the cluster:

kubectl auth can-i --list --token=XXXX

Using Compromised Tokens

26 of 51

Attack Chain

Find a service to exploit

Exploit it

Steal its tokens

Curl the API to find endpoints

Kubectl with compromised token

Try to deploy pod blocked

Net scan for other services

Socat tunnels to access pods

Exploit Pod across namespaces

Takeover the host

Deploy malicious privileged Pod

Krew with access_matrix

Bypass PSP

Port Forward to deployed pod

...

27 of 51

Roles and Rolebindings

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

rules:

- apiGroups:

- '*'

resources:

- '*'

verbs:

- '*'

- nonResourceURLs:

- '*'

verbs:

- '*'

More about RBAC auditing http://bit.ly/nccrbac

28 of 51

Attack Pods

Deploying your own Pod into a cluster could be:

Favorite tools you’re good at
Customized terminal commands
Pre-compiled binaries you don’t want to have to build

If the cluster allows to use third party registries, there’s a lot of opportunity:

antitree/brick

29 of 51

Attack Chain

Find a service to exploit

Exploit it

Steal its tokens

Curl the API to find endpoints

Kubectl with compromised token

Try to deploy pod blocked

Net scan for other services

Socat tunnels to access pods

Exploit Pod across namespaces

Takeover the host

Deploy malicious privileged Pod

Krew with access_matrix

Bypass PSP

Port Forward to deployed pod

...

30 of 51

Pod Security Policies

Pod Security Policies (PSPs) defines the minimum level of security a Pod must follow

PSPs applied on top of Roles

Even cluster-admin a PSP can (try to) stop you from generating services that do not follow a policy.

We have 2 namespaces with different PSPs:

Default: No PSP applied
Secure: Restricted environment to prevent a compromised Pod affecting the cluster

31 of 51

The Kubernetes API receives a request to run a Pod

It sends the request to an admission controller which decides YES or NO based on PSP

32 of 51

Privilege Escalation

A PSP cannot monitor containers to prevent something from becoming UID 0

This is why the “AllowPrivilegeEscalation” PSP policy is so important to set.

Why do we even need to be root?

New service tokens aren’t world readable anymore
Some functions (tcpdump) requires UID 0
Root means you can bypass all permission checks

apiVersion: policy/v1beta1

kind: PodSecurityPolicy

metadata:

spec:

privileged: false

seLinux:

rule: RunAsAny

supplementalGroups:

rule: RunAsAny

runAsUser:

# Require the container to run without root privileges.

rule: 'MustRunAsNonRoot'

fsGroup:

rule: RunAsAny

volumes:

- '*'

33 of 51

Attack Chain

Find a service to exploit

Exploit it

Steal its tokens

Curl the API to find endpoints

Kubectl with compromised token

Try to deploy pod blocked

Net scan for other services

Socat tunnels to access pods

Exploit Pod across namespaces

Takeover the host

Deploy malicious privileged Pod

Krew with access_matrix

Bypass PSP

Port Forward to deployed pod

...

34 of 51

35 of 51

Kubernetes Namespaces

Our Pod exists within a namespace:

RoleBindings apply only to our namespace
PSPs apply only to our namespace
Secrets are scoped only to our namespace

Simply a logical group of Kubernetes objects within the cluster.

The API tells you if your object can access other objects
Nothing built into namespaces for networking controls

Creates the opportunity for inter-pod communications

10.10.10.10/24

.1

.2

.3

.4

36 of 51

Attack Chain

Find a service to exploit

Exploit it

Steal its tokens

Curl the API to find endpoints

Kubectl with compromised token

Try to deploy pod blocked

Net scan for other services

Socat tunnels to access pods

Exploit Pod across namespaces

Takeover the host

Deploy malicious privileged Pod

Krew with access_matrix

Bypass PSP

Port Forward to deployed pod

...

37 of 51

38 of 51

Attack Chain

Find a service to exploit

Exploit it

Steal its tokens

Curl the API to find endpoints

Kubectl with compromised token

Try to deploy pod blocked

Net scan for other services

Socat tunnels to access pods

Exploit Pod across namespaces

Takeover the host

Deploy malicious privileged Pod

Krew with access_matrix

Bypass PSP

Port Forward to deployed pod

...

39 of 51

Socat to Internal Services

We’ve deployed a malicious Pod into the Secure namespace our options are:

Tool up our malicious Pod and attack adjacent services
Forward connections to adjacent services and use the tools on your box

Socat: Can be used to create a proxy to other services

kubectl run -n secure --restart=Never \

--image=alpine/socat socat1 \

-- -d -d tcp-listen:9999,fork,reuseaddr \

tcp-connect:10.24.2.3:5000

40 of 51

Krew

Kubectl is the base tool to interact with a cluster

Krew is a repository of plugins to extend it

like `pip` for kubectl
Supports dozens of tools that can run along side kubectl
Ideal for pentests as you're not modifying the cluster

Net-forward is a kubectl plugin listed on Krew that lets you make arbitrary TCP connections to any IP the cluster can access

More details about net-forward: http://bit.ly/nccfwd

41 of 51

42 of 51

Privilege Escalation From Pod To Host

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- name: brick

image: gcr.io/shmoocon-talk-hacking/brick

volumeMounts:

- mountPath: /chroot

securityContext:

runAsUser: 999

privileged: true

volumes:

- name: host

hostPath:

path: /

type: Directory

43 of 51

Attack Chain

Find a service to exploit

Exploit it

Steal its tokens

Curl the API to find endpoints

Kubectl with compromised token

Try to deploy pod blocked

Net scan for other services

Socat tunnels to access pods

Exploit Pod across namespaces

Takeover the cluster

Deploy malicious privileged Pod

Krew with access_matrix

Bypass PSP

Port Forward to deployed pod

...

44 of 51

45 of 51

eKlowd Security Issues

ClusterRole granted over-privileged service account: The webadmin service token could create, update, delete all Pods within the namespace

Do not automount service credentials
Fix the ClusterRole to only grant the service access to what it needs

PodSecurityPolicy able to be bypassed: The policy neglected to restrict SETUID-style attacks

Use AllowNewPrivileges=false directive in the PSP (or existing GKE ones)

Lack of image registry restrictions: The cluster could deploy from any registry it wanted including user supplied hostnames. A custom image deployment demonstrated that.

Restrict to only private, “blessed” images in controlled registries

Lack of Network Namespace Controls: The company assumed that the Namespace would provide adequate restrictions but there were no network policies or services (e.g. Callico) that would enforce the restriction

Implement a network security control (NetworkPolicy, Callico, Istio)

Default Namespace granted privileged deployments: There were no PSPs applied to the default Namespace so this allowed a privileged Pod deployment with a Hostmount to chroot the Node

Never allow any service to run as privileged
Disable HostMounts in PSPs

46 of 51

Future Topics To Dive Into

Service Mesh Restrictions:

What are the implications of running in an Istio Pod with UID 1337?
Does your service mesh truly enforce TLS? �

Exfiltration and Evasion

What are ways to hide within a Node to maintain persistence without triggering alerts?
Does auditing on your node understand the different between UID 0 in a container and UID 0 on the host?

Practical Compromising:

Compromising a cluster is fun but real world attacks are targeting the workloads
What kind of damage can an attacker do with access to the Images themselves?

eBPF, a Defense Feature or Attacker Tool:

We’ve talk about the attack vectors of eBPF (See Blackhat)
New research will better understand how eBPF fits into attacking containers at lower levels

47 of 51

Case Study Threat Models

Ticketing is our Jam

We commit ourselves to cloud fineness!

The power maniacs!

Do any of these organizations have the same security assumptions?

48 of 51

Where is the security boundary?

49 of 51

Conclusions

Kubernetes offers features that hopefully can fit with your threat model (not always true)

Namespace isolation is difficult
Privileged pods exist
RBAC applied depending on the generator

We’re not talking about 0days and CVE’s, we’re talking about design decisions

PSPs sometimes bypassable
Third party image registries
Seccomp still difficult to employ (Pod or container?)

�

50 of 51

NCC Group Tools and Research

Keyctl-unmask: exploit lack of seccomp policies to steal the Linux kernel keyrings

https://antitree.com/

go-pillage-registries: Extract secrets from the metadata of Docker registries at scale

http://bit.ly/nccpillage �

Net-forward: Create tunnels through Kubernetes clusters to arbitrary hosts

http://bit.ly/nccfwd �

RBAC tools: Discussion on using different tools for analyzing RBAC policies:

http://bit.ly/nccrbac

51 of 51

Thank you!

@antitree

Email: mark.manning@nccgroup.com

https://www.antitree.com

Jack Leadford @randohacker

Rory McCune @raesene

Ian Coldwater @IanColdwater

Brad Geesaman @BradGeesaman

Chris Le Roy @brompwnie

Greg Castle @mrgcastle

Tim Allclair @tallclair

Maya @MayaKaczorowski

Jay Beale @jaybeale