1 of 36

Pod Security Admission Integration

by Stanislav Láznička

1

CONFIDENTIAL Designator

2 of 36

Feature Use Case

Use cases

OpenShift clusters can run with the restricted Pod Security admission level by default without breaking pod controller workloads
Privileged users can opt-in/opt-out a namespace to and from the Pod Security admission
Workloads can be admitted based on the current pod security standards with minimal effort or no effort at all

Non-Goals

Make sure that bare pods (pods w/o pod controllers) continue working

Both the admission systems are triggered on pod creation, possibly preventing the API server from persisting the pod on admission denial. It is therefore impossible to take the Pod as an input without touching code of both the admission controllers. Such a code change would also very likely end up being rather unreasonable.

Add the ability to tune the cluster-wide PSa configuration

CONFIDENTIAL Designator

3 of 36

Global Pod Security Admission Configuration

Currently, the global PSa configuration looks as such:

Important takeaways from the configuration:

Globally, the admission plugin is turned on but runs in privileged mode

=> this allows to enforce more restrictive pod security profiles on a per-namespace basis

Auditing and warnings are turned on and are working on “restricted” level

=> possible audit events and warnings on namespaces that aren’t labelled otherwise

apiVersion: pod-security.admission.config.k8s.io/v1beta1�kind: PodSecurityConfiguration

defaults:

enforce: "privileged"

enforce-version: "latest"

audit: "restricted"

audit-version: "latest"

warn: "restricted"

warn-version: "latest"

exemptions:

usernames:

- system:serviceaccount:openshift-infra:build-controller

CONFIDENTIAL Designator

4 of 36

OpenShift Automated Pod Security Namespace Labeling

A mechanism to automatically synchronize Pod Security admission (hereinafter PSa) namespace labels

It is based on ServiceAccount permissions to use SCCs

=> the exists a mapping of SCC->PSa profile

It is opt-out
Synchronizes all non-system namespaces not prefixed with “openshift-”

“openshift-” prefixed namespaces are opt-in for the synchronization

The opting mechanism is driven per-namespace by the ”security.openshift.io/scc.podSecurityLabelSync” namespace label with “false” or “true” values (the latter for the “openshift-” specific namespaces)

Currently, we synchronize the “warn” and “audit” labels with their specific version labels to “v1.24”
The plan is to enable restricted pod security profile in OpenShift 4.12, the syncer will start synchronizing the “enforce” labels in that version as well

CONFIDENTIAL Designator

5 of 36

New SCCs

New SCCs were added to default installation in order to be able to match upstream pod security standards
These are:

restricted-v2
nonroot-v2
hostnetwork-v2

These new SCCs are a more restrictive version of their likely-named predecessors

They require the securityContext.seccompProfile to be set to “runtime/default”
All capabilities are dropped but it is possible to add NET_BIND_SERVICE
securityContext.allowPrivilegeEscalation must be set to “false”

In new installations, the restricted-v2 SCC replaces the old restricted in being the SCC that any authenticated user can use

CONFIDENTIAL Designator

6 of 36

Demo

kind: Pod

apiVersion: v1

metadata:

spec:

restartPolicy: Never

containers:

- name: fedora

image: fedora:latest

command:

- sleep

args:

- "infinity"

securityContext:

privileged: true

The next slide will show a set of commands that allow working more closely with the pod security admission and the label synchronization mechanism in 4.11

The following pod manifest is used as the privileged pod in the example:

CONFIDENTIAL Designator

7 of 36

Demo

$ oc new-project demo # create a new project

$ oc get ns demo -oyaml

apiVersion: v1

kind: Namespace

metadata:

annotations:

openshift.io/description: ""

openshift.io/display-name: ""

openshift.io/requester: system:admin

openshift.io/sa.scc.mcs: s0:c26,c10

openshift.io/sa.scc.supplemental-groups: 1000670000/10000

openshift.io/sa.scc.uid-range: 1000670000/10000

creationTimestamp: "2022-06-08T15:39:42Z"

labels:

kubernetes.io/metadata.name: demo

pod-security.kubernetes.io/audit: restricted

pod-security.kubernetes.io/audit-version: v1.24

pod-security.kubernetes.io/warn: restricted

pod-security.kubernetes.io/warn-version: v1.24

resourceVersion: "30293"

uid: c89c78b5-adc0-4aa7-b475-2e23e41e8884

spec:

finalizers:

- kubernetes

status:

phase: Activ

$ # the next command will print a warning, the label selector correctly evaluated the NS to be restricted according to the SA permissions but we are applying pod directly as a privileged user

$ oc create pod -f ~/privileged_pod.yaml

Warning: would violate PodSecurity "restricted:v1.24": privileged (container "fedora" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "fedora" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "fedora" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "fedora" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "fedora" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

pod/mypod-p created

$ # opt the namespace out of the labelsync

$ oc label ns demo --overwrite \

security.openshift.io/scc.podSecurityLabelSync=false \

pod-security.kubernetes.io/audit=privileged \

pod-security.kubernetes.io/warn=privileged

$

$ oc delete pods --all

$ # the following command no longer prints warnings as we configured PSa warnings and audits to privileged level

$ oc create pod -f ~/privileged_pod.yaml

pod/mypod-p created

$ oc label ns demo \

pod-security.kubernetes.io/audit- \

pod-security.kubernetes.io/audit-version- \

pod-security.kubernetes.io/warn- \

pod-security.kubernetes.io/warn-version-

$

$ oc delete pods –all

$ # the following command prints the warning as the namespace now uses the global configuration

$ oc create pod -f ~/privileged_pod.yaml

Warning: would violate PodSecurity "restricted:latest": privileged (container "fedora" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "fedora" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "fedora" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "fedora" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "fedora" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

pod/mypod-p created

CONFIDENTIAL Designator

8 of 36

Documentation

Product documentation for this feature is planned for 4.11 and is being tracked here: OSDOCS-3535
The enhancements:

CONFIDENTIAL Designator

9 of 36

Testing

The tests shall include these scenarios:

a namespace labelled for label synchronization gets a more privileged label when it contains a workload controller whose SA can run the pods with matching SCC
a namespace labelled for label synchronization does not get a more privileged label when it contains a workload controller whose SA cannot run the pods with matching SCC
a namespace not labeled for label synchronization does not get its PSa labels synchronized

CONFIDENTIAL Designator

10 of 36

11 of 36

Managing seccomp and SELinux policies with Security Profiles Operator

Presenter: Jakub Hrozek, ISC team

11

CONFIDENTIAL Designator

12 of 36

[SPO] Security Profile Operator

Epic: https://issues.redhat.com/browse/CMP-1091
The Security Profile Operator is not fully GA in OCP 4.11
Please expect another TE alongside OCP 4.12 or 4.13 to get more details
Check this blogpost that explains the real life use case for such operator: https://jhrozek.wordpress.com/2021/05/25/prevent-cve-exploits-in-your-kubernetes-cluster-with-seccomp-and-selinux-profiles/

CONFIDENTIAL Designator

13 of 36

The Use Case for Security Profiles

Containers are a weaker level of isolation compared to VMs or physical machines (see Dan Walsh’s talk)

A containerized process running as root without a security profile (a “privileged container”) is equivalent to root on the host - exploit that app and you might own the whole cluster

OCP provides good defaults (way better than upstream kube).

“Random” non-privileged user by default regardless of what the Dockefile says
“Random” per-namespace SELinux profiles

But, neither kube nor OCP provide an easy way to give the containerized process just a little bit more privileges. Users often open up everything (SELinux case: spc_t)

A typical case is a pod that needs to access something from the host fs (logs, devices, …)

Seccomp and SELinux (security profiles from now on) can restrict what the app can do

Seccomp limits the syscalls the process can make (e.g. read a file but not write to a file)
SELinux is more complex, label based system that restricts what the process can interact with and how (e.g. read and write log files but not files under /etc)

Bottom line: security profiles make exploiting that app and by extension the cluster harder

CONFIDENTIAL Designator

14 of 36

The Use Case for Security Profiles Operator

Kubernetes has had support for using SELinux or seccomp profiles for a number of releases
It’s hard however to use those profiles

Need to create them as files on nodes (seccomp) and even install them afterwards (SELinux)
This is decoupled from the app and even the cluster (what if the cluster scales), no k8s API

Even harder to create them

How many app developers can write a seccomp or selinux policy from scratch?

Without an operator helping with profile management, the profiles are hard to create, install & use
All of the above means that people all too often resort to running workloads with too high privileges (spc_t) even if the workload needs only a subset of elevated privileges
SPO allows you to

Create (record) profiles for you by inspecting an app as it runs
Manage profiles as first-class kubernetes objects

View state, link from other objects, protect from accidental deletion with finalizers…

CONFIDENTIAL Designator

15 of 36

Feature Use Case

As an SRE, I need the Security Profiles Operator to deploy a seccomp and SELinux profile along with my application to secure it
As an application developer, I need the SPO to record security profiles for my application and deploy them using similar tools as those that will be used in production
See upstream User Stories and Personas for more
SPO is an operator that makes it easier to use and record security profiles for applications

CONFIDENTIAL Designator

16 of 36

Making profile installation and usage easier

The profiles are traditionally not exposed as k8s objects
Instead, you need to manage them yourself as files
On OCP, this means pushing them to the cluster via MachineConfigs

..which triggers a reboot
Even harder for SELinux profiles (create a file, then install it with semodule -i)
Not usable in production

No way to delegate creating of the profile per namespace and manage with RBAC
Is the profile ready to be used after install? Did it maybe have a syntax error? What is the status per node? Is the profile still in use? Can it be safely removed?

CONFIDENTIAL Designator

17 of 36

Making profile creation easier

Creating the security profiles is hard
There are tools that help on local machines

Udica for SELinux
Oci-seccomp-bpf hook for seccomp

But profiles should be created for the same environments where they are used

Container runtime might matter (esp for seccomp)
Different OSs might have different SELinux labels on objects
Hard to set these up in CI

SPO allows to record seccomp and SELinux profiles for workloads

CONFIDENTIAL Designator

18 of 36

Installation and configuration

SPO is an additional operator, does not come pre-installed
Install from the Console UI

Operators -> OperatorHub -> Search -> Security Profiles Operator

Configuration through the spod/spod CR

oc explain spod.spec
oc patch spod spod -p '{"spec":{"verbosity": 1}}' --type=merge

CONFIDENTIAL Designator

19 of 36

API overview

SeccompProfile, SELinuxProfile

Represent the security profile itself and an overview of its status

SecProfNodeStatus

Status of a profile per node

ProfileBinding

Make sure that all images matching X get assigned profile Y

ProfileRecording

Create a profile for a matching pod
Upstream supports several ways of recording profiles (hook, built-in BPF program, tailing logs), OCP only support tailing logs as of 4.12 (4.13 might support BPF)

SPOD

SPO configuration object

CONFIDENTIAL Designator

20 of 36

API usage: create a profile

Minimal example: a profile that allows everything, just logs

$ cat <<EOF | oc create -f -

apiVersion: security-profiles-operator.x-k8s.io/v1beta1

kind: SeccompProfile

metadata:

namespace: my-namespace

spec:

defaultAction: SCMP_ACT_LOG

EOF

CONFIDENTIAL Designator

21 of 36

API usage: observe a profile

Wait for profile to be installed

$ oc wait --for=condition=ready seccompprofile profile1

Profile status

$ oc -n my-namespace get seccompprofile profile1 --output wide

Profile status per nodes

$ oc -n my-namespace get secprofilenodestatus -lspo.x-k8s.io/profile-id=profile1

CONFIDENTIAL Designator

22 of 36

API usage: take a profile into use

In pod manifest:

spec:

securityContext:

seccompProfile:

type: Localhost

localhostProfile: operator/my-namespace/profile1.json

CONFIDENTIAL Designator

23 of 36

API usage: record a profile

In pod manifest:

spec:

securityContext:

seccompProfile:

type: Localhost

localhostProfile: operator/my-namespace/profile1.json

CONFIDENTIAL Designator

24 of 36

API usage: bind a profile to an image

apiVersion: security-profiles-operator.x-k8s.io/v1beta1

kind: ProfileBinding

metadata:

spec:

profileRef:

kind: SeccompProfile

image: nginx:1.19.1

CONFIDENTIAL Designator

25 of 36

Architecture overview

Security-profiles-operator

SPOD DS pod (node1)

SPOD DS pod (node2)

seccomp-controller

Selinux-controller

selinuxd

seccomp-controller

Selinux-controller

selinuxd

Audit log parser

Profile recorder

Audit log parser

Profile recorder

spod/spod

SPO

Webhook

binding

recording

CONFIDENTIAL Designator

26 of 36

TIP: If needed cover this section in multiple slides!

DEMO

Show off the awesome work you/your team did, and explain how awesome this feature is!

CONFIDENTIAL Designator

27 of 36

Known issues and gotchas

Installing the selinux policies has nontrivial memory usage (spike, not continuous)
Using custom policies requires either a custom SCC or the privileged SCC
Profile binding and recording rely on having a webhook

By default the webhooks listen to all namespaces except those with runlevels 0 and 1
But the selector is configurable

At the moment (May 2022), the recorded profiles are per container instance and don’t merge automatically after being recorded.

We work towards fixing this for code freeze

Even though profile CRs are namespaced, the profiles can be used across namespaces

This is no different than without the operator, but should be restricted with another webhook or an external policy engine in a later release

Not all APIs have been graduated completely

Seccomp is beta, selinux is alpha2

CONFIDENTIAL Designator

28 of 36

Links and documentation

isc-team@redhat.com
#forum-compliance on Slack

CONFIDENTIAL Designator

29 of 36

Testing

CMP-1091 Make Security Profiles Operator a part of OpenShift�Test cases are available at link�Test scenarios:

the operator can be installed and uninstalled
seccomp profile management

install, uninstall, report status
assign a profile directly to a workload
bind image to a profi
test that the profile is actually taken into use - make sure that the permitted operations are executed, but those that are not permitted are denied

selinux profiles management

install, uninstall, report status
assign a profile directly to a workload
bind image to a profile
test that the profile is actually taken into use - make sure that the permitted operations are executed, but those that are not permitted are denied

seccomp and selinux profile recording

for both profile types, record several types of workloads (pods, deployments, ...)

metrics

:

�

CONFIDENTIAL Designator

30 of 36

Enable the RuntimeDefault seccomp profile in OpenShift for all workloads

Presenter: PETER HUNT

30

CONFIDENTIAL Designator

31 of 36

Feature Use Case

As an Openshift admin, I would like to reduce the attack surface of compromised pods.
As a developer using Openshift, I would like my pods to be secure by default.
As an Openshift admin I would like upgrading my clusters to not cause unexpected breakages.

TIP: If needed cover this section in multiple slides!

CONFIDENTIAL Designator

32 of 36

Overview

Upstream Kubernetes uses “unconfined” seccomp profile by default (which means it’s disabled)

SeccompDefault KEP (#2413) will fix this, aimed for beta in 1.25.

CRI-O has a configuration option “seccomp_use_default_when_empty”
For *new* Openshift 4.11 installations, it is on by default.

Users upgrading from 4.10 will *not* have it suddenly turn on

TIP: If needed cover this section in multiple slides!

CONFIDENTIAL Designator

33 of 36

Unshare

The “unshare” syscall (used to create new namespaces) is now blocked by default
This means podman will *not* work for users who are unable to change seccomp profile.

We are evaluating whether there should be an SCC that is added that gives access to unshare for these workloads (TODO: add card when it’s created)

TIP: If needed cover this section in multiple slides!

CONFIDENTIAL Designator

34 of 36

Documentation

Product documentation for this feature is planned for 4.11 and is being tracked here: https://issues.redhat.com/browse/OCPNODE-1025

CONFIDENTIAL Designator

35 of 36

Testing

Standard regression testing
Performance testing (tracked as part of https://issues.redhat.com/browse/CNV-18369)

TIP: QE can help you fill out this slide! Talk to your testers to assist with filling out this slide.

CONFIDENTIAL Designator

1 of 36

2 of 36

3 of 36

4 of 36

5 of 36

6 of 36

7 of 36

8 of 36

9 of 36

10 of 36

11 of 36

12 of 36

13 of 36

14 of 36

15 of 36

16 of 36

17 of 36

18 of 36

19 of 36

20 of 36

21 of 36

22 of 36

23 of 36

24 of 36

25 of 36

26 of 36

27 of 36

28 of 36

29 of 36

30 of 36

31 of 36

32 of 36

33 of 36

34 of 36

35 of 36

36 of 36