1 of 26

Security Symposium

#SecuritySymposium

2 of 26

Security Symposium

The security implications of running software in containers

Scott McCarty

Principal Technical Product Manager

2020

2

3 of 26

2020 Security Symposium Welcome

Thank you for joining us for two days of security technology conversations.

A few notes:

We have three tracks with multiple talks:

Security and Compliance Automation (May 14)
Containers and Kubernetes Security (May 14)
DevSecOps (May 15)

→ You must register for each unique webinar and panel.
All sessions will be available on-demand, kindly register and you’ll be invited to view on-demand presentations.
View attachments tab for links to presentations and/or collateral.
Want more? Grab this ebook on Boosting Hybrid Cloud Security red.ht/security101
The panels are live, send us your questions throughout each day to infrastructure@redhat.com.
Keep an eye out for the ‘Financial Services Security Automation Summit’ on June 11 on BrightTalk .

#SecuritySymposium

3

Scott McCart - Twitter: @fatherlinux

4 of 26

The security implications of running software in containers

4

“Just because you’re paranoid doesn’t mean they aren’t after you.”

― Joseph Heller, Catch-22

Scott McCart - Twitter: @fatherlinux

5 of 26

The problems

6 of 26

The problems

6

Virtualization

Hypervisor

Virtual machine

Kernel space

Move the kernel around or move the user space around:

Fancy processes
Breaking the OS into 2 pieces
All containers share a kernel
Root-only exploits can be bad

Containers don’t contain

User space

Containerization

Hypervisor

Container image

Kernel space

User space

Dan Walsh (my shirt is dedicated to you)

Scott McCart - Twitter: @fatherlinux

Notes

Virtual Machines - Critically, the hypervisor gets to define and control the definition of what a virtual machine is. There is literally guaranteed compatibility because the hypervisor defines which virtual machines it can run. For example,

For example a vSphere 4.1 Hypervisor can only run 4.1 (and some older) virtual machines. vSphere 4.1 will never be able to run a virtual machine created with vSphere 6.7 in 2019.
Virtual Machines control the definition of the bits of the virtual machine that live in the white box. This includes the VMDK/QCOW2 as well as the metadata which define devices like network cards, video cards, etc.
A virtual machine cannot use any piece of hardware not defined in that “virtual machine” construct
There are only 13 hyper calls that the virtual machine can make to the hypervisor

Containers - the only thing defined with containers is the packaging format (OCI). There are no guarantees made that the binaries and libraries inside of a container will be compatible with the kernel on the container host.

Containers can make any one of 100s of system calls (
Portability – we can move the image anywhere we want. This also allows you to share infrastructure like Registry Servers.
Compatibility – they are designed and engineered to work together (See: Engineering compatibility with the Red Hat Universal Base Image). Compatibility is based on hardware architecture, operating system (Linux versus Windows), distribution of Linux (Ubuntu versus RHEL), and even age of the Linux distro in the container image (very old images may not work on newer hosts, while very new images may not work on older hosts).
Supportability – Red Hat can fix problems in the Container Image, Container Host, Container Engine, and the Linux kernel to make sure that all of these components work together over a defined life cycle. Supportability is based on a vendor’s ability to release, patch, version, and test a set of components together. Also, high quality support is based on a set of components that are designed and engineered to work together.
Deeper dive in this blog: http://crunchtools.com/deeply-understanding-the-different-between-portability-compatibility-and-supportability/
Deeper dive in this lab: https://learn.openshift.com/subsystems/container-internals-lab-2-0-part-1

7 of 26

The problems

7

Container images

Developers, operations, middleware, performance, and security specialists all have a role to play.

Fancy files?
Who controls what?
Who is responsible for what?
What about bad content?

WordPress (repository)

Registry server

Drupal (repository)

Image layer

WordPress (repository)

Image layer

Joomla (repository)

Image layer

WordPress

Php + fpm + httpd

corebuild

Layers and tags

Registry

Currency for collaboration

Scott McCart - Twitter: @fatherlinux

8 of 26

The problems

8

Code: mysqld

Configuration: /etc/my.cnf

Data: /var/lib/mysql

Hard work

Other stuff

Scott McCart - Twitter: @fatherlinux

9 of 26

New concepts

10 of 26

New concepts

10

CIA

Has data leaked from the container platform?

Confidentiality

Has somebody tampered with the container?

Integrity

Is the container up�and running?

Availability

Not them, but yeah, they might be after you too …

Scott McCart - Twitter: @fatherlinux

11 of 26

New concepts

11

Integrity

Container

Virtual server

Container

Virtual server

Container

Virtual server

Scott McCart - Twitter: @fatherlinux

Security is about levels of isolation, or tenancy

Back in the day, I used to telnet into a shared webserver in college. Literally, we could all run processes side by side, like a free system in Tron

Now days, sometimes you need an application in two different data centers with affected by different weather patterns or fault lines

The conversation is how much isololation is enough? Walk from left to right

Sometimes a process is enough isolation

Sometimes we need a container. If you are running in a shell in a container, you can run “ps -ef” and you don’t see other people’s processes

Sometimes we need the extra isolation of virtual server

Sometimes a physical server

Sometimes a different rack in the data center

Sometimes a different data center affected by different weather patterns or earchquake fault lines.

12 of 26

New concepts

12

Defense in depth

The practice of arranging defensive lines or fortifications so that they can defend each other, especially in case of an enemy incursion

Can we harden each layer?

Image scanning, signing, and blueprinting
Container host hardening
Platform delegation practices

Container platform

Container hosts

Container images

Scott McCart - Twitter: @fatherlinux

13 of 26

New concepts

13

The tenancy scale

Process

Container

Virtual server

Physical server

Rack

Datacenter

Scott McCart - Twitter: @fatherlinux

Security is about levels of isolation, or tenancy - it’s use case dependent
Back in the day, I used to telnet into a shared web server in college. Literally, we could all run processes side by side, like a free system in Tron
Now days, when analyzing a disaster recovery scenario, you may need an application in two different data centers, affected by different weather patterns or fault lines
The conversation is how much isololation is enough? Walk from left to right

Sometimes a process is enough isolation
Sometimes we need a container. If you are running in a shell in a container, you can run “ps -ef” and you don’t see other people’s processes
Sometimes we need the extra isolation of virtual server
Sometimes a physical server
Sometimes a different rack in the data center (may also think about region in a cloud, or different virtualization cluster)
Sometimes a different data center affected by different weather patterns or earthquake fault lines.

Problem: the containers versus VMs conversation is almost always restricted to compute, and rarely includes network and storage

14 of 26

New concepts

14

SELinux

SECCOMP

Security controls

Who you can talk to �Which objects in the kernel can communicate with other objects

What you can say�Limiting system calls is like limiting what words can be said

Scott McCart - Twitter: @fatherlinux

15 of 26

New technical controls

16 of 26

New technical controls

16

Bill of materials

Signing

Read-only containers

Podman diff to see what changed in a container

Container images

Our current operating model controls

Trusted content
Content provenance
Security scans
Remediation/patching
Bill of materials
CVE databases
Security response teams
Limited root access
Limited user access

Containers add the ability to easily apply techniques

Scott McCart - Twitter: @fatherlinux

17 of 26

New technical controls

17

Since containers are just fancy processes with a well-controlled user space, it’s easier to apply techniques like ...

NO_NEW_PRIVS, Read Only Images, –cap-drop=ALL, –user=user

Container host

Kernel quality
Capabilities
Read-only images
Limiting SSH access
Well understood and controlled configuration
Tenancy

SECCOMP + sVirt

Hardening:

We apply many of these techniques today:

Scott McCart - Twitter: @fatherlinux

18 of 26

New technical controls

18

Container platform

This layer exists in the world of physical and virtual servers but is typically an administrator-only tool, such as vCenter or HPSA.

In the world of containers, it’s much more common to delegate some access to developers, architects, and application owners.

Role-based authorization
Authentication
Environment isolation
User demarcation
Network separation
Key management

Scott McCart - Twitter: @fatherlinux

19 of 26

New technical controls

19

Network firewall (possibly layer 7)

Host-based firewall

Kernel quality

CVE database

Well-understood tenancy

Understood remediation and patching

Security scanning

Standard web application

Many security controls are inconvenient

Tripwire, SELinux, SECCOMP usually disabled

Mutable user space

No temporal understanding

No spatial understanding (code, configuration, data)

No platform delegation granularity

Patched infrequently

Benefits

Limitations

Scott McCart - Twitter: @fatherlinux

20 of 26

New technical controls

20

Containerized web application

All tools from standard web application

Read-only containers

Signing

Platform delegation

Spatial and temporal understanding of containers and application

Updates practiced more

Benefits

Limitations

Tenancy not well understood

Shared kernel

Applications hard to break up into code/configuration/data

More infrastructure (platform and management)

Need better understanding of applications

Many security controls are essentially free

Scott McCart - Twitter: @fatherlinux

21 of 26

Questions?

22 of 26

Questions

22

Citations

GitHub: Supply chain demo: http://bit.ly/2aY1WEO

The New Stack: Container defense in depth: http://bit.ly/2buXflB

Red Hat: Architecting containers series: h ttp://red.ht/2aXjVJF

Red Hat: A practical introduction to Docker terminology: http://red.ht/2beXHDD

WhatIs: Confidentiality, Integrity, and Availability: http://bit.ly/2bcStO9

Scott McCart - Twitter: @fatherlinux

23 of 26

Questions

23

Scott McCart - Twitter: @fatherlinux

24 of 26

Security Symposium

#SecuritySymposium

25 of 26

26 of 26

The security implications of running software in containers

Taming container fears

Scott McCarty

Principal Product Manager

Containers: Red Hat Enterprise Linux & Red Hat OpenShift