1 of 23

The security implications of running software in containers

Taming Container Fears

Scott McCarty

Principal Product Manager, Containers - RHEL & OpenShift

05/14/2019

2 of 23

“Just because you're paranoid doesn't mean they aren't after you.”

― Joseph Heller, Catch-22

3 of 23

THE PROBLEMS

4 of 23

CONTAINERS DON’T CONTAIN

Dan Walsh (my shirt is dedicated to you)

Move the kernel around or move the user space around

Fancy processes
Breaking the OS in two pieces
All containers share a kernel
Root only exploits can be ba’a’a’a’ad

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

5 of 23

CONTAINER IMAGES

Currency for collaboration

Developers, operations, middleware, performance, and security specialists all have a role to play

Fancy files
Who controls what?
Who is responsible for what?
What about bad content?

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

6 of 23

Hard Work

Code: mysqld
Configuration: /etc/my.cnf
Data: /var/lib/mysql
Other stuff :-)

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

7 of 23

NEW CONCEPTS

8 of 23

CIA

CONFIDENTIALITY

Has data leaked from the container platform?

Has somebody tampered with the container?

INTEGRITY

Is the container up and running?

AVAILABILITY

Not them, but yeah, they might be after you too….

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

9 of 23

Integrity

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

Security is about levels of isolation, or tenancy
Back in the day, I used to telnet into a shared webserver in college. Literally, we could all run processes side by side, like a free system in Tron
Now days, sometimes you need an application in two different data centers with affected by different weather patterns or fault lines
The conversation is how much isololation is enough? Walk from left to right

Sometimes a process is enough isolation
Sometimes we need a container. If you are running in a shell in a container, you can run “ps -ef” and you don’t see other people’s processes
Sometimes we need the extra isolation of virtual server
Sometimes a physical server
Sometimes a different rack in the data center
Sometimes a different data center affected by different weather patterns or earchquake fault lines.

10 of 23

Defense in Depth

the practice of arranging defensive lines or fortifications so that they can defend each other, especially in case of an enemy incursion.

Can we harden each layer?

Image scanning, signing, and blueprinting
Container host hardening
Platform delegation practices

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

11 of 23

The Tenancy Scale

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

Security is about levels of isolation, or tenancy - it’s use case dependent
Back in the day, I used to telnet into a shared web server in college. Literally, we could all run processes side by side, like a free system in Tron
Now days, when analyzing a disaster recovery scenario, you may need an application in two different data centers, affected by different weather patterns or fault lines
The conversation is how much isololation is enough? Walk from left to right

Sometimes a process is enough isolation
Sometimes we need a container. If you are running in a shell in a container, you can run “ps -ef” and you don’t see other people’s processes
Sometimes we need the extra isolation of virtual server
Sometimes a physical server
Sometimes a different rack in the data center (may also think about region in a cloud, or different virtualization cluster)
Sometimes a different data center affected by different weather patterns or earthquake fault lines.

Problem: the containers versus VMs conversation is almost always restricted to compute, and rarely includes network and storage

12 of 23

Security Controls

SELinux

Who you can talk to. Which objects in the kernel can communicate with other objects.

SECCOMP

What you can say. Limiting system calls is like limiting what words can be said

13 of 23

NEW TECHNICAL CONTROLS

14 of 23

CONTAINER IMAGES

Our current operating model controls:

Trusted Content (What’s in the container matters. Don’t install from hackme.com.)
Content Provenance (Track who changed what.)
Security Scans
Remediation/Patching
Bill of Materials
CVE Databases
Security Response Teams
Limit Root Access (Don’t oversell User Namespaces.)
Limit User Access (Who controls content.)

Containers add the ability to easily apply techniques such as:

Bill of Materials
Signing
Read-only Containers (Read-only servers were popular in the late 90s.)
Podman diff to see what changed in a container

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

15 of 23

CONTAINER HOST

Many of these techniques, we apply today.

Kernel Quality
Capabilities
Read Only Images
Limiting ssh access (root access and users)
Well understood/controlled configuration (cloud-init, Ansible)
Tenancy

Since containers are just fancy processes with a well-controlled user space, it’s easier to apply techniques like:

SECCOMP
sVirt
Hardening: NO_NEW_PRIVS, Read Only Images, –cap-drop=ALL, –user=user

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

16 of 23

CONTAINER PLATFORM

This layer exists in the world of physical and virtual servers but is typically an administrator only tool, such as vCenter or HPSA. In the world of containers, it’s much more common to delegate some access to developers, architects, and application owners.

Role-Based Authorization
Authentication (LDAP, network level access/restriction to the platform)
Environment Isolation (development, testing, production)
User Demarcation (kubectl exec)
Network Separation
Key Management

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

17 of 23

STANDARD WEB APPLICATION

Many security controls are inconvenient

Benefits

Network firewall (possibly layer 7)
Host based firewall
Kernel quality
CVE database
Well understood tenancy
Understood remediation/patching
Security scanning

Limitations

Tripwire, SELinux, SECCOMP usually disabled
Mutable user space
No temporal understanding
No spatial understanding (code, configuration, data)
No platform delegation granularity
Not patched often

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

18 of 23

CONTAINERIZED WEB APPLICATION

Many security controls are essentially free

Benefits

All tools from standard web application
Read only containers
Signing
Platform delegation
Spatial and temporal understanding of containers and application
Updates practiced more

Limitations

Tenancy not well understood
Shared kernel
Applications hard to break up into code/configuration/data
More infrastructure (platform and management)
Need better understanding of applications

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

19 of 23

Questions?

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

20 of 23

Citations

Supply Chain Demo on GitHub: http://bit.ly/2aY1WEO
The New Stack: Container Defense in Depth: http://bit.ly/2buXflB
Architecting Containers Series: http://red.ht/2aXjVJF
A Practical Introduction to Docker Terminology: http://red.ht/2beXHDD
Confidentiality, Integrity, Availability: http://bit.ly/2bcStO9

By Scott McCarty @fatherlinux

Scott McCarty. Twitter: @fatherlinux Blog: bit.ly/fatherlinux

21 of 23

For help getting started, visit http://brand.redhat.com/applications/presentations

to download the official Red Hat Presentation Guide

For more information on how to use this template within Google Slides, view this step-by-step guide.

RH_PREStemp_169_dark_v1.1_011116

22 of 23

THANK YOU

plus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHatNews

23 of 23

Load Applications at the Factory, not the Dock

Scott McCarty, Red Hat

There are several main value propositions to containers

Standardization: instead of loading boxes, bags, crates, barrels, bicycles, pianos, etc, we can now load one unit
Performance: Containers are fancy processes so there is no overhead for virtualization
Intermodal: Package/Load at the factory instead of the dock (New: never mentioned by Docker Inc)

In 1906, it took 45 days to load a ship, coincidentally about the same amount of time it takes to requisition a physical server
Imagine you are loading a bunch of lamps. If you break them on the dock, you have to wait for new ones to arrive. If you load at the factory, you go grab a new lamp and keep loading. Now, with code if something doesn’t work, you can fix it at the factory, on the developers laptop
“It works on my machine” always holds true for Chuck Norris