1 of 37

L1051/L1052 - LINUX CONTAINER INTERNALS

How they really work

Scott McCarty, 5/21/2018

2 of 37

ARCHITECTURE

3 of 37

ARCHITECTURE

The Internet is WRONG :-)

Important corrections

Containers do not run ON docker. Containers are processes - they run on the Linux kernel. Containers are Linux.
The docker daemon is one of the many user space tools/libraries that talks to the kernel to set up containers

3

Scott McCarty, Twitter: @fatherlinux

4 of 37

Production-Ready Containers

What are the building blocks you need to think about?

4

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

5 of 37

TYPICAL ARCHITECTURE

Bringing it All Together

In distributed systems, the user must interact through APIs

5

Scott McCarty, Twitter: @fatherlinux

6 of 37

CONTAINERS ARE LINUX

The Libraries, and Data Structures

Userspace libraries interact with the kernel to isolate processes

Libraries

LXC, LXD, LibContainer, systemd nspawn, LibVirt

Kernel Data Structures

Name Spaces
Cgroups
SELinux

6

Scott McCarty, Twitter: @fatherlinux

7 of 37

THE USER SPACE TOOL CHAIN

On a Single Host

The user space tool chain adds the following:

A local daemon
Simple CLI/REST interface
Support for container images (OCI) and connection to registries

7

Scott McCarty, Twitter: @fatherlinux

8 of 37

THE ORCHESTRATION TOOLCHAIN

On Multiple Hosts

The orchestration toolchain adds the following:

More daemons (it’s a party) :-)
Scheduling across multiple hosts
Application Orchestration
Distributed builds (OpenShift)
Registry (OpenShift)

8

Scott McCarty, Twitter: @fatherlinux

Notes

This outlines the Red Hat tool chain at a very high level
There are other registries, build tools, etc and OpenShift is flexible and can integrate well with existing tools as well
Basically, the user has four main options for creating processes. Some are referred to as processes, some are referred to as containers, but really it’s not black and white

Creates processes through a shell - these are normal processes. The path is basically: user -> Linux kernel
Configures Systemd to create processes (daemons) - these are generally normal processes, but often include cgroups, and SELinux contexts. The path is basically: user -> Systemd -> Linux kernel
Requests that the Docker daemon create processes on their behalf - these are generally referred to as containerized processes. They include cgroupus, SELinux, AND kernel namespaces. The path is basically: user -> dockerd -> contianerd -> runc -> Linux kernel
Request that the Kubernetes master create processes on their behalf - these are referred to as containerized processes and are "scheduled" on a cluster. They include cgroupus, SELinux, AND kernel namespaces. The path is basically: user -> kubernetes master (openshift master) -> kubelet (openshift node)-> dockerd -> contianerd -> runc -> Linux kernel

9 of 37

THE COMMUNITY LANDSCAPE

Open Source, Leadership & Standards

The landscape is made up of committees, standards bodies, and open source projects:

Docker/Moby
Kubernetes/OpenShift
OCI Specifications
Cloud Native Technical Leadership

9

Scott McCarty, Twitter: @fatherlinux

10 of 37

OVERVIEW OF THE DIFFERENT STANDARDS

Vendor, Community, and Standards Body driven

Many different standards

10

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

11 of 37

WORKING TOGETHER

Standards in different places achieve different goals

Different standards are focused on different parts of the stack.

Container Images & Registries
Container Runtimes
Container Networking

11

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

12 of 37

CONTAINER IMAGES

13 of 37

Fancy Files

People forget about Glibc...

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

14 of 37

Container Images

Virtual machines and container environments

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

From a developer’s perspective this is what a running application now looks like
Historically, developers did not have a lot of influence over how the libraries and language runtimes were managed because they were connected to the operating system which was managed as a single primitive
They have tight control, or at a minimum very good collaboration to create all of the content in the container
Above and below the layer of abstraction:

Below the interface line is the container host and that region is clearly and cleanly controlled by operations
Above the interface line is the container image. This is the region that the developer now has influence over
The interface on the left are hypercalls (loosely coupled: can run any operating system)
The interface on the right are system calls (tightly coupled to operating system)

15 of 37

Fancy Files

Actually, they are layers...

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

16 of 37

Fancy File Servers

Actually, they are repositories

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

17 of 37

Fancy Files

How do we currently collaborate in the user space?

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

18 of 37

Fancy Files

The future of collaboration in the user space....

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

19 of 37

Fancy Files

The future of collaboration in the user space....

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

20 of 37

CONTAINER HOST

21 of 37

Fancy Processes

People forget about Glibc...

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

22 of 37

Linux Containers

Fancy Processes

Regular Linux Process

Containerized Process

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

23 of 37

Containerized Processes

Starting the process with namespaces, cgroups, and security controls

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

24 of 37

Containerized Processes

Starting the process in a namespace

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

25 of 37

Containerized Processes

The containerized process still use the underlying kernel abstractions...

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

26 of 37

Mounts

Copy on write vs. bind mounts

Scott McCarty Twitter: @fatherlinux Blog: bit.ly/fatherlinux

27 of 37

CONTAINER ORCHESTRATION

28 of 37

Application Containers

This is what most people think of with Docker

RED HAT AND CONTAINERS - CONFIDENTIAL - NDA REQUIRED

29 of 37

Container Orchestration

Multiple nodes changes everything

RED HAT AND CONTAINERS - CONFIDENTIAL - NDA REQUIRED

30 of 37

Container Orchestration

You can hack a solution together yourself, but it’s ugly...

RED HAT AND CONTAINERS - CONFIDENTIAL - NDA REQUIRED

31 of 37

Kubernetes/OpenShift

This Standardizes Everything

RED HAT AND CONTAINERS - CONFIDENTIAL - NDA REQUIRED

32 of 37

The Daemons

Bringing it All Together

User -> OpenShift -> Docker -> Kernel

32

Scott McCarty, Twitter: @fatherlinux

33 of 37

THE LOGIC

Bringing it All Together

33

Scott McCarty, Twitter: @fatherlinux

34 of 37

THANK YOU

plus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHatNews

35 of 37

AGENDA

10:15AM—10:25AM

INTRODUCTION

11:35AM—12:05PM

CONTAINER ORCHESTRATION

10:25AM—10:40AM

ARCHITECTURE

12:05PM—12:15PM

CONCLUSION

10:40AM—11:05AM

CONTAINER IMAGES

11:05AM—11:35PM

CONTAINER HOSTS

L103118 - Linux container internals

35

Scott McCarty, Twitter: @fatherlinux

36 of 37

Materials

The lab is made up of multiple documents and a GitHub repository

Presentation (Google Presentation): http://bit.ly/2pYAI9W
Lab Guide (this document): http://bit.ly/2mIElPG
Exercises (GitHub): http://bit.ly/2n5NtPl

36

Scott McCarty, Twitter: @fatherlinux

37 of 37

CONTACT INFORMATION

We All Love Questions

Jamie Duncan: @jamieeduncan jduncan@redhat.com
Billy Holmes: @gonoph111 biholmes@redhat.com
John Osborne: @openshiftfed josborne@redhat.com
Scott McCarty: @fatherlinux smccarty@redhat.com

37

Scott McCarty, Twitter: @fatherlinux