1 of 39

DO LINUX DISTRIBUTIONS STILL MATTER WITH CONTAINERS?

Balancing the value of innovation & maintenance

Scott McCarty

Principal Offering Product Manager (popm)

02/01/2020

2 of 39

2

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

3 of 39

3

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

4 of 39

“I don’t want to care about the operating system anymore”

4

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

5 of 39

Let’s use tires as an analogy...

5

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

6 of 39

6

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

7 of 39

7

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

8 of 39

8

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

9 of 39

9

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

10 of 39

10

Safety

Road Performance

Amature Racing

Professional Racing

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

11 of 39

OK, so we do still care. But, what criteria? What context?

11

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

12 of 39

UNDERSTANDING THE CRITERIA�WITH CONTAINERS

13 of 39

THERE ARE A LOT OF DIFFERENT OPTIONS

Figuring out which container base image to use can be difficult

Traditional Options

Red Hat Enterprise Linux
Fedora
CentOS
Debian
Ubuntu
Windows

Minimal Options

Distroless
Scratch
RHEL Minimal
Alpine

13

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

14 of 39

There is no cloud!

Just someone else’s computer

14

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

15 of 39

There is no distroless!

Just another dependency you manage

15

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

16 of 39

HOW TO SELECT THE RIGHT IMAGE

There is some standard criteria that can help

Architecture

C Library
Core Utilities
Size
Life Cycle
Compatibility
Troubleshooting
Technical Support
ISV Support
Distributability

Security

Updates
Tracking
Security Response Team

Performance

Automated
Performance Engineering

16

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

17 of 39

HOW DO THINGS WORK?

18 of 39

IT ALL STARTS WITH COMPILING

Statically linking everything into the binary

Starting with the basics:

Programs rely on libraries
Especially things like SSL - difficult to reimplement in for example PHP
Math libraries are also common
Libraries can be compiled into binaries - called static linking
Example: C code + glibc + gcc = program

18

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

Points

Think about:

Compatibility
Size of the ecosystem
How hard libraries are to code natively
Having multiple copies of the same libraries laying around, needing patched

Choosing the Right C Library Matter Because:

Porting is the only way to move between C libraries is difficult (glibc, musl, dietlibc, newlib, ulibc). The following are concerns:

Code Changes - Source level compatibility - recompilation problems - Applications are rarely written in clean ISO C, and so they have dependencies on whatever you built with and whatever you tested on.
Feature Parity - there is not feature parity between C libraries.

Examples:

Locales: Muslc only implements UTF-8, while glibc used in RHEL can provide localizations for all sorts of encoded customer data e.g. Japanese SHIFT-JIS, IBM903, etc.
Symbol versioning: musl does not implement all of the GNU symbol versioning specification, and so some behaviours which customers expect are undefined or unsupported
Syscalls: Likewise musl does not support dlclose() on ideological grounds that it causes problems, but glibc can and does support it to allow plugins to be unloaded (when written properly).

ABI Compatibility - Your existing compiled programs probably won’t run without the library it was compiled with.

There are runtime differences between all the "standard C libraries" and none of them offers binary compatibility with the other.
If you rely on 3rd party ISVs for shared objections, or pre-compiled objects then you will need to use whatever "C library" is compatible with their compiled objects. Generally this is "glibc" because it's been in active use and development for 30+ years: https://sourceware.org/glibc/wiki/Glibc%20Timeline

Generally, you should use the C library that comes with a Linux distro. There’s no technical reason why you “couldn’t” use another C libraries, but it will cost you:

You will have to patch your glibc to stay up to date with our glibc and the changes going into the kernel (this kind of integration is common in Fedora/Red Hat Enterprise Linux).

The RHEL kernel team backports new features all the time. Those new features sometimes can enable experimental support for something in glibc (since it detects the new kernel), and that support then fails. Red Hat then has to go fix it in coordination with the RHEL kernel team and release together. If you aren't tracking RHEL kernel development your glibc-in-the-container may have an API that stops working properly after the kernel update. This is common between hardware platforms x86, ARM, PPC, s390x
Examples:

gettimeofday() for ppc64
send() function for s390x (most recently)

19 of 39

LEADS TO DEPENDENCIES

Dynamically linking libraries into the binary

Getting more advanced:

This is convenient because programs can now share libraries
Requires a dynamic linker
Requires the kernel to understand where to find this linker at runtime
Not terribly different than interpreters (hence the operating system is called an interpretive layer)

19

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

20 of 39

PACKAGING & DEPENDENCIES

RPM and Yum were invented a long time ago

Dependencies need resolvers:

Humans have to create the dependency tree when packaging
Computers have to resolve the dependency tree at install time (container image build)
This is essentially what a Linux distribution does sans the installer (container image)

20

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

21 of 39

PACKAGING & DEPENDENCIES

Interpreters have to handle the same problems

Dependencies need resolvers:

Humans have to create the dependency tree when packaging
Computers have to resolve the dependency tree at install time (container image build)
Python, Ruby, Node.js, and most other interpreted languages rely on C libraries for difficult tasks (ex. SSL)

21

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

22 of 39

CONTAINER IMAGE PARTS

Governed by the OCI image specification standard

Lots of payload media types:

Image Index/Manifest.json - provide index of image layers
Image layers provide change sets - adds/deletes of files
Config.json provides command line options, environment variables, time created, and much more
Not actually single images, really repositories of image layers

22

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

23 of 39

LAYERS ARE CHANGE SETS

Each layer has adds/deletes

Each image layer is a permutation in time:

Different files can be added, updated or deleted with each change set
Still relies on package management for dependency resolution
Still relies on dynamic linking at runtime

23

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

24 of 39

LAYERS ARE CHANGE SETS

Some layers are given a human readable name

Each image layer is a permutation in time:

Different files can be added, updated or deleted with each change set
Still relies on package management for dependency resolution
Still relies on dynamic linking at runtime

24

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

25 of 39

CONTAINER IMAGES & USER OPTIONS

Come with default binaries to start, environment variables, etc

Each image layer is a permutation in time:

Different files can be added, updated or deleted with each change set
Still relies on package management for dependency resolution
Still relies on dynamic linking at runtime

25

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

26 of 39

INTER REPOSITORY DEPENDENCIES

Think through this problem as well

You have to build this dependency tree yourself:

DRY - Do not repeat yourself. Very similar to functions and coding
OpenShift BuildConfigs and DeploymentConfigs can help
Letting every development team embed their own libraries takes you back to the 90s

26

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

27 of 39

CONTAINER IMAGE

Open source code/libraries, in a Linux distribution, in a tarball

Even base images are made up of layers:

Libraries (glibc, libssl)
Binaries (httpd)
Packages (rpms)
Dependency Management (yum)
Repositories (rhel7)
Image Layer & Tags (rhel7:7.5-404)
At scale, across teams of developers and CI/CD systems, consider all of the necessary technology

27

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

28 of 39

UNDERSTANDING THE CONTEXT�WITH CONTAINERS

29 of 39

IT WORKS ON MY LAPTOP, AND...

From an architecture perspective

29

CONTAINER

GENERIC CONTAINER IMAGE

LANGUAGE RUNTIMES

APP

PRODUCTION

CONTAINER

GENERIC CONTAINER IMAGE

LANGUAGE RUNTIMES

APP

LAPTOP

Works on my laptop

The service starts in production

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

30 of 39

IT WORKS ON MY LAPTOP, BUT...

What about performance?

30

CONTAINER

GENERIC CONTAINER IMAGE

LANGUAGE RUNTIMES

APP

PRODUCTION

CONTAINER

GENERIC CONTAINER IMAGE

LANGUAGE RUNTIMES

APP

LAPTOP

Works on my laptop

But, what about at 1M TPS

1M �Transactions/

Second

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

31 of 39

IT WORKS ON MY LAPTOP, BUT...

What about security?

31

CONTAINER

GENERIC CONTAINER IMAGE

LANGUAGE RUNTIMES

APP

PRODUCTION

CONTAINER

GENERIC CONTAINER IMAGE

LANGUAGE RUNTIMES

APP

LAPTOP

Works on my laptop

What about hackers?

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

32 of 39

THE QUALITY OF THE BITS MATTERS

It goes beyond works on my laptop

32

CONTAINER

OS

(USER SPACE)

LANGUAGE RUNTIMES

APP

OS

(USER SPACE)

OS

(USER SPACE)

OS

(USER SPACE)

OS

(USER SPACE)

OS

(USER SPACE)

Compute Intensive

I/O Intensive

Fault Sensitive

Highly Secure

Latency Sensitive

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

33 of 39

WHAT IS THE RED HAT UNIVERSAL BASE IMAGE?

Three base images, language runtime images, and software packages

33

ALL RED HAT ENTERPRISE LINUX PACKAGES

UNIVERSAL�BASE IMAGE

PACKAGES

Base

Images

Pre-Built Language

Images

Package�Subset

STANDARD

MULTI�SERVICE

MINIMAL

Node.js

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

Base Images - Derived from RHEL, the Red Hat Universal Base image provides three main base images:

ubi - a standard base image built on enterprise grade packages from RHEL. Good for 80% of user’s needs
ubi-minimal - when users are sensitive to size, this is the image to use. Built on microdnf, it provides the smallest on disk size
ubi-init - this image allows users to easily run multiple services (ex. mysql, httpd) in a single container. It allows a user to leverage the knowledge built into systemd unit files without having to figure out how to start the service

Language Runtime Images

These are ready to run container images, a user only needs to add their application code. A set of language runtime images is provided for UBI7 and UBI8
UBI7 - php-72, nodejs-8, ruby-25, python-27, python-36 - update to public list on Red Hat portal TBD
UBI8 - dotnet-21-runtime, perl-526, php-72, nodejs-10, ruby-25, python-27, python-36 - update to public list on Red Hat portal TBD

YUM updates

A dedicated YUM repository with access to the latest RPMs is provided so that users can rebuild container images anytime they want, picking up all of the latest changes
By default, only the latest RPM for each package will be provided
If users need access to historic RPMs, they will need to synchronize them themselves

34 of 39

RECOMMENDATIONS

35 of 39

PEOPLE DON’T UNDERSTAND THE VALUE

This is the fundamental problem

35

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

36 of 39

CALL TO ACTION

All Linux distros need to think about market problems

Recommendations:

Educate people on current value: dependency tree which provides a catalog of software and libraries
Create new value: smaller images, environment variables to configure software inside, sane defaults, new optimized security, optimized tooling, meta-data

36

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

Points

Think about:

Compatibility
Size of the ecosystem
How hard libraries are to code natively
Having multiple copies of the same libraries laying around, needing patched

Choosing the Right C Library Matter Because:

Porting is the only way to move between C libraries is difficult (glibc, musl, dietlibc, newlib, ulibc). The following are concerns:

Code Changes - Source level compatibility - recompilation problems - Applications are rarely written in clean ISO C, and so they have dependencies on whatever you built with and whatever you tested on.
Feature Parity - there is not feature parity between C libraries.

Examples:

Locales: Muslc only implements UTF-8, while glibc used in RHEL can provide localizations for all sorts of encoded customer data e.g. Japanese SHIFT-JIS, IBM903, etc.
Symbol versioning: musl does not implement all of the GNU symbol versioning specification, and so some behaviours which customers expect are undefined or unsupported
Syscalls: Likewise musl does not support dlclose() on ideological grounds that it causes problems, but glibc can and does support it to allow plugins to be unloaded (when written properly).

ABI Compatibility - Your existing compiled programs probably won’t run without the library it was compiled with.

There are runtime differences between all the "standard C libraries" and none of them offers binary compatibility with the other.
If you rely on 3rd party ISVs for shared objections, or pre-compiled objects then you will need to use whatever "C library" is compatible with their compiled objects. Generally this is "glibc" because it's been in active use and development for 30+ years: https://sourceware.org/glibc/wiki/Glibc%20Timeline

Generally, you should use the C library that comes with a Linux distro. There’s no technical reason why you “couldn’t” use another C libraries, but it will cost you:

You will have to patch your glibc to stay up to date with our glibc and the changes going into the kernel (this kind of integration is common in Fedora/Red Hat Enterprise Linux).

The RHEL kernel team backports new features all the time. Those new features sometimes can enable experimental support for something in glibc (since it detects the new kernel), and that support then fails. Red Hat then has to go fix it in coordination with the RHEL kernel team and release together. If you aren't tracking RHEL kernel development your glibc-in-the-container may have an API that stops working properly after the kernel update. This is common between hardware platforms x86, ARM, PPC, s390x
Examples:

gettimeofday() for ppc64
send() function for s390x (most recently)

37 of 39

QUESTIONS?

Citations

37

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux

38 of 39

THANK YOU

plus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHat

39 of 39

SPECIFICALLY CONTAINER IMAGES

This is the fundamental problem

39

Scott McCarty - Product Manager Container at Red Hat - Twitter: @fatherlinux