1 of 34

Building Smaller Container Images

...and more™

Adam Samalik

Senior Software Engineer

Scott McCarty

Principal Product Manager

CONFIDENTIAL Designator

Abstract

Red Hat helped pioneer the concept of a Linux distribution, now we're working on how to get rid of it... OK, you're paying attention, right? Technically, we're trying to shift the problem of managing the distro to Red Hat so that you don't have to worry about it. Distroless container images can be tiny because they don't include package management tools like RPM and DNF, but still provide users needed software like glibc, openssl, and httpd. Cloud is just somebody else's computer, Serverless is just somebody else's server, and Distroless is just somebody else's Linux distro.

In this talk, we'll analyze the dependencies in container images using rpm-show, and explain how work is constantly being done to make container images smaller while still providing battle tested, pre-built software that is convenient to consume.

https://devconfcz2021.sched.com/event/gmSS/building-smaller-container-images

Source Material

2 of 34

Analysis

SERIOUS ANALYSIS

We started off with complaints that container images were too large, but analyzed the entire problem set from end to end.

3 of 34

What does a user really want when they ask for smaller container images?

ANALYSIS

Less Attack Surface

They want less total software in their environment which can be attacked.

Smaller Images

They believe this gives them less attack surface, but it’s a trap! Having 1000 different 5MB images is still 5GB of attack surface.

No Thinking

The user doesn’t want to have to think about attack surface. They want it built into the tools they are using.

Cool Tweets

They want to post on twitter how small their container images are. This will get earn them overt prestige among their peers!

Twitter: @adsamalik @fatherlinux

4 of 34

Things we looked at and tried

ANALYSIS

Scratch Container Builds

In the Container world, this is different than in the packaging world. Started with an empty directory and a package manager:

yum install --installroot /mnt/rootfs

Distroless

The claim is that smaller is better for security, but you’re pulling random binaries from The Internet

Minimizing dependencies

This innovation happens in Fedora, and will be inherited in RHEL.

Compile from scratch

Instead of leveraging packages from distributions

Less container images

A bunch of small images can still add to be larger than a few mid-sized images

Twitter: @adsamalik @fatherlinux

5 of 34

Scratch Container Builds

ANALYSIS

Twitter: @adsamalik @fatherlinux

6 of 34

Minimizing Dependencies

ANALYSIS

Twitter: @adsamalik @fatherlinux

7 of 34

Distroless

ANALYSIS

Cletis Hykes

Unemployed

Twitter: @adsamalik @fatherlinux

8 of 34

Compile From Scratch

ANALYSIS

Twitter: @adsamalik @fatherlinux

9 of 34

Less container images

BACKGROUND

Twitter: @adsamalik @fatherlinux

10 of 34

What if we could have the best?

BACKGROUND

High Quality Dependency Tree

Leverage existing linux distributions like Fedora/RHEL

Use Existing Tools

Use RPM and YUM, but not in the container

Minimize Dependencies Within Distro

Minimize hard dependencies within the Linux distribution

Single Linux Distribution

Minimize total attack surface within your organization

Smaller Images

Launched Red Hat Universal Base Image for RHEL 8 and RHEL 7

Twitter: @adsamalik @fatherlinux

11 of 34

Building

Smaller Images

12 of 34

examples are using Fedora

that's where the innovation happens

Twitter: @adsamalik @fatherlinux

13 of 34

Multi-stage builds

Main idea:

When you buy a car, you only get the car.

They won't ship the assembly line with it.

Our application:

Package manager is the assembly line.

Twitter: @adsamalik @fatherlinux

14 of 34

The package manager in Fedora (DNF) can:

install into an empty directory from scratch
disable weak dependencies
disable documentation packages

Twitter: @adsamalik @fatherlinux

15 of 34

Fedora package ecosystem

has alternatives, such as:

glibc-minimal-langpack coreutils-single

Twitter: @adsamalik @fatherlinux

16 of 34

Building out a Dockerfile

Twitter: @adsamalik @fatherlinux

17 of 34

Building out a Dockerfile: nginx

FROM fedora:33

RUN dnf -y install nginx && \

dnf -y clean all

# ...

224 MB

Twitter: @adsamalik @fatherlinux

18 of 34

Building out a Dockerfile: nginx

FROM fedora:33 as build-env

RUN dnf -y --installroot /output --releasever 33 install nginx && \

dnf -y --installroot /output --releasever 33 clean all

FROM scratch

COPY --from=build-env /output /

# ...

450 MB (oh no!)

Twitter: @adsamalik @fatherlinux

19 of 34

Building out a Dockerfile: nginx

FROM fedora:33 as build-env

RUN dnf -y --setopt=install_weak_deps=false --nodocs \

--installroot /output --releasever 33 \

install glibc-minimal-langpack coreutils-single nginx && \

dnf -y --installroot /output --releasever 33 clean all

FROM scratch

COPY --from=build-env /output /

# ...

116 MB (oh yes!)

Twitter: @adsamalik @fatherlinux

20 of 34

FROM fedora:33

RUN dnf -y install nginx && \

dnf -y clean all

# ...

FROM fedora:33 as build-env

RUN mkdir /output

RUN dnf -y --installroot /output --releasever 33 \

--setopt=install_weak_deps=false --nodocs \

install glibc-minimal-langpack coreutils-single nginx && \

dnf -y --installroot /output --releasever 33 clean all

FROM scratch

COPY --from=build-env /output /

# ...

224 MB

116 MB

With a package manager

Without a package manager + a few other tricks

Twitter: @adsamalik @fatherlinux

21 of 34

Other example: "distroles" base

FROM fedora:33 as build-env

RUN dnf -y --setopt=install_weak_deps=false --nodocs \

--installroot /output --releasever 33 \

install glibc bash glibc-minimal-langpack coreutils-single && \

dnf -y --installroot /output --releasever 33 clean all

FROM scratch

COPY --from=build-env /output /

ENTRYPOINT /bin/bash

38MB

Twitter: @adsamalik @fatherlinux

22 of 34

Other example: busybox "distroless" base

FROM fedora:33 as build-env

RUN dnf -y --setopt=install_weak_deps=false --nodocs \

--installroot /output --releasever 33 \

install busybox && \

dnf -y --installroot /output --releasever 33 clean all

RUN mkdir /output/bin

RUN for i in $(/output/sbin/busybox --list); do ln -s /sbin/busybox \

/output/bin/$i; done

FROM scratch

COPY --from=build-env /output /

ENTRYPOINT /bin/sh

1.8MB

Twitter: @adsamalik @fatherlinux

23 of 34

Minimizing

Dependencies

24 of 34

Dependencies are complex!

Image shows Fedora repository

25 of 34

Minimization

is one of Fedora's main objectives.

26 of 34

f32

f33

Twitter: @adsamalik @fatherlinux

27 of 34

From https://tiny.distro.builders

28 of 34

From https://tiny.distro.builders

29 of 34

From https://tiny.distro.builders

30 of 34

From https://tiny.distro.builders

31 of 34

https://tiny.distro.builders

32 of 34

Conclusions

OPTIONAL SECTION MARKER OR TITLE

It’s important to use a distro and a package manager when using distroless distros :-P

33 of 34

Conclusions

OPTIONAL SECTION MARKER OR TITLE

Red Hat Universal Base Image (RHEL 7, 8, and 9)

Launched Red Hat Universal Base Image for RHEL 8 and RHEL 7. For production, we think standardizing on UBI makes sense to reduce total attack surface in an environment.

Distroless Images

Yes and no. We still think it makes sense to use the packages with the distro, but not include the package manager (RPM) and dependency manager (DNF) in the resultant container image

Minimal Dependency Tree

The work Adam and many teams are doing in Fedora will land in RHEL 8 and RHEL 9

Smaller Attack Surface

Combining some tricky build techniques (distroless/scratch), a minimized dependency tree, and high quality package content with rich security metadata is what it truly takes

Twitter: @adsamalik @fatherlinux

34 of 34

Thank you

Red Hat is the world’s leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make �Red Hat a trusted adviser to the Fortune 500.

CONFIDENTIAL Designator

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHat