1 of 21

distri

package managers are too slow

Michael Stapelberg

@zekjur�2019-11-27

2 of 21

debian

distri

demo: installing “ack”

3 of 21

debian

distri

demo: installing “qemu”

4 of 21

package manager speed: install “ack” (Perl)*

rate = data ÷ wall-clock time�* standard installation, includes metadata & package download and dependencies�→ https://michael.stapelberg.ch/posts/2019-08-17-linux-package-managers-are-slow/

distribution

package manager

data

wall-clock time

rate

Fedora

dnf

98 MB

25s

3.92 MBps

NixOS

Nix

15 MB

12s

1.25 MBps

Debian

apt

16 MB

10s

1.6 MBps

Arch Linux

pacman

6.4 MB

1s

6.4 MBps

Alpine

apk

41 MB

0.5s

82 MBps

​

5 of 21

Why are package managers slow?

  • 2 most widely used package formats:
    • deb (Debian package), tar(1) in ar(1)
    • rpm (Red Hat Package Manager), metadata around cpio(1)�
  • task: make package contents available�→ e.g. apt install nginx results in /usr/bin/nginx�
  • traditionally: resolve deps, download, extract, configure�→ need to carefully fsync(2) to make I/O as safe as possible

6 of 21

How can we go faster?

7 of 21

append-only package store of immutable images

  1. use an image format (e.g. SquashFS) instead of an archive format�
  2. mount each image under its own path (“separate hierarchies”):�e.g. /ro/nginx-amd64-1.14.1/…�e.g. /ro/zsh-amd64-5.6.2/…�
  3. (rest of the system as usual, e.g. /etc, /var/cache, …)

8 of 21

advantages

  • mount instead of extract�→ faster package installation�→ faster build environment composition�
  • append-only: can use unsafe I/O�
  • immutable: no longer possible to screw up your installation

9 of 21

hermetic packages

  • when run, use the same version of dependencies as when built�
  • a wrapper script sets e.g. LD_LIBRARY_PATH, PYTHONPATH, PERL5LIB, …�

​

10 of 21

separate hierarchies: exchange dirs

  • packages exchange data via directories with well-known paths, e.g.:�man(1) ⟷ nginx(1) via /usr/share/man�gcc(1) ⟷ libusb(3) via /usr/include�
  • prudent approach: emulate well-known paths �e.g.:�/usr/include/jpeglib.h is a symlink to /ro/libjpeg-turbo-amd64-2.0.0/out/include/jpeglib.h

11 of 21

separate hierarchies: exchange dirs (per package)

  • loose coupling (global) vs. tight coupling (per package)�→ typically suitable for plugin mechanisms where ABI must match�
  • e.g. /ro/xorg-server-amd64-1.20.3/out/lib/xorg/modules/

12 of 21

separate hierarchies: advantages

  • move conflicts from package installation to program execution�→ only need to resolve /bin/python (2.7 or 3?) when assembling /bin�
  • packages always co-installable�e.g. zsh-amd64-5.6.2 and zsh-amd64-5.6.3�→ partial updates/rollbacks easily possible�
  • package manager can be version-agnostic!�→ entirely eliminates a large source of slowness�→ no need for global metadata, package-specific metadata sufficient

13 of 21

immutability

  • package contents and exchange dirs are read-only�
  • rarely, programs expect the system to be writable�e.g. GNOME’s gsettings wants a cache in the exchange directory�
  • such designs need to be improved upstream:�1. good caches are not required (fallback to slow path)�2. good caches are transparently created�3. good caches are automatically updated when needed

14 of 21

no hooks/triggers (1)

  • hook (or maintscript, postinst, …): program run after package installation�trigger: program run after other package installation (e.g. man-db)�→ work at package-installation time which may be unnecessary�
  • preclude concurrent package installation�(not implemented concurrency-safe)�
  • arbitrary code, can be slow�

15 of 21

no hooks/triggers (2)

  • claim: we can build a functioning system without hooks/triggers�
  • 1. packages declare what they need (e.g. sysusers)�
  • 2. move work from package installation to program execution�e.g. ssh needs a hostkey: create it in sshd(8) wrapper script�
  • very few exceptions: bootloader or firmware�(need to install them outside of the file system)

16 of 21

practicality

  • FUSE file system for providing /ro�→ easier to implement than managing separate mounts, overlays, unions, …�→ faster (!), as kernel mounts are slow�
  • packages need to be built with --prefix=/ro/nginx-amd64-1.14.1 etc.�
  • a small number of packages need to be patched�→ path-related issues (e.g. service files, gcc, gobject, automake, …)�→ deep system integration (e.g. dracut)

17 of 21

practicality (2)

  • removal of hooks is not for everyone�→ configuration layers (debconf, YAST, …) might be a feature to some

18 of 21

Why is distri faster?

  • traditionally: resolve deps, download, extract, configure�+ careful fsync(2) to make I/O as safe as possible�
  • distri: resolve deps, download image, extract, configure�(unsafe I/O okay)�→ scales to 12+ Gbit/s (!) on fast links using Go’s net/http

19 of 21

conclusion

  • append-only package stores are more elegant than mutable systems�→ simpler design, faster implementation�
  • exchange directories make things seem normal to third-party software�→ can compile unpackaged software, can run closed-source binaries�
  • all of these ideas are practical�→ live CDs (read-only) and cross-compilation paved the way

20 of 21

project goals

  • Not trying to build a community or user base!��
  • Instead, distri enables (my) Linux distribution research,�with regular proof of concept releases�

21 of 21

Thanks! Questions?