Conda-forge: the boomerang meme

conda-forge stats

• ~7,500 packages
• ~1,500 contributors
• Supported platforms:
• Linux
• Windows
• Mac OS X
• ARM^new
• PowerPC^new
• Python, R, C, C++, FORTRAN, Go, Rust, Haskell, etc.

Complete dependency graph

• 7,500 nodes (packages)
• 49.476 edges (dependency relationships)

Conda-forge goals over the past year

• One stop shop for packages (including for science)
• Easier maintenance for package authors*
• Interoperability with packages provided by other channels*
• Streamline continuous integration (CI) services*

​

* These are part of the ESIP Lab incubator.

Easier maintenance with auto-tick bot

• Scans several sources for new releases
• Opens pull request to update
• Package version
• Update feedstock boilerplate
• Does not update metadata (yet)

Towards interoperability

• Adopt conda-build 3
• Python “noarch” packages: one package for all the Pythons and platforms
• Multiple outputs from one build recipe
• Better abstraction of toolchain and requirements
• Upgraded compiler toolchain to more modern compilers: rebuild almost everything, in the right order
• Solution: improved auto-tick bot that
• Determines which packages need rebuilding
• Determines build order
• Opens pull requests in proper order

Towards streamlined CI usage

​

• Lag from opening or merging pull request to completed build and tests
• Potential cost for large rebuilds
• Limited build capacity
• Maintenance burden for three CI services��
• Solution: Azure pipelines

Coming!

• Additional optimization of conda
• Package size reduction
• 1.8G current total pangeo env size
• ~295 MB (16%) is due to static libraries that are not used!
• GPU packages
• More packages for Aarch64/PPC64LE
• + Python 3.8
• - Python2.7
• No packages will be removed
• New conda-forge builds for Python 2.7 will end
• More automation to update package metadata