Spack E4S Facility Pipeline Update

Matt Belhorn (Oak Ridge National Laboratory)

Jamie Finney (Oak Ridge National Laboratory)

Frank T Willmore (Argonne National Laboratory)

Shahzeb Siddiqui (Lawrence Berkeley National Laboratory)

Aditya Kavalur (Lawrence Berkeley National Laboratory)

​

ECP Annual 2021

​

April 14^th 2021

Business Sensitive Information

1

Software Integration Team

Shahzeb Siddiqui

Aditya Kavalur

Frank Willmore

Matt Belhorn

Jamie Finney

Ryan Adamson

2.4.4.01 Software Integration

2.4.4.04 Continuous Integration

2.4.4.03 Shasta Testing

2.4.4.05 HPCM/Slingshot Testing

2

What is E4S?

• The Extreme-Scale Scientific Software (E4S) is a curated set of software packages for developing and running scientific application on high-performance computing systems. Software Products can become an E4S member package if they adhere to E4S community policies.
• E4S is distributed as container, buildcache, and spack.yaml
• The E4S Testsuite is a repository of tests for E4S packages that can be used to verify E4S stack installed at the facility.

​

3

E4S Distribution as spack.yaml

• The DOE facilities are interested in building E4S from source that is optimized for the hardware. This led to the creation of https://github.com/E4S-Project/e4s in order to distribute spack.yaml as part of the distribution.
• E4S team adopted the calendar version format as pose to semantic version for spack.yaml distribution
• E4S announced its first release v20.10 in Oct 2020 based of spack v0.15.4 built at University of Oregon.
• NERSC installed E4S 20.10 on Cori and the spack configuration is published at https://github.com/spack/spack-configs/tree/master/NERSC/cori/e4s-20.10

​

4

Facility Requirements for Installing E4S

• NERSC, OLCF and ALCF are interested in installing E4S from source
• The E4S team must provide a spack.yaml with the following specifications:
1. List of E4S packages with pinned versions (hdf5@1.12.0) for each package. Packages can’t be tied to commit, branch i.e slate@develop
2. A spack tag or commit to reference E4S build
3. All spack configuration should be defined in spack.yaml no config.yaml, modules.yaml, packages.yaml, compilers.yaml, mirrors.yaml
• The facility can pick which packages to install and their choice of compilers

​

5

Standard Naming Convention for spack.yaml

1. Specify specs in alphabetical order
2. Only specify variants for packages not available by default
3. Alphabetize packages section
4. Mirror Naming Convention in the form of <system>-e4s-<version>: cori-e4s-21.02, ascent-e4s-21.02, aurora-e4s-21.02
5. Define compilers directive in spack.yaml and not compilers.yaml
6. Have comments in spack.yaml for build error with link to CI job or package preference
7. Push spack builds to https://cdash.spack.io (optional)

​

1

2

4

5

6

3

7

6

AD / ST / SD / Facility Software Support Model

• ST Product Teams
• SDKs
• Other Spack Contributors

7

AD / ST / SD / Facility Software Ecosystem Lifecycle

ST Product Release

ST Product Release

SDK Product Release

E4S Product Release

AD/ST Team Consumption

spack / develop @ HEAD

User

Maintained

Spack Environment

‘module load e4s/20.10’

‘module load e4s’

Handwritten module

to expose a view of

a fully built e4s stack

with single toolchain

8

E4S Facility Pipeline Overview

9

NERSC Facility Pipeline Update

10

Cori E4S Develop Pipeline

E4S Pipeline

Spack Mirror Job

• The NERSC E4S pipeline is run through the batch runner cori on login nodes. The SCHEDULER_PARAMETERS specify options to Slurm scheduler that is propagated to each child job.
• We use intel/19.1.2.254 and cray-mpich/7.7.10 modules as our spack compiler wrappers. Similar to Ascent the specs are unconstrained so that we use the default version provided by Spack.

11

Cori E4S Develop Pipeline – Spack Configuration (spack.yaml)

• The NERSC E4S pipeline is run through the batch runner cori on login nodes. The SCHEDULER_PARAMETERS specify options to Slurm scheduler that is propagated to each child job.
• We use intel/19.1.2.254 and cray-mpich/7.7.10 modules as our spack compiler wrappers. Similar to Ascent the specs are unconstrained so that we use the default version provided by Spack.

12

Cori E4S Develop Pipeline – Pull Mirroring

• Cori Gitlab instance doesn’t support Gitlab pull mirroring, therefore we have a mirror.bash script to pull from upstream
• The mirror spack project https://software.nersc.gov/ecp/e4s/mirror-spack is triggered once per hour to pull changes from develop branch

Pull Mirror

13

Cori E4S 20.10 Pipeline

• Cori E4S 20.10 pipeline is based on specs + versions provided by E4S at https://github.com/E4S-Project/e4s. The pipeline is built using spack commit 1e0bbb4cbe11a3f0d7e50466ffa86071ee653b7
• We skip any packages tied to develop tag, openmpi and packages that require GPU support such as magma
• The NERSC facility pipelines push builds to CDASH build groups DOE nightly E4S builds
• The E4S job is composed of multiple gitlab jobs, all jobs are pushed to CDASH with preview of build logs.

​

​

14

Cori E4S 20.10 Spack Configuration (spack.yaml)

15

Cori E4S 20.10 – Deployment Configuration

• E4S 20.10 stack is deployed via TCL modules on Cori
• Currently we provide a 8 bit hash length to avoid module clash since there are several specs with same version
• We enable conflict on same package name this prevents user from loading two versions of same software.

​

16

Access E4S 20.10 at Cori

• E4S 20.10 can be accessed via modules and spack environment. Just load e4s/20.10 module and it will drop you into a spack environment

17

E4S User Documentation

https://docs.nersc.gov/applications/e4s/

18

Cori E4S 21.02 (In-Progress)

• We are building E4S 21.02 similar to previous release (20.10) with same compilers (intel + cray-mpich).
• Builds are pushed to CDASH
• The current spack release enables DAG-pruning which allows spack to optimize pipeline generation (spack ci generate) and avoid rebuilding software that was previously built

​

​

​

19

E4S Nightly Build Pipeline for ST and AD teams

• Objective: We wanted to provide ST and AD teams a spack user workflow to automate building ST products using gitlab infrastructure
• We stood up a nightly build of HDF5 to push to buildcache along with user documentation to install from the buildcache
• This project can be found at https://software.nersc.gov/siddiq90/hdf5-nightly-build

​

20

Gitlab Pipeline and Spack Configuration

21

Pipeline Output

22

Spack GitLab Pipeline User Documentation

https://docs.nersc.gov/applications/e4s/spack_gitlab_pipeline/

23

OLCF Facility Pipeline Update

24

Ascent E4S Pipelines

• Ascent is an 18-node stand-alone system with the same architecture as Summit.
• Used as the training system for Summit participants of OLCF training programs during workshops and conferences.
• Also hosts Continuous Testing and Integration platform, Gitlab.
• More information: Link�
• Ascent pipeline is triggered via a weekly build using Gitlab CI using the nobatch runner.

​

• Ascent develop pipeline is hosted at https://code.ornl.gov/ecpcitest/e4s.

​

• Ascent 20.10 pipeline is hosted at https://code.ornl.gov/ecpcitest/ascent-e4s.

​

​

25

Ascent E4S Develop Pipeline

• Uses gcc@8.1.1 and spectrum-mpi@10.3.1.2 as compiler wrappers for Spack

​

​

Current Package List:

​

​

26

Ascent E4S 20.10 Pipeline

• Uses gcc@8.1.1 and spectrum-mpi@10.3.1.2 as compiler wrappers for Spack v0.16

​

​

Current Package List:

​

​

27

Ascent E4S 20.10 Pipeline

28

Summit E4S Current Deployment

Current Package List:

​

​

29

Summit E4S Current Deployment

• Only exposed as individual module files currently
• E4S v20.10 minus a few exceptions due
• Build failures
• Incompatible packages
• Dependency on Slurm
• Dependency on mpich
• Against policies

​

​

​

​

30

Ascent E4S Develop Pipeline

• Ascent pipeline is triggered via Multi-Project build through a spack mirror hosted at https://code.ornl.gov/ecpcitest/spack.
• The spack mirror is configured to pick up a custom CI configuration path that is part of upstream spack. This file is located at share/spack/gitlab/ascent_pipeline.yml that was merged in #18655 and #18875

​

Pull Mirror

31

Ascent E4S Develop Pipeline

• The Ascent E4S pipeline pushes builds to CDASH at https://cdash.spack.io using build group DOE nightly E4S builds.
• We picked gcc/8.1.1 with cuda/10.1.243 based on the cuda compatibility matrix https://docs.nvidia.com/cuda/archive/10.1/cuda-installation-guide-linux/index.html. Ascent provides cuda/11.x modules however we didn’t pursue these cuda instances because POWER9 support requires RHEL 8.x while Ascent is running RHEL 7.6. Cuda 10.1.x version support RHEL 7.6 for Power 9 system.
• All packages were built without version constraints since this pipeline needs to test default package version because it is tested upon every PR to develop
• Currently, Ascent pipeline is triggered upon merge to develop branch instead of PR testing so Spack maintainers can’t view Ascent pipeline until PR is merged. For the interim, we have configured build the spack mirror (https://code.ornl.gov/ecpcitest/spack) to push build notifications to slack at #e4s-pipeline-notifications in spack slack.
• Currently, we can’t allow PR testing on Ascent therefore we intentionally setup pipeline upon merge

​

32

Ascent E4S Develop Pipeline Notification

• Currently, Ascent pipeline is triggered upon merge to develop branch instead of PR testing so Spack maintainers can’t view Ascent pipeline until PR is merged. For the interim, we have configured build the spack mirror (https://code.ornl.gov/ecpcitest/spack) to push build notifications to slack at #e4s-pipeline-notifications in spack slack.
• Currently, we can’t allow PR testing on Ascent therefore we intentionally setup pipeline upon merge

​

33

ALCF Facility Pipeline Update

ONEAPI INTEGRATION WITH SPACK at ALCF

DR FRANK T WILLMORE

​

Software Integration Admin

ALCF Operations

willmore@anl.gov

34

OVERVIEW OF ECP SOFTWARE DEPLOYMENT AT ALCF

• Software Integration
• Quarterly full deployment of E4S
• Built with OneAPI
• On early hardware (Iris, Yarrow, Arcticus)
• With CI setup for:
• automated deployment of full E4S release
• Incremental release testing
• Vendor Integration
• Current efforts center on OneAPI Integration
• Eventual Integration with Cray PE stack

​

35

35

E4S IS:

• Extreme Scale Scientific Software Stack
• Developed by Sameer Shende’s group (University of Oregon)
• A curated software bundle of many packages
• A referenceable build for a reference architecture
• CI-enhanced to rebuild stack when there are changes to:
• spack itself
• the versions of any E4S components
• the E4S build environment
• A working example of spack’s CI generate/rebuild interface
• https://github.com/E4S-Project/e4s

36

36

ONEAPI

• Intel has shifted compiler development from ‘classic’ to LLVM-based
• There are new compiler names: icx, icpx, ifx, dpcpp
• Introduces data-parallel C++ (DPCPP) which is SYCL
• The new paradigm is intended to:
• Support single-source codebase
• Make use of heterogeneous hardware (CPU, GPU, FPGA)
• This is all work-in-progress:
• Many open compiler issues
• Libraries are being re-worked, e.g. oneMKL
• Spack integration WIP

​

37

37

CHANGES TO SPACK TO ACCOMMODATE ONEAPI:

• $spack/lib/spack/spack/compilers/oneapi.py – the OneAPI compiler class in spack
• Previously, oneapi compilers were declared as either ‘clang’ or ‘intel’
• Written to clarify break from both clang and Intel classic compilers
• Understands new compiler names and how to wrap them
• Handles oneapi-specific settings
• Enables detection as ‘oneapi’
• Submitted and accepted as a pull request to spack, released in spack v0.16.0
• $spack/lib/spack/spack/build_systems/oneapi.py – the oneAPI build system
• Superclass for all oneapi packages
• Requires package.py for each of the oneAPI packages
• Includes packages for compilers, oneMKL, oneDAL, oneTBB, etc.

38

38

INSTALLING ONEAPI COMPILERS AND PACKAGES

• OneAPI compilers can be installed:
• By Intel, ad hoc, as is currently done in JLSE
• From an Intel bundle using the Intel installer
• From an Intel bundle using spack
• This is new
• Installs oneapi packages as well as modules to use them
• Module generation is currently under re-factor
• Intel-installed versions closer to final product; spack installs closer to final packaging
• Installed compilers are detectable
• OneapiPackage classes in spack:

​

39

intel-oneapi-ccl intel-oneapi-dal intel-oneapi-ipp intel-oneapi-mkl intel-oneapi-tbb

intel-oneapi-compilers intel-oneapi-dnn intel-oneapi-ippcp intel-oneapi-mpi intel-oneapi-vpl

39

USING ONEAPI IN SPACK

• icx, ifx, icpx, dpcpp – the new OneAPI compilers
• Declare a ‘oneapi’ compiler stanza in any of the spack scopes
• Detect the compiler using spack compiler find to generate a compiler stanza
• oneMKL, oneDAL, oneTBB, etc. – the OneAPI packages
• A package can satisfy a consuming dependency directly, via a ‘requires’ directive in the dependent package
• Can satisfy a virtual dependency, e.g. intel-oneapi-mpi is a provider of mpi
• Packages can also be placed in environment outside of spack via modules
• We care about this use case
• Not everything needs to be built with spack
• sometimes we just want a compiler or a library

40

40

SYSTEM COMPATIBILITY LAYER

• A set of packages
• Defined as a spack environment
• Concretizes together
• Built with GCC for generic x86_64 target
• Packages need not be performant
• Newer versions of packages typically provided by OS vendor
• Once installed, can be used to generate a packages stanza or inlined to an environment
• Generated stanza can be included for use by another environment (e.g. E4S)
• Designed to support quarterly updates of E4S
• not needed if building E4S with GCC

​

​

​

41

41

SYSTEM COMPATIBILITY LAYER

• OneapiPackage classes:
• intel-oneapi-compilers
• intel-oneapi-mpi
• intel-oneapi-mkl
• intel-oneapi-tbb
• etc.
• Declare a packages stanza:

​

42

spack:

view: false

concretization: together

definitions:

- gcc_version: ['%gcc@10.2.0’]

- with_gcc:

- autoconf@2.70

- automake@1.16.2

- berkeley-db@18.1.40

- binutils@2.33.1+gold+headers+libiberty~nls

- boost@1.73.0 visibility=global cxxstd=17

- bzip2@1.0.8

- cmake@3.17.3

- diffutils@3.7

- elfutils+bzip2~nls+xz ^m4 ^zlib+shared

- expat@2.2.9

- gdbm@1.18.1

- gettext@0.20.2

…

- libsigsegv@2.12

- m4@1.4.18

- ncurses@6.2

- perl@5.30.3

- python@3.7.8

- readline@8.0

- sqlite@3.31.1

- tar@1.32

- xz@5.2.5

- zlib@1.2.11

42

SPACK CI STAGES: THE BUILD PIPELINE

• Generate stage
• Runs on login node using shell runner
• Takes a spack environment (e.g. E4S) as input
• Generates CI jobs for one or more ordered sets of specs to be built
• One spec will be built per generated CI job
• Rebuild stage
• Run on compute nodes using batch runner
• Each job builds one spec:
• Will search spack mirrors for same hash as the requested spec
• If built spec exists and is current, spack will move on
• If built spec does not exist, will build the spec and push it to the mirror

​

​

​

​

​

43

43

.GITLAB-CI.YAML

variables:

JLSE_YARROW_SCHEDULER_PARAMETERS: "-A Operations -n 1 -t 30 -q yarrow"

SPACK_REPO: /soft/spack

SPACK_REF: develop

generate_spack_ci:

stage: build

artifacts:

paths:

- .spack-ci.yml

tags:

- jlselogin3_shell-01

​

script:

- . "select_spack_instance.sh"

- spack env activate .

- spack config blame packages

- spack gpg list

- spack ci generate --output-file ./.spack-ci.yml

run_rebuild_ci:

stage: test

trigger:

include:

- artifact: .spack-ci.yml

job: generate_spack_ci

strategy: depend

44

GITLAB-CI STANZA IN SPACK.YAML

• Specifies how rebuild jobs will run
• content used when generating each CI rebuild job

spack:

…

gitlab-ci:

mappings:

- match: [os=opensuse_leap15]

runner-attributes:

tags:

- yarrow

variables:

SPACK_REPO: $env:SPACK_REPO

SPACK_REF: $env:SPACK_REF

SPACK_GNUPGHOME: /soft/spack-mirror/.gnupg

before_script:

- . "select_spack_instance.sh"

script:

- spack env activate .

- spack gpg list

- spack -d ci rebuild

- cd /soft/spack-mirror && spack buildcache update-index

…

mirrors:

spack-mirror: file:///soft/spack-mirror

45

SPACK CI BUILD OF PETSC

46

SPACK CI SUMMARY

47

Generate staged jobs

deployment

mirrors

software

modules

Repo/environment in gitlab:

​

- spack.yaml�- .gitlab-ci.yaml

Rebuild any new or altered specs not already in mirror

Triggered by commit

47

SUMMARY OF DEPLOYMENT PIPELINE AT ALCF

• Spack now integrates with OneAPI both as a provider and consumer of compilers and libraries
• ECP leadership has selected E4S as a vehicle for deployment of software
• It is supplied as a spack environment and released quarterly
• Instrument-specific system layer
• Specified as a spack environment
• Concretized together
• Serves as platform for higher-level dependent builds
• Automatically generates a spack packages stanza
• Modules are created by spack as part of deployment process, can then be curated
• Spack CI automates the building of an environment, as well as the re-building of specs as required by a change to the environment

48

48

Iris/Yarrow Spack Environment

• Spack environment on Iris & Yarrow are built using oneapi compiler using oneapi version 2020.09.15.001.
• To activate spack environment you can run source /soft/spack/share/spack/setup-env.sh
• Intel provides periodic shipment of oneapi version with bug fixes until stable release is announced.
• Spack support for oneapi compiler was added in v0.16.0 release

​

NDA Material

49

Challenges

• E4S Develop pipeline is triggered upon merge to develop, ideally we need pipeline to trigger on PR and report CI check back to PR in GitHub. Building of PR is not feasible on production system (Cori, Ascent) due to security concerns
• It’s not clear when to enable build from source vs build from buildcache for E4S Develop pipeline.
• GitLab Pull mirroring feature is required to run facility pipeline, Cori Gitlab instance doesn’t have this feature it requires Starter feature. Pull mirroring will be more complex when its PR from Forks
• Need to determine # of pipeline builds for PR pipeline test. Ideally we should have one pipeline build per PR, subsequent commit to same PR should not trigger build but can be run manually.
• Curation of deployed software and modules generated by spack
• Maturity of oneAPI compiler

50

Upcoming Work

NERSC

• Build E4S 21.02 or 21.05 for Perlmutter
• Build E4S 21.01 on Cori (In-Progress)
• Improve User Documentation for E4S page
• Increase test coverage for facility deployed E4S stack

OLCF

• Build and Deploy E4S 20.10 for Ascent and/or Summit for RHEL 7
• RHEL 8 Upgrade for Summit is delayed until after Gordon Bell work is completed
• Include Spack test in pipelines

ALCF

• Build E4S 21.02 pipeline using oneAPI compiler on Yarrow, ATS.
• Requires a stable release of oneAPI from Intel and new release from E4S team
• Migrating Gitlab runner to OpenSUSE login nodes
• External package support for oneAPI, one-mkl, one-dal, etc.
• Working through compiler tics for E4S builds

Roadblocks

• GitLab Premium support is required for CI/CD to external repos in order to push gitlab builds to spack PR. NERSC and OLCF Gitlab instance are not Premium support
• We need to a build node (Frontier, Aurora) to initiate Spack PR pipeline.

​

51

Q/A

• POC:
• Shahzeb Siddiqui - shahzebsiddiqui@lbl.gov
• Aditya Kavalur – akavalur@lbl.gov
• Jamie Finney - finneyjm@ornl.gov
• Matt Belhorn – belhornmp@ornl.gov
• Frank Wilmore – willmore@anl.gov
• Join the spack #e4s channel for support with E4S
• 2.4.4.01 Confluence Page: https://confluence.exascaleproject.org/display/HISD01

​

​

52

ACKNOWLEDGEMENT

This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of two U.S. Department of Energy organizations (Office of Science and the National Nuclear Security Administration) responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, in support of the nation’s exascale computing imperative.

53