2 of 80

Our Journey begins in 2020

The ECP Software Integration (SI) Team comprised of NERSC, OLCF and ALCF collaborated with E4S Team to determine how facilities will install E4S.
Some of the key discussion points we discussed:

What packages will be available in E4S?
What version of spack to use?
Release E4S as a spack configuration (spack.yaml) that can be used by facilities
Facility are interested in deploying E4S from source and not containers

This led to the creation of E4S project (https://github.com/E4S-Project/e4s) that provides facilities with a spack release in the form of spack.yaml that will be consumed by facilities.
We proposed E4S change their release from semantic version (v1.0) → calendar version (YY.MM) see https://github.com/E4S-Project/e4s/issues/2.
E4S has a total of 14 releases with latest version being 23.11.
The spack configuration for E4S can be found at https://github.com/E4S-Project/e4s/tree/master/environments

3 of 80

Team Members

ECP Software Integration Team:

Shahzeb Siddiqui (NERSC)
Erik Palmer (NERSC)
Justin Cook (NERSC)
Sean Koyama (ANL)
Jamie Finnie (ALCF)
Matt Belhorn (ALCF)

E4S Team:

Sameer Shende
Wyatt Spear
Luke Peyralans

External Contributions

Zack Galbreth (Kitware)
Pieter Ghysels (Strumpack)
Yang Liu (SuperLU)
Jonathon Anderson (HPCToolkit)
Wileam Phan (HPCToolkit)
Asim Yarkhan (Slate)
John-Luke Navarro (LibEnsemble)

4 of 80

ECP Software Integration Team

Shahzeb Siddiqui

Erik Palmer

Justin Cook

Matt Belhorn

Jamie Finney

Ryan Prout

2.4.4.01 Software Integration

2.4.4.04 Continuous Integration

2.4.4.03 Shasta Testing

2.4.4.05 HPCM/Slingshot Testing

Sean Koyama

5 of 80

E4S Deployment Timeline

Dec 2023�23.08 on Perlmutter

Aug 2023�23.05 on Perlmutter

October 2020�First release of E4S: 20.10�20.10 on Summit

January 2021�20.10 on Cori

January 2022�21.11 on Perlmutter

21.02

21.05

21.08

22.02

22.11

August 2021�21.05 on Arcticus�21.05 on Summit

November 2021�21.08 on Arcticus

December 2021�21.11 on Arcticus

June 2021�21.02 on Cori

October 2021�21.05 on Cori�21.08 on Summit

March 2022�22.02 on Cori

July 2022�22.05 on Perlmutter�22.05 on Summit�22.05 on Arcticus

September 2022�22.08 on Arcticus

May 2022�22.02 on Crusher, Frontier, Summit

August 2022�22.05 on Crusher, Frontier

22.08

21.11

22.05

May 2021�21.02 on Summit

November 2022�22.08 on Sunspot

December 2022�22.11 on Sunspot

23.02

23.05

23.08

NERSC Systems
OLCF Systems
ALCF Systems

6 of 80

Meeting our Objectives for ECP

Requirements

Deploy E4S on pre-exascale systems (Summit, Sunspot, Perlmutter)
Report build failures to ECP Teams
Add compiler support for Intel OneAPI (ALCF), NVHPC (NERSC), and RoCM + HiP (OLCF)
Run E4S tests on facility deployments using E4S Testsuite (https://github.com/E4S-Project/testsuite) and spack test

Execution Plan

ECP SI team will deploy E4S every 3-month on their facility systems.
ECP SI team will report build failures to Spack and E4S Project
ECP SI will contribute to Spack Project through code contribution and reporting issues
ECP SI team will run E4S tests after deployment and report test failures

7 of 80

E4S Presence at NERSC

As part of our first E4S release, we developed our E4S documentation which is now available at https://docs.nersc.gov/applications/e4s/
E4S can be accessible via modulefile, also available as a prebuilt container
We have deployed a total of 9 E4S versions with a dedicated support model for user support
We hosted 2 NERSC events for E4S with invited speakers from ECP leadership including spack and e4s training

8 of 80

Buildtest-NERSC: An Automated E4S Testing Infrastructure

What is it?

A git repository using buildtest to run tests primarily focused on testing facility deployments of E4S

How does it work?

Leverages Gitlab CI/CD using scheduled pipelines to run a subset of tests
Utilizes a custom gitlab runner based on Jacamar CI to run tests via a single user account
All test are run using the devel branch of buildtest

Where do the test get published?

At completion of CI job, buildtest will publish test results to CDASH at https://my.cdash.org/index.php?project=buildtest-nersc

9 of 80

Collaboration with Kitware we picked CDASH for publishing test results

In collaboration with Kitware, we used CDASH to publish test results. Zack Galbreath (Kitware) helped us create a project and how to publish test results to CDASH. Buildtest added support for CDASH to enable us to publish test results through command line
Several ECP teams including Spack and E4S were using CDASH for sharing test results. We picked CDASH as our platform for sharing test results since it was familiar with everyone

10 of 80

Continuous Automated Testing Workflow

Our automated testing workflow is driven by Gitlab CI using scheduled pipelines
We have multiple scheduled pipelines to run different workloads on Perlmutter and Muller
Test may run on login node or compute node.
At end of execution, buildtest will upload test report to CDASH for post-analysis

11 of 80

E4S Testsuite: A standalone testsuite for validating E4S Packages

E4S Test Suite is a collection of tests to validate E4S stack and increase test coverage for deployed stack.
Makes use of several shell-scripts and shell functions to mimic spack commands.
Flexible enough to run on any spack instance

12 of 80

E4S Testsuite comes with several scripts

Each package contains a few shell-scripts clean.sh, compile.sh, run.sh and setup.sh that are used for running the test
Some tests will use the spack test feature for verifying package therefore, no tests are maintained in Testsuite repository
The command spackLoadUnique and spackTestRun are shell functions defined by Testsuite meant to mimic spack load and spack test run command.
Testsuite repo will contain source files and Makefile to run tests for particular packages that may not have built in spack smoke tests.

13 of 80

An example output of E4S Testsuite on Cori

15 of 80

Buildtest: A HPC Testing Framework

Buildtest is a testing framework that builds and executes tests on your HPC systems
Intended for HPC staff, developers, and end-users.
Tests are written declaratively in YAML
Integration with Batch Schedulers, modules, CrayPE and spack
Project started in 2017 and available on GitHub at https://github.com/buildtesters/buildtest
Currently, we have total of 48 releases with a 8-week release cycle

16 of 80

Key Features

CLI interface for building and analyzing test results
Automatic discovery of buildspecs
Query test report
Support multi test generation
Upload test report to CDASH
View, Edit or Validate buildspec files
Automatically discover compilers, test and remove compilers
Several mechanism for passing test (returncode, regex match, file existence, performance check)

17 of 80

New Features

Limit number of test can be run in single run via buildtest build –limit
Set a maximum number of jobs to run in a single iteration via buildtest build -–max-jobs
Support for running tests in containers
Add support for logical assertion AND/OR in performance checks
Run tests by name via buildtest build –name

18 of 80

Buildtest comes with many features

Buildtest is command line tool that comes with many features such as

Building test
query test results
Query report file
Interacting with buildspecs

19 of 80

Installing Buildtest is Easy

Installation is easy, just clone repo and source setup script.
Buildtest requires python 3.8+

20 of 80

Build & Test Pipeline

Discover: Find buildspecs based on search criteria (file, directory, tags, executor, name)
Parse: Validates buildspec with JSON Schema
Build: Generates a shell script from YAML
Run: Executes tests and retrieve return code and output/error file.
Sanity Check: Performs sanity checks to determine if test pass or fail
Update Report: Update report file with test results including any metadata

21 of 80

What does a typical build look like in Buildtest

22 of 80

Example Test in Buildtest

In buildtest, test are written in YAML called buildspecs
Every test must start with declaration named buildspecs
Test are declared in buildspecs section
Every test must specify type which determines the JSON schema used for validating test
The executor field is used to determine where test will be run

Name of Test

Schema Type

Description of Test

Name of Executor

Tag Name

Script

Declaration of tests

23 of 80

Buildtest supports several criteria for status check on how test will PASS

Buildtest supports several types of status check that can be used to determine how test will PASS.
This includes the following:

Returncode Match
Regular Expression Match
Runtime Match
File Checks
Performance Checks
Explicit Declaration

24 of 80

Test Status based on Regular Expression

Test can be passed on content of output or error stream (stdout, stderr).
For instance, you can specify a regular expression via regex property, the stream must be stdout or stderr used for opening the file. The exp field is the regular expression applied to file, if there is a match the test will PASS otherwise it will FAIL
All status checks are defined via the status property

25 of 80

Status Check by Return Code Match

By default, buildtest uses the returncode for status checks where a returncode of 0 is a PASS and a non-zero is a FAIL
You can uses the returncode property to alter status checks where the returncode can be an integer or a list of integers

26 of 80

Status Check by File Checks

Buildtest can perform file checks such as the existence of directories or files as part of the status check
The is_dir and is_file fields are a list of files or directories to check.
In this example $HOME/.bashrc is a file, not a directory, which implies the is_dir check will fail and therefore file_and_dir_checks will have a FAIL status

27 of 80

Multi Test Generation by Compiler

Buildtest can support multiple compiler tests via the compilers property which will detect the compilers from the buildtest configuration
buildtest will declare some built-in variables such as

Compilers wrappers: BUILDTEST_CC, BUILDTEST_CXX, BUILDTEST_FC
Compiler flags: BUILDTEST_CFLAGS, BUILDTEST_CXXFLAGS, BUILDTEST_FFLAGS
Linker flags: BUILDTEST_LDFLAGS
Preprocessor flags: BUILDTEST_CPPFLAGS

28 of 80

Batch Submission Support

Buildtest supports batch job submissions to Slurm, LSF, PBS and Cobalt
First you must configure executors in the buildtest configuration to map to queues. In this example, we have setup an executor to submit jobs to the debug queue on the KNL partition.
You can pass #SBATCH arguments via the sbatch property
Buildtest will dispatch a job, acquire that job’s id, and poll until that job finishes

29 of 80

Running Test on Multiple Executors

Buildtest can run tests with multiple executors, this can be useful when running test with multiple shells or running test on multiple batch queues.
The executor property can be a regular expression which is matched with available executors defined in configuration file.
The buildtest config executors list will list available executors defined in configuration file

30 of 80

Performance Checks

buildtest supports several performance checks that can be used for performance regression testing.
Currently, we support several comparison operators including: >, >=, <, <=, ==, !=
For more details on Performance Checks please see https://buildtest.readthedocs.io/en/latest/writing_buildspecs/performance_checks.html

31 of 80

Assert Greater Equal

Buildtest can capture performance metrics via metrics property.
Every metric must have a name and it must have a value by applying a regular expression on output or error stream
The assert_ge is a list of assertions for >= for every metric with corresponding reference value (ref)

32 of 80

Running Test in Containers (Experimental Feature)

Buildtest now has support for running tests in container platform.
The container field can be used for specifying container platform, image, and command to run.
We can bind mount volumes into the container as well

33 of 80

Buildtest Features

34 of 80

Querying the Buildspec Cache

Buildtest creates a buildspec cache file (JSON) that can be used to query test and interact with buildspecs
Buildtest can summarize content of cache file that can be useful to see breakdown of tests

35 of 80

Interacting with buildspecs

Buildtest provides support for viewing content of test, validating buildspecs and finding all invalid buildspecs
The buildtest buildspec show will show content of buildspec given test name
The buildtest buildspec validate will validate a buildspec file with JSON schema
The buildtest buildspec find invalid will show all invalid buildspecs

36 of 80

Query Test Results

The buildtest inspect query command allows one to analyze test metadata such as output, error, test content for tests that have completed
The positional argument is name of test
Buildtest will query the report file given the test name and extract the appropriate content
Tab completion is supported

37 of 80

Querying Report File

Buildtest will record all test completions in a JSON report file. This file can be queried by users to see
The buildtest report command will report all tests in report
Buildtest can retrieve all PASS or FAIL tests and format table columns. The –fail option will retrieve all failed tests and –format is can be used to specify format fields that will alter table columns

38 of 80

Integration with CDASH

buildtest has the ability to upload test reports to CDASH. The buildtest configuration can be used to specify CDASH configuration such as path to CDASH server, project name, site, etc…
The buildtest cdash upload will upload test results given a build name which is required in group test in a single entity
This integration enables us to share test results publicly

39 of 80

Buildtest Spack Integration

40 of 80

Spack support in buildtest

Spack support was added in buildtest v0.10.0 to write buildspecs using the spack schema.
Current support includes

Installing specs
Managing environments (create, activate, remove)
spack test support
Specify scheduler options.

For more details on spack support see https://buildtest.readthedocs.io/en/devel/buildspecs/spack.html

41 of 80

Installing Spack Packages from a buildspec

In order to use a spack schema for validating a buildspec, you must specify type: spack in the test.
You must specify where spack is installed in-order to source the spack setup script.
You can specify a list of specs to install via the specs property which will use the spack spec syntax

Use Spack schema

Root of spack

List of specs to install

42 of 80

Running test via spack test

The test property maps to the spack test command and run property expects a list of specs to test that is defined via the specs property.
This will get mapped to spack test run <spec1> <spec2> in the script.
The spack test run –alias command allows one to name a testsuite in-order to fetch results, in buildtest this gets mapped to the name of test

44 of 80

Challenges we faced

Bugs in tests, CI job or buildtest
High CPU utilization and long runtimes (~45min)
Gitlab runner availability
Non Deterministic output due to concurrency limit in CI jobs
Failures in spack test can’t be fixed in production deployment
User Environment (shell, startup configuration, modules) can impact test behavior

45 of 80

Community Impact

Our testing efforts has helped uncover bugs and test failures that have been reported to developers.
NERSC has reported a total of 8 issues to E4S Project to help improve E4S package support
We reported 7 issues to Spack Project pertaining to test failures for packages: amrex, heffte, strumpack, slate, raja, and tasmanian
Developers from AMReX, HPCToolkit, Slate, libEnsemble, superlu, strumpack have contributed standalone tests to https://github.com/buildtesters/buildtest-nersc
Currently, we have up to 156+ tests with tag names e4s, e4s-22.11, e4s-23.05, e4s-23.08 most of them are from spack test or E4S Testsuite

46 of 80

Concluding Remarks

We came a long way … 0 tests → 150+ tests
We setup an automated testing infrastructure using buildtest coupled with E4S Testsuite and Spack Test
We have reported test failures back to developers to help improve their product
Buildtest integration with CDASH has helped us publish test results to web that is vital for sharing results with ECP teams
The E4S Software Stack has helped NERSC provide software to users with comprehensive testing for key software applications.

47 of 80

What’s Next

To learn more about buildtest, we will have a tutorial on Dec 21st 10-12:30pm PST. To join please use zoom meeting: https://lbnl.zoom.us/j/93417223752?pwd=c2dpVVdzVHlNd1I0V1lMb2lsL0tnZz09
Try out buildtest on NERSC system using the test repo https://github.com/buildtesters/buildtest-nersc
Your feedback are welcome! Post on slack or Github issue

49 of 80

References

Buildtest

Documentation: https://buildtest.rtfd.io/
Source Code: https://github.com/buildtesters/buildtest
Slack: http://hpcbuildtest.slack.com/
Buildtest at NERSC: https://github.com/buildtesters/buildtest-nersc
CDASH for buildtest-nersc: https://my.cdash.org/index.php?project=buildtest-nersc

E4S

E4S Home Page: https://e4s.io
E4S Docs: https://e4s.rtfd.io/
E4S Testsuite: https://github.com/E4S-Project/testsuite

50 of 80

Tutorial Logistic

51 of 80

Join Slack

For this tutorial, please join the buildtest slack channel.
Please do the following

Join slack by typing your email address in https://communityinviter.com/apps/hpcbuildtest/buildtest-slack-invitation
Check your email for confirmation
Join the slack channel at http://hpcbuildtest.slack.com/
For this tutorial please join the tutorial channel

Click Channels > Manage > Browse channels
Search for channel and click Join

52 of 80

Install Buildtest on Perlmutter

In order to install buildtest, you need to clone buildtest.
Please load the python/3.9-anaconda-2021.11 module
Please create a new virtual environment when using buildtest.
Upon completion, please check of buildtest command is in your $PATH
See Installing Buildtest for more details
Note: Please check your startup shell configuration (~/.bashrc) for any user customization. For this tutorial, its best to rename your startup .bashrc file to backup location (.bashrc.bak) and re-login

53 of 80

What you will be Learning Today

Install Buildtest
Quick Start Guide
Buildtest Command Line Tutorial
Overview of Buildspecs
Hands-on Exercise on Perlmutter

54 of 80

Logistics

Please follow instructions outlined in documentation by opening a browser window and navigate to https://buildtest.readthedocs.io/. We recommend you have documentation and terminal window side-side so you can follow along
Slides will be available in slack channel and zoom chat for reference
Most commands should work with copy-paste, use the Copy to clipboard button to copy commands into terminal
For further assistance, please post your questions in slack channel. Please keep messages in thread to help diagnose errors
Please report any bugs at https://github.com/buildtesters/buildtest/issues
We will have a 10 min break in middle of session

55 of 80

Reference Slides

56 of 80

Discover Buildspecs

Buildtest can search for buildspecs based on

File
Directory
Name
Tags
Executors

buildtest can exclude buildspecs based on file or directory or tags

57 of 80

Test Dependency By State and Return Code

Test dependencies can be further tweaked, not just by name of test but you can rely on test status.
The state and returncode are attributes to needs section. The state maps to the test state (PASS/FAIL) upon completion
The returncode maps to the actual returncode of test.
Each test must complete execution prior to dispatching next test.
A test can be dispatched once all dependency have been satisfied

58 of 80

Metrics Exceptions and mismatch

If you specify an invalid metric name, the assertion check will fail
The assert_eq can support multiple metrics check, if any metrics fails assertion, the assertion check will fail.
Metrics will be undefined if regular expression is not a match

59 of 80

Assertion Range with lower and upper bound

Buildtest can support a lower and upper bound during assertion check. This is can be done via assert_range property.
The lower and upper fields are used for specifying the lower and upper bound for comparison and they must be integer or float
In this example, we run STREAM benchmark on laptop, the performance results may vary

60 of 80

Equality Check

If you want to perform equality check (==), you can use assert_eq property.
String comparison is supported with assert_eq but not for comparison operators (>, <, >=, <=)

61 of 80

Contains and Not Contains

So far, we have specified only one reference value for comparison, but what if we want to specify multiple reference values
This can be done via contains and not_contains which perform the python operator in and not in on a list
The list may contain any element (int, float, str)

62 of 80

Test Dependency

Buildtest has support for test dependencies via the needs property. Where one can define conditions on how a test should be executed.
The needs property is a list of conditions, in this example

jobA has no dependency
jobB dependens on jobA
jobC depends on jobA and jobB

63 of 80

Custom Status By Executors

The executors property can be used to define custom logic per executor for a test. For instance, if you want to have different status check when running tests across multiple executors
In this example, we expect a test failure with executor generic.local.sh since we are expecting a returncode of 1 while the program exited with 0

64 of 80

Creating spack environment via spack.yaml

You can create a spack environment from a spack.yaml file via the manifest property which expects the path to the spack.yaml file. This can be useful if you want to Bring Your Own spack.yaml when writing tests.

65 of 80

Removing spack environments

Buildtest provides two methods for removing spack environments.
The rm property is used to remove a spack environment by name.
Alternatively, you can remove remove environment prior to creating environment via remove_environment
The remove_environment expects a boolean. The rm is property under env which maps to the spack command spack env rm.
remove_environment property will remove an environment based on the name property

66 of 80

Specify Pre and Post Commands

The pre_cmds and post_cmds allow one to specify arbitrary Linux commands in the test.
The pre_cmds will add commands prior to the sourcing of spack
The post_cmds are commands that will be run at end of the script.
Shown below is the generated test, take note of the location where pre and post commands
You can use these properties (pre_cmds, post_cmds) to customize your test further and not just limited to spack commands

67 of 80

Creating spack environments

The create property under env is used for creating spack environments.
You are responsible for activating spack environments upon creation.
The install property maps to spack install and one can pass options via the option property

Create spack environment

Options to spack install

Find spack compilers

Activate spack environment

68 of 80

Example of installing and testing spack packages in a spack environment

69 of 80

View all builds

Buildtest keeps track of every buildtest build command, which can be used to query previous builds.
Every build can be queried via the id property which is automatically incremented as new builds are performed
Buildtest will store all historical data in $BUILDTEST_ROOT/var/.history

build.json: Metadata for build including test results
output.txt: Output of buildtest build command
buildtest_*.log: The log file for build

70 of 80

View Output of Historical Builds

The –output option can be used to see output of buildtest build for a historical build. Buildtest will store the output in file

71 of 80

Getting the path to tests

The buildtest path command can retrieve the path to a test given a test name. If no options are specified we retrieve the directory where the test is available.
You can specify a test ID by specifying the name followed by a backslash (/) and the name of the test ID
By default, buildtest will fetch the latest run if ID is not specified

72 of 80

Querying Test Report

The –count can be used to limit number of rows to print
The buildtest report summary can summarize test report and give a breakdown result
Buildtest can display output in machine readable format via –terse format that can be used for parsing results
The –row-count option can be used retrieve number of rows in query which can be useful for determining

# of tests, # of pass/fail tests
# of tests by filtering
# of tests by returncode

73 of 80

Automate Compiler Testing

buildtest can automatically test your compiler definitions via buildtest config compilers test. Buildtest makes use of Lmodule, a Python API for Lmod which automates module testing.
Compiler definition is stored in buildtest configuration file, that includes name of compiler, compiler wrapper and any associated modulefile used for loading the compiler.
Compilers can be defined explicitly or auto-generated via command buildtest config compilers find

1 of 80