1 of 27

Shahzeb Siddiqui (LBNL)�Sijie Xiang (Carnegie Mellon University)�PEARC22�July 11^th, 2022

Testing your HPC System with Buildtest

https://buildtest.rtfd.io/

https://github.com/buildtesters/buildtest

http://hpcbuildtest.slack.com/

2 of 27

Tutorial Logistics

The slides can be found at https://buildtest.readthedocs.io/en/devel/references.html
Join the slack channel at https://hpcbuildtest.herokuapp.com/, upon registration you can navigate to http://hpcbuildtest.slack.com under #tutorials to post your questions, we will be monitoring the slack channel. Please reply in threads when troubleshooting an issue
This tutorial will be conducted on NERSC systems
We will provide training accounts which will grant you access to system.

3 of 27

Go to https://iris.nersc.gov/train and fill out the form. Please use the Training Code bYEK and submit the form. Upon creation you will get a NERSC account

4 of 27

Connecting to NERSC system

You can ssh into Cori with your training account typically starts as train followed by 3 digits.
Your password will be shown after completing form. Please SAVE IT

5 of 27

Getting Help with NERSC resources

Please see our user documentation https://docs.nersc.gov/
If you need help you can submit a ticket to https://help.nersc.gov and one of the NERSC staff will get back to you

6 of 27

What will you be learning today?

Introduction to buildtest concepts
How to use buildtest command line interface
How to write test in buildtest
Run buildtest on NERSC with real test examples

7 of 27

Agenda

Topics	Time (EST)
Logistics	8:30am – 8:40am
Buildtest concepts	8:40am – 8:50am
Buildtest Tutorial Part 1	8:50am – 10:00am
Morning Break	10:00am – 10:15am
Writing Test in buildtest	10:15am – 10:30am
Buildtest Tutorial Part 2	10:30am – 11:30am
Using buildtest at NERSC	11:30am – 12:00pm

Slides
Tutorial
Break

8 of 27

What is buildtest

Buildtest is a testing framework that builds and execute tests on your HPC systems
Buildtest is intended for HPC Staff, developers, end-users who want to test the HPC system
Tests are written in YAML called buildspecs which buildtest process to generate shell scripts.
Buildspecs are validated with JSON schema.
Buildtest can run tests locally and batch schedulers. Currently we support Slurm, LSF, PBS and Cobalt.
The framework is implemented in python
Available on GitHub at https://github.com/buildtesters/buildtest

Installation

9 of 27

Design Goals

Perform component level testing for system and software stack
Provide a standard template for writing tests
Abstract low-level system configuration
Framework should automate build and execution of test
Framework must support local and batch submission test

10 of 27

Terminology

Name	Description
Buildspec	Is a YAML file that buildtest interprets when building and running the test.
Global Schema	Is a JSON schema that defines top-level structure for buildspec and validates the buildspec file. All buildspecs are validated with global schema.
Sub Schema	A test instance in buildspec file is validated with one sub-schema defined by type field. Sub-schemas are versioned schema.
Executor	Is responsible for running the test. Executors are defined in your buildtest configuration.

11 of 27

Preview of buildtest

The buildtest help command can show an overview of all buildtest commands. Shown below is a preview of building commands. For more details see https://buildtest.readthedocs.io/en/devel/what_is_buildtest.html#preview-of-buildtest

12 of 27

General Pipeline

For every discovered buildspecs, buildtest will do the following:

Parse: Validates buildspec with JSON Schema
Build: Generates a test script from YAML
Run: Runs tests via executor where test can be run locally or batch submission
Gather Results: Write output/error file and get return code
Update Report: Update report file with test results with all metadata associated with test

13 of 27

Buildspec Validation Process

Every buildspec is validated by global schema and a subschema defined by type field.
Buildtest will skip any buildspecs that fails validation.

14 of 27

Buildtest Tutorial Part 1

15 of 27

Buildspec Script Schema

Name of Test

Schema Type

Description of Test

Name of Executor

Tag Name

Script

Declaration of tests

16 of 27

What is an Executors

In buildtest, executors are responsible for running test. An executor can be a local or batch executor (Slurm, LSF, PBS, Cobalt).
Executors are defined in the buildtest configuration file.
Local executors will run jobs locally where one defines name of shell (bash, csh, tcsh, zsh)
Batch executors such as slurm will map to queue/partition where jobs will be submitted queue when test

Name of System

Declaration of Local Executors

Declaration of ‘bash’ executor

17 of 27

Return Code Matching

The return code field can be used to customize how test is passed, by default a return code 0 is a PASS. The return code can be a single number or a list of return codes to match

18 of 27

Buildspec Compiler Schema

The compiler schema is used for compiling single source code with compilers
You must use type: compiler to define tests using this schema
This test will be built with all gcc compilers
Compilers are defined in buildtest configuration, one can retrieve compilers using buildtest config compilers

Source File

Compiler Schema

Start of Compiler Block

Select Compilers based on Regular Expression

Default Section for compilers organized by compiler groups

Default Section for gcc compilers

Set cflags

Set ldflags

19 of 27

Override Compiler Default

Compiler defaults can be overridden in config section which expects compiler names defined in buildtest setting.
Buildtest will ignore compiler in config if it’s not picked up in regular expression.

Compiler Names

20 of 27

Scheduler Support

Slurm

Cobalt

LSF

PBS

21 of 27

Buildtest Tutorial Part 2

22 of 27

How is buildtest used at NERSC

We have a test repository https://github.com/buildtesters/buildtest-nersc that will run series of tests for our HPC system – Cori and Perlmutter
We use Gitlab with scheduled pipelines to run a subset of test periodically at different interval and push results to CDASH server https://my.cdash.org/index.php?project=buildtest-nersc
We use the devel branch of buildtest to run all of our tests.

23 of 27

Scheduled Pipelines

We have several scheduled pipelines to run test on Cori and Perlmutter.
Each scheduled pipeline has a predefined variable named TESTNAME that distinguish between different schedule pipeline

24 of 27

25 of 27

Buildtest at NERSC Tutorial

For this tutorial, clone the following repo https://github.com/buildtesters/buildtest-nersc on NERSC system. Clone via HTTPS
Please follow the instructions in README.md to setup buildtest and proceed to tutorials.md

26 of 27

Closing Remarks

Testing your HPC system can help increase test coverage and help discover bugs. Whether you are a developer, end-user or system-administrator, you can use buildtest to write test for your HPC system
Consider using some CI tool like Gitlab, Jenkins to automate execution of tests.
It is vital you run your test periodically to uncover bugs as underlying system changes. You can incorporate your test process into the maintenance process where one test all changes on your TDS system prior to applying changes to production system
You should pick a version of buildtest and configure buildtest for your site. See https://buildtest.readthedocs.io/en/latest/buildtest_site.html for more details

27 of 27

How to get involved

You can join the slack channel http://hpcbuildtest.slack.com to join the buildtest community for additional support
Star this repo or fork, we welcome your contributions to buildtest
Have a feature in mind? Or file a bug report you can create an issue at https://github.com/buildtesters/buildtest/issues
Share your feedback on this tutorial and how we can improve this for future events.