1 of 13

Shahzeb Siddiqui (LBNL)�Sameer Shende (University of Oregon)�Prathmesh Sambrekar (LBNL/Arizona State University)

MVAPICH2 at NERSC

10^th Annual MVAPICH User Group (MUG)�Aug 24^th, 2022

2 of 13

NERSC

National Energy Research Scientific Computing Center (NERSC) is scientific computing facility as part of the Office of Science in the U.S Department of Energy
At NERSC we support up to 8000+ users that span a wide range of disciplines including climate modeling, simulation of universe, high energy physics experiments, protein simulation, and many more.
NERSC most recent supercomputer called Perlmutter is named after astrophysicist Saul Perlmutter for his Nobel Prize in Physics for discovering the rate of expansion of the universe

Perlmutter

Perlmutter is a HPE Cray Shasta platform composed of CPU-only and GPU accelerated nodes.
The system will be delivered in two phases
The system contains 1536 GPU accelerated nodes with NVIDIA A100 GPUs and 3072 CPU-only nodes powered with AMD EPYC Milan CPUs
Perlmutter is currently ranked #7 on the Top500 list with a peak performance of 93.75 PFlop/s

MVAPICH2 for building Spack stack

We recently published a technical report Software Deployment Process at NERSC: Deploying the Extreme-scale Scientific Software Stack (E4S) Using Spack at the National Energy Research Scientific Computing Center (NERSC) that outlines our software deployment process.
We are working with MVAPICH2 team to enable MPI support for Perlmutter when building the spack stack.
Currently we are building the spack stack with cray-mpich which works for building most of the stack, however some spack packages don’t build hence we will try to build remainder of packages with mvapich2-gdr

MVAPICH2 modules on Perlmutter

We have modulefiles for mvapich2 on shared filesystem, one must add the directory to MODULEPATH since we don’t expose these modules to all users

Common Errors during spack builds

We have noticed a few spack packages fail to build with cray-mpich due to proper support with cray compilers.
Some packages expect MPI wrappers such as mpicc but on Cray we expect to use cray wrappers cc, CC, ftn when invoking MPI programs

MVAPICH2 installation on Perlmutter

MVAPICH2 3.0a supports SLURM and Slingshot 11 network interfaces on both CPU and GPU nodes of Perlmutter
To use MVAPICH2:

Building E4S on Perlmutter

The Extreme-scale Scientific Software Stack (E4S) is installed on Perlmutter using the mvapich2/3.0a module.
For more details regarding this deployment, please refer to E4S documentation page https://e4s.readthedocs.io/en/latest/deployment.html#perlmutter
module use /global/cfs/cdirs/m3896/shared/modulefiles
module avail e4s

e4s/22.05/mvapich2-3.0a …

E4S 22.05 built with MVAPICH2/3.0a

E4S products built using MVAPICH2

E4S products built using E4S (contd.)

Running OSU Benchmark with MVAPICH2

We have developed a test to run the OSU benchmark for the mvapich2 installation.
Test are run via buildtest, an HPC testing framework that will automate build and execution of test
Test results are pushed to CDASH via buildtest

Closing Remarks

NERSC will provide support for the vendor MPI (cray-mpich), on the contrary NERSC users will experiment with any MPI provider they have access to including OpenMPI, MVAPICH2, MPICH which can be troubling since we don’t have proper support for any other MPI
MVAPICH2-GDR is promising, considering it provides support with Slurm and Slingshot 11
With sufficient testing, we can provide user with documentation on how to use MVAPICH2-GDR on Perlmutter

MVAPICH2-GDR can provide user to experiment with spack build instead of being locked into cray-mpich, we have seen several spack packages fail to recognize MPI wrappers.