Updates on CMake, Spack and ADIOS2�

Apr 1st, 2024

​

Chang Liu

​

​

​

Motivations

Introduce CMake and Spack for a new build system for M3D-C1

​

• Simplify the deployment workflow of M3D-C1 at various computing platforms including all the dependencies
• Easy to introduce new dependent packages (and variants of existing ones)
• Use a unified build system for m3dc1_scorec and M3D-C1
• Make the code more friendly for users and maintainers

​

Develop a new I/O interface based on ADIOS2

​

• Part of the project of “Center of Advanced Computing” for developing interfaces between codes
• Improve the parallel I/O performance

​

2

Introduction to the new tools

CMake

• CMake is a build automation tool that can be used to generated Makefile.
• CMake can be used for projects combining multiple languages (C++, Fortran), and can deal to various compilers
• CMake can help find the dependent packages and set up the compiler flags

Spack

• Spack is a package manager for deloying software and dependencies, for both privileged and unprivileged users.
• ECP has chosen Spack as the default package management system
• NIMROD has decided to switch from their own build system to Spack

ADIOS2

• ADIOS2 is a framework for scientific data I/O to publish and subscribe to data when and where required.
• Similar to HDF5, with more choices of interfaces and better parallel performances

3

Building M3D-C1 with CMake

​

mkdir build && cd build

cmake .. -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn -DCMAKE_PREFIX_PATH="/ccs/home/cliu1/project/spack/var/spack/environments/m3dc1/.spack-env/view;/ccs/home/cliu1/project/spack/var/spack/environments/m3dc1/.spack-env/view/petsc-real" -DENABLE_PETSC=ON -DENABLE_ZOLTAN=ON -DENABLE_3D=ON -DENABLE_COMPLEX=OFF

make -j8 VERBOSE=1 or cmake --build . --verbose --parallel 8

​

• CMake will analyze all the source files and generate a Makefile including the dependency relations
• CMake will try to find all the dependencies (Petsc, PUMI, etc) and generate a Makefile containing include and library paths
• Compiler flags of M3D-C1 follows the notation in m3dc1_scorec. Now only two switches work (ENABLE_3D, ENABLE_COMPLEX)
• CMake will only generate one variant of M3D-C1. To build a different version, one can create a separate build directory.
• The “make” stage can be accelerated by setting the number of parallel jobs

​

4

Building M3D-C1 with Spack

​

git clone -c feature.manyFiles=true https://github.com/spack/spack.git spack

. spack/share/spack/setup-env.sh

cp ~/M3DC1/unstructured/spack/m3dc1/package.py spack/var/spack/repos/builtin/packages/m3dc1/

spack install m3dc1+3d~complex

spack cd -i m3dc1

​

• We recommend create a Spack environment and edit the spack.yaml file before installing M3D-C1
• Examples can be found at unstructured/spack
• One can customize the compiler and package provider (MPI, Lapack, etc) in the spack.yaml file

5

Developing ADIOS2 I/O interface

​

​

​

• ADIOS2 is capable of dealing with large-scale parallel I/O with MPI, with built-in multi timestep write function
• ADIOS2 can work through its own BP file, HDF5 file, and in-memory access
• ADIOS2 BP file can work with C++, Fortran, Python and MATLAB. No IDL interface yet
• For BP file, it is not easy to modify blocks of an existing data file
• We plan to use ADIOS2 for all the timestep field output, while continue using HDF5 for scalars and parameters

6

Performance test of ADIOS2

​

​

• We did test on Frontier for writing one timestep file using HDF5 and ADIOS2 (BP5).
• 128 nodes, 8192 processes, 3908*128 elements, 3.8GB

​

​

​

​

​

​

​

​

• The output speed of ADIOS2 outperforms HDF5
• It is possible to do field output at each timestep
• Next step: Develop ADIOS2 interface in FusionIO, try different compression options and in-memory output

​

7