1 of 1

Flash-X Performance Orchestration�With the NP ENAF Partnership

Scientific Achievement

We started developing an end-to-end performance portability solution for Multiphysics applications under the Exascale Computing Project. The solution is language agnostic, obviating the need to rewrite the code in C++ as most other solutions do. The solution is now deployed in Flash-X for some of the physics.

Significance and Impact

Our approach provides an effective solution that is designed to evolve with the evolution of hardware and software complexity. It can be applied to many legacy codes also.
Some of our tools provide complimentary features to existing C++ solutions, and therefore can be used in combination with those solutions.

This figure shows performance comparison of using shock hydrodynamics with adaptive mesh refinement in three different modes. The top dotted line is the performance using OpenACC offloading within a block, and the bottom green lines shows offloading done with our toolchain. The solid blue line is CPU only performance..

Technical Approach

. It uses three sets of tools, each one addressing one aspect of performance.

Unify expression of arithmetic with different data layouts for different devices: This is done with macros that can have multiple alternative definitions, and a tool that can arbitrate on selecting the most suitable one.
Unify algorithmic variants and map computation to devices: Algorithmic variants are expressed in pseudocode like recipes specifying order of execution and where to execute which computation.
Move data and computation to targets specified in the recipe. This is done with a domain specific runtime Milhoja. The code to interface with Milhoja is also generated along with recipe translation.

PI(s)/Facility Lead(s): Person Name; Anshu Dubey

Collaborating Institutions: Virgina Tech, Riken

ASCR Program: [ECP, SciDAC.]

ASCR PM: Lali Chatterji, Kalyan Perumalla

Publication(s) for this work : Youngjun, et al., in preparation

Dubey et al., https://doi.org/10.1016/j.future.2023.07.014

O’Neal et al, https://doi.org/10.1007/978-3-031-06156-1_13

LOCAL LAB POC: Anshu Dubey, Youngjun Lee

TALKING POINTS:

There are not many non-C++ solutions for tackling platform heterogeneity. Several very useful applications are still in Fortran, converting them to C++ to be able to use existing solutions will be very costly.
C++ based solutions are typically only geared for the first of three aspects of performance portability mentioned above. They do not address different control structures or algorithmic variants that the combination of CGKit and Milhoja can handle. The code generators and assemblers provided with the toolchain parse the recipe, determine which specific definition of macros to use, generate the code for the specified control structure, and generate code to interface with Milhoja, the runtime.
For existing codes wanting to use the tools needed modifications are modest. These include decomposing relevant code sections into macros and annotating function definitions with information needed by the code generators as comments.
At present emitted code is mostly in Fortran with some C++, but adding ability to emit code in other languages is the easiest part of the toolchain

METADATA:

Name of the associated awarded project: ECP, RAPIDS, and ENAF

PI name(s): Daniel Kasen, Rob Ross, Raph Hix

Name of the program manager: Supported by both ECP and ASCR Research (Lali Chatterjee)

CITATIONS:

A. Dubey, Y. Lee, T. Klosterman, E. Vatai, A tool and a methodology to use macros for abstracting variations in code for different computational demands, Future Generation Computer Systems,

2023, https://doi.org/10.1016/j.future.2023.07.014

Johann Rudi, Youngjun Lee, Aidan H. Chadha, Mohamed Wahib, Klaus Weide, Jared P. O'Neal, Anshu Dubey, CG-Kit: Code Generation Toolkit for Performant and Maintainable Variants of Source Code Applied to Flash-X Hydrodynamics Simulations, https://doi.org/10.48550/arXiv.2401.03378, in revision for FGCS

O’Neal, J., Wahib, M., Dubey, A., Weide, K., Klosterman, T., Rudi, J. (2022). Domain-Specific Runtime to Orchestrate Computation on Heterogeneous Platforms. In: Chaves, R., et al. Euro-Par 2021: Parallel Processing Workshops. Euro-Par 2021. Lecture Notes in Computer Science, vol 13098. Springer, Cham. https://doi.org/10.1007/978-3-031-06156-1_13

BACKGROUND AND CONTEXT INFORMATION:

The end-to-end working solution is an early result. Individual tools are been published and/or are in the process, this one still needs many more experiments