1 of 1

sKokkos: Enabling Kokkos with Transparent Device Selection on Heterogeneous Systems using OpenACC�

Scientific Achievement

An ORNL research team elevated the programming productivity for Kokkos applications by enabling the Kokkos runtime to decide which device to use on heterogeneous systems in a transparent way thanks to OpenACC capabilities.

Significance and Impact

sKokkos eliminates the burden for architecture or device selection for Kokkos application developers
Provides a real performance-portable and high-programming productivity solution on heterogeneous systems
Achieves high speedups and power efficient hardware exploitation by selecting the proper device
Nominated for Best Paper Award from the ACM HPC Asia’24 conference

Figure 1: LULESH overall performance for NVIDIA H100 Hopper GPU, showing sKokkos speed up of up to 14x.

Y-axis: (left) execution time (s) in logarithmic scale and speedup (right). X-axis: input problem size (3D grid). Speedup is computed as the ratio between the slowest KokkACC time (using either a CPU or a GPU) and sKokkos time.

Technical Approach

Used OpenACC capabilities within Kokkos for multi-device support and runs
Developed an agnostic and high-level autotuning technique to decide which device to use depending on hardware features and application demands

PI(s)/Facility Lead(s): Pedro Valero-Lara, Jeffrey S. Vetter; Scott Klasky

Collaborating Institutions: Oak Ridge National Laboratory

ASCR Program: RAPIDS-2 and ECP

ASCR PM: Kalyan Perumalla

Publication(s) for this work: Pedro Valero-Lara, et al., sKokkos: Enabling Kokkos with Transparent Device Selection on Heterogeneous Systems using OpenACC. HPC Asia 2024: 23-34.

doi:10.1145/3635035.3635043

and/or Code Developed or Datasets: https://code.ornl.gov/5pv/skokkos

Pedro Valero-Lara, Seyong Lee, Joel E. Denny, Keita Teranishi, Jeffrey S. Vetter, and Marc Gonzalez Tallada, sKokkos: Enabling Kokkos with Transparent Device Selection on Heterogeneous Systems using OpenACC. HPC Asia 2024: 23-34. doi:10.1145/3635035.3635043

LOCAL LAB POC:

Scott Klasky

Pedro Valero-Lara

TALKING POINTS:

sKokkos eliminates the burden for device selection to Kokkos applications.
Provides a real performance-portable and high–programming productivity solution.
sKokkos also achieves important accelerations by selecting the proper device, depending on the hardware characteristics and application particularities via autotuning.
Tested on real applications such as the ECP applications LULESH, a molecular dynamics application, and MiniFE, a finite-element CFD application, providing high accelerations.

METADATA:

Name of the associated awarded project:

PI name(s): Pedro Valero-Lara (ORNL), Jeffrey S. Vetter (ORNL)

Name of the program manager: Hal Finkel, Kalyan Perumalla

CITATIONS:

Pedro Valero-Lara, Seyong Lee, Marc Gonzalez-Tallada, Joel E. Denny, and Jeffrey S. Vetter, KokkACC: Enhancing Kokkos with OpenACC, Ninth Workshop on Accelerator Programming Using Directives (WACCPD), in conjunction with SC22 (Best Paper Award), 2022.

AWARDS:

Nomination for the Best Paper Award from the ACM HPC Asia’24 conference

REPRODUCIBILITY:

The ACM HPC Asia’24 did require to provide any support for reproducibility. However, the previous paper on the OpenACC back end was accepted by the WACCPD workshop in conjunction with SC’23. In that case, we provided the required support for reproducibility, and we received the badges and recognitions for reproducibility. For this paper, although it was not required, we provided repositories containing the necessary resources to reproduce the results shown in the paper. However, we did not get any badge or recognition for that.

BACKGROUND AND CONTEXT INFORMATION:

This is the second award achieved by the efforts for KokkACC, the OpenACC back end of Kokkos. KokkACC is part of the Kokkos ecosystem, and it is accessible for anybody. The first award was a best paper award from the SC workshop WACCPD in 2022.