1 of 1

Streaming Data in HPC Workflows Using ADIOS

Scientific Achievement

We present SST, the Sustainable Staging Transport, an approach which allows direct streaming between traditional file writers and readers. SST is an engine of the ADIOS I/O framework, that provides scalable staging on the entire scale of the Frontier supercomputer.

Significance and Impact

SST is used in various applications, for:

PIConGPU: Feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier’s file system.
WDMApp: Strong coupling of separately developed applications for multi-physics multi-scale simulation.
WRF: In situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.

PIConGPU data staging test on Frontier. Data from all GPUs sent to a second application for the purpose of training a model. The data transfer bandwidth with SST is much higher than using the file system.

Technical Approach

SST separates the Control Plane from the Data Plane.
Control Plane is designed for dynamic connections of multiple consumers to a producer, and for scalable distribution of application metadata to all processes. Consumer failures do not affect the producer.
Data Plane has several implementations to utilize Remote Direct Memory Access (libfabric, ucx), MPI inter-communicators, or reliable transfer protocols (tcp, rudp).

PI(s)/Facility Lead(s): Scott Klasky, ORNL

Collaborating Institutions: Oak Ridge National Laboratory

ASCR Program: SciDAC RAPIDS2

ASCR PM: Kalyan Perumalla

Publication(s) for this work: Greg Eisenhauer et al. "Streaming Data in HPC Workflows Using ADIOS", CUG'24, Perth, AU

Credit:

F. Pöschel, K. Steiniger

Center for Advanced Systems

Understanding (CASUS), Germany

LOCAL LAB POC:

Norbert Podhorszki and Scott Klasky, ORNL

TALKING POINTS:

SST was designed to be the generic, flexible and scalable, but well-performing as well, staging solution for the ADIOS I/O framework.
The Data Plane has a single task, to deliver data from one process of a producer, to one process of any consumer. Implementations use different communication libraries to optimize of specific platforms, e.g. libfabric on HPE/Cray networks, UCX on Mellanox/NVIDIA hardware, MPI inter-communicators in general cluster computing, TCP/IP and R-UDP for reliable wide-area-networking.
The Control Plane is designed to operate with ADIOS semantics, to gather and distribute application metadata in a scalable manner to all consumers, all information that is necessary to create specific data transfer requests to the Data Plane.
The Control Plane allows consumers to connect and disconnect at any time. It is either blocking the producer so that consumers are able to process all output steps or discarding output steps and letting the producer run unimpeded by slow consumers. \
Figure: Application: PIConPGU, machine: Frontier. Kelvin-Helmholtz Instability simulation, about 50 TB data produced in a step at full scale of Frontier (5.6 GB / GPU). Data is staged from PIConGPU to a second application (not a real training code yet, just a test for I/O)

METADATA:

Name of the associated awarded project: SciDAC RAPIDS2

PI name(s): Scott Klasky (ORNL) and Robert Ross (ANL)

Name of the program manager: Kalyan Perumalla

CITATIONS:

(Accepted for publication) Greg Eisenhauer et al, "Streaming Data in HPC Workflows Using ADIOS", CUG'24, Perth, AU
Selected for further journal publication in Concurrency and Computation Practice and Experience” (CCPE) Journal special edition.�