1 of 1

HDF5 Sustainable Strategies for Longevity�

Scientific Achievement

The HDF5 library offers connectors for the Virtual Object Layer (VOL), which is an abstract layer that enables custom operations for improved I/O performance, such as (1) The Asynchronous connector overlapping I/O and computation operations that would otherwise be performed sequentially, (2) the Cache connector using fast storage layers for caching and staging data, and (3) the Log-based connector using a time log layout. Connectors can significantly reduce the I/O latency, reducing the total execution time of scientific codes.

Significance and Impact

HDF5 is a versatile data model, library, and file format for storing and managing data. It is a crucial component for many scientific applications. Maintaining its performance features and guaranteeing efficient I/O performance on a large scale is essential to ensuring continued support for these applications.

Comparison of the observed data write time at different time steps of a Magnitude 7 earthquake simulation (EQSIM) run on 256 Frontier nodes. The asynchronous I/O capability can effectively hide the I/O latency by overlapping the I/O operations on the CPUs with the computation operations on the GPUs. The last three writes are at the end of the simulation when there is no computation to overlap with.

Technical Approach

Development and maintenance for connectors, including bug fixes, performance improvements, code quality, testing, and scheduled releases.
Also, facilitating DOE applications and large-scale HDF5-dependent libraries through continuous performance testing, feature improvements, and connector usage.

M. Scot Breitenfeld, Houjun Tang, Huihuo Zheng, Jordan Henderson, Suren Byna, "HDF5 in the Exascale Era: Delivering Efficient and Scalable Parallel I/O for Exascale Applications", Under review in The International Journal of High Performance Computing Applications.

PI(s)/Facility Lead(s): Houjun Tang, Jean Luca Bez, Scot Breitenfeld

Collaborating Institutions: LBNL, The HDF Group

ASCR Program: RAPIDS2

ASCR PM: Kalyan Perumalla

44X speedup

LOCAL LAB POC: Houjun Tang, LBNL

TALKING POINTS:

The HDF5 library serves as a crucial I/O component for numerous scientific applications.
By utilizing connectors for the Virtual Object layer in the HDF5 library, the I/O performance can be enhanced.
The Asynchronous connector can effectively reduce the I/O time, especially for GPU-accelerated applications (e.g. EQSIM), by performing I/O operations on the CPUs without blocking the computation on the GPUs.
The primary objective of this work is to integrate sustainable strategies that aid in ensuring the longevity of these improvements for continued beneficial use in scientific applications.

METADATA:

Name of the associated awarded project: RAPIDS

PI name(s): Houjun Tang (LBNL), Jean Luca Bez (LBNL), Scot Breitenfeld (The HDF Group)

Name of the program manager: Kalyan Perumalla

BACKGROUND AND CONTEXT INFORMATION:

The impact of HDF5 and various connectors can be found in the recent papers:

J.L. Bez, H.Tang, S. Breitenfeld, et al. “h5bench: A Unified Benchmark Suite for Evaluating HDF5 I/O Performance on Pre-exascale Platforms”, Concurrency Computat Pract Experience 2024, doi: 10.1002/cpe.8046

H. Tang, Q. Koziol, J. Ravi and S. Byna, "Transparent Asynchronous Parallel I/O Using Background Threads" IEEE Transactions on Parallel and Distributed Systems, 2022, doi: 10.1109/TPDS.2021.3090322