1 of 1

AIIO: Using Artificial Intelligence for Job-Level and Automatic I/O Performance Bottleneck Diagnosis

Scientific Achievement

The first work to demonstrate the feasibility to automatically identify the I/O performance bottlenecks at the job-level for scientific applications running on high-performance computing (HPC) systems

Significance and Impact

Take the human out the loop of I/O performance bottleneck diagnosis with the cutting-edge Artificial Intelligence (AI)
Lay the foundation for methods to automatically fix the I/O performance issues of scientific applications
Open the possibility to use AI technologies to identify bottlenecks for communication and computing of scientific applications

AIIO can identify the I/O bottlenecks of applications, which can be fixed to improve performance up to 146X

Technical Approach

Multiple linear regression models based performance function to connect I/O counters with I/O performance
Game-theory based diagnosis functions with SHAP to calculate the impact of various factors on I/O performance
Incorporating the diverse characteristics (e.g., sparsity) of applications into both performance and diagnosis functions

PI(s)/Facility Lead(s): Bin Dong, Jean Luca Bez, Suren Byna

Collaborating Institutions: Lawrence Berkeley National Laboratory, The Ohio State University

Publication(s) for this work: Bin Dong, et al., “AIIO: Using Artificial Intelligence for Job-Level and Automatic I/O Performance Bottleneck Diagnosis”, HPDC 2023, https://doi.org/10.1145/3588195.3592986.

Code Developed or Datasets: https://github.com/hpc-io/aiio

LOCAL LAB POC: Bin Dong, LBNL

TALKING POINTS:

Identifying I/O bottleneck of scientific applications is hard problem
Existing methods only can identify I/O bottlenecks at the system or group level, which can not provide diagnosis for each job
Our work proved that the diagnosis of I/O bottlenecks at the job level is possible.
We used cut-edge AI methods to perform the I/O bottleneck analysis.
E2E (I/O kernels of the Chimera Supernova code and Pixie3D): AIIO finds “Lots of small writes caused by writing a subset of a 3D array”; we improve it by writing subsets to individual files
OpenPMD (Plasma Particle Simulation): AIIO finds “Lots of small writes caused by independent I/O”; we improve it by using collective I/O
DASSA (Subsurface Event, i. e., earthquake Detection): AIIO finds “Each process opens too many files (i.e., 21 files)”; we improve it by merging 21 files into 1 file before read.

METADATA:

Name of the associated awarded project:

PI name(s): Bin Dong (LBNL), Suren Byna (Ohio State)

CITATIONS:

Bin Dong, Jean Luca Bez, and Suren Byna. 2023. AIIO: Using Artificial Intelligence for Job-Level and Automatic I/O Performance Bottleneck Diagnosis. In Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’23), June 16–23, 2023, Orlando, FL, USA. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3588195.3592986

AWARDS:

N/A

REPRODUCIBILITY:

N/A

BACKGROUND AND CONTEXT INFORMATION:

Manually diagnosing the I/O performance bottleneck for a single application (hereinafter referred to as the ``job level'') is a tedious and error-prone procedure requiring domain scientists to have deep knowledge of complex storage systems. However, existing automatic methods for I/O performance bottleneck diagnosis have one major issue: the granularity of the analysis is at the platform or group level and the diagnosis results cannot be applied to the individual application. To address this issue, we designed and developed a method named ``Artificial Intelligence for I/O'' (AIIO), which uses AI and its interpretation technology to diagnose I/O performance bottlenecks at the job level automatically. By considering the sparsity of I/O log files, employing multiple AI models for performance prediction, merging diagnosis results across multiple models, and generalizing its performance prediction and diagnosis functions, AIIO can accurately and robustly identify the bottleneck of an even unseen application. Experimental results show that real and unseen applications can use the diagnosis results from AIIO to improve their I/O performance by at most 146X.