1 of 1

Machine learning-based application grouping and knowledge extraction for HPC I/O

Scientific Achievement

Developed an approach for structuring HPC jobs in a traversable hierarchy,

that is in turn used for efficient knowledge extraction of I/O throughput issues. This framework is available as an opensource tool (Gauge) for HPC system workload exploration.

Significance and Impact

Diagnosing storage system utilization and application I/O patterns is slow and does not scale. Our approach, through a combination of feature engineering, hierarchical clustering and interpretable models provides a holistic framework to quickly analyze groups of similar jobs and extract application I/O insights at different granularity of groupings.

Research Details

Hierarchical clustering to create an easy-to-navigate tree of jobs
ML models predicting I/O throughput trained on clusters of HPC jobs reveal local insight into sources of I/O throughput variance
New training set generation method tackles extrapolation gap
Demonstrated on ALCF Theta system

Fig.2: Visualization of a cluster of jobs, where the blue application significantly outperforms the red one. Gauge helps diagnose these I/O problems.

Fig.1: ALCF Theta Workload Hierarchy

ANL: S. Madireddy, P. Balaprakash, P. Carns, R. Ross

TAMU: M.Kinsy, M. Isakov, E. del Rodario

M. Isakov, E. del Rosario, S. Madireddy, P. Balaprakash, P. Carns, R. Ross, and M. Kinsy, “HPC I/O throughput bottleneck analysis with explainable local models,”

in SC’20: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2020.

Notes:

Due to the scale and evolving complexity of high-performance computing(HPC) systems, there remain critical gaps in our understanding of HPC applications’ runtime behaviors, specifically, compute, communication, and storage behaviors. This situation is further exacerbated by the fact that HPC applications come from a diverse set of scientific domains, can have vastly different characteristics, and are executed simultaneously, thereby contending for shared resources. One of these understanding gaps is the I/O utilization in these systems. Application programmers and systems administrators still heavily rely on limited observations, anecdotes, scatter experiential experiences to develop design patterns for applications and manage their runtime performance either at the node level or system level -often leading to sub-optimal I/O resource utilization or management.

In this project we investigate, design, and prototype an interactive data-driven analysis and interpretation tool for HPC application I/O Performance. Specifically, we (i) examine multiple years’ worth of Darshan logs from the Argonne Leadership Computing Facility’s Theta supercomputer,(ii) perform an automated unsupervised grouping of applications based on their I/O behaviors, and (iii) prototype an illustrative, working, interactive, data-driven analysis, interpretation, and virtualization tool for HPC application I/O performance – called Gauge.

By finding groups of jobs that at first sight are highly heterogeneous but share certain behaviors, we analyze these groups instead of individual jobs, allowing us to reduce the workload of domain experts and automate I/O performance analysis. We conduct a case study where a system owner using Gauge was able to arrive at several clusters that do not conform to conventional I/O behaviors, as well as find several potential improvements, both on the application level and the system level.

Links:

M. Isakov, E. del Rosario, S. Madireddy, P. Balaprakash, P. Carns, R. Ross, and M. Kinsy, “HPC I/O throughput bottleneck analysis with explainable local models,” in SC’20: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2020.
M. Isakov, E. del Rosario, S. Madireddy, P. Balaprakash, P. Carns, R. Ross, and M. Kinsy, “Toward Generalizable Models of I/O throughput,” in ROSS’20: Proceedings of the 10^th International Workshop on Runtime and Operating Systems for Supercomputers. IEEE, 2020.
E. del Rosario, M. Currier, M. Isakov, S. Madireddy, P. Balaprakash, P. Carns, R. Ross, and M. Kinsy, “Gauge: An interactive data-driven visualization tool for HPC application I/O performance analysis,” in 2020 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW), 2020