1 of 1

�

Scientific Achievement

We have developed a policy framework and middleware for resource elasticity in in-situ analysis and visualization to reduce load imbalance. Load imbalance is a significant source of inefficiency in HPC applications, but can develop and resolve dynamically, making it difficult to predict and resolve a priori.

Our adaptive elasticity framework leverages the elastic capabilities of many in-situ analysis and visualization resources to resolve dynamically-developing load imbalance by adaptively reassigning resources or even reallocating resource when supported by the underlying system.

Significance and Impact

Adaptive elasticity reduces inefficiency in the growing class of elastic in-situ analysis and visualization workflows. This makes these workflows more resilient to load imbalance, improving performance and saving both time and core hours. When evaluating for an in-situ visualization of a deep-water asteroid impact model, a 64% reduction in overall execution was observed, as well as a 41% reduction in overall core hours.

Visualization of a distributed iso-surface analysis (right). This analysis is performed in-situ, iteratively upon the output of a time-stepping Gray-Scott Reaction/Diffusion simulation. At different timesteps, different ranks of the analysis take longer to perform the analysis (below). This dynamic load imbalance slows the collective analysis.

Technical Approach

Developed an analytic policy to predict dynamically-developing load-imbalance
Added rebalancing phase to workflow, in which processes report timing data, and initiate elastic rebalancing if autonomic trigger has fired
Created autonomic trigger interface for data staging framework, directing rebalancing and reallocation of job resources

Wang, Z., Dorier, M., Subedi, P., Davis, P. E., & Parashar, M. (2023). Adaptive elasticity policies for staging-based in situ visualization. Future Generation Computer Systems, 142, 75-89.

Adaptive Elasticity for In Situ Analysis and Visualization

Runtime (left) and core-hour consumption (right) for various elasticity strategies with an in-situ deep-water impact visualization.

TALKING POINTS:

Load balance between processes is a significant source of inefficiency in HPC applications, and can develop dynamically over the course of an application’s run.
Many HPC applications and systems have or could have resource elasticity – the ability to resize applications or allocations
We have developed a model for predicting when elasticity can be used to reduce imbalance, and software to autonomically rebalance resources.
Evaluation shows that adaptive elasticity improves performance for many classes of scientific applications, improving time to results and reducing compute hour consumption.

METADATA:

Utah: Pradeep Subedi, Philip Davis, Manish Parashar

Rutgers: Zhe Wang

Argonne: Matthieu Dorier

CITATIONS:

Wang, Z., Dorier, M., Subedi, P., Davis, P. E., & Parashar, M. (2023). Adaptive elasticity policies for staging-based in situ visualization. Future Generation Computer Systems, 142, 75-89.

Wang, Z., Dorier, M., & Parashar, M. (2022, November). Research Perspectives Toward Autonomic Optimization of In Situ Analysis and Visualization. In 2022 IEEE/ACM International Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV) (pp. 7-13). IEEE.

Wang, Z., Subedi, P., Dorier, M., Davis, P. E., & Parashar, M. (2021, December). Adaptive Placement of Data Analysis Tasks For Staging Based In-Situ Processing. In 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC) (pp. 242-251). IEEE.

Wang, Z., Dorier, M., Subedi, P., Davis, P. E., & Parashar, M. (2021, November). An adaptive elasticity policy for staging based in-situ processing. In 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS) (pp. 33-41). IEEE.