1 of 15

ADVANCED VISUALIZATION OF POWER, TEMPERATURE, AND ENERGY METRICS IN HPE CRAYEX SYSTEMS

Sustainable HPC State of the Practice Workshop 2024

Authors:

Lavanya L

Stefan Ceballos

September 24, 2024

¹

This is a sample Section Header slide ideal for including a full-slide, impactful picture in the background.

To Replace the Picture on this Sample Slide (this applies to all slides in this template that contain replaceable pictures)

Select the sample picture and press Delete. Click the icon inside the shape to open the Insert Picture dialog box. Navigate to the location where the picture is stored, select desired picture and click on the Insert button to fit the image proportionally within the shape.

Note: Do not right-click the image to change the picture inside the picture placeholder. This will change the frame size of the picture placeholder. Instead, follow the steps outlined above.

Tip: use the Crop tool to reposition a picture within a placeholder. From the Picture Tools Format tab on the ribbon, click the Crop button. Click and drag the picture within the placeholder to reposition. To scale the picture within the placeholder (while Crop is active), grab a round corner handle and drag to resize. Hold Shift key to constrain picture aspect ratio when resizing.

2 of 15

Agenda

²

Introduction to HPCM & Monitoring Framework

Limitations with the traditional approach to monitoring

HPE Clusterview Plugin

Automation of Clusterview Configuration

Challenges with Clusterview Configuration

Visual Clusterview Dashboard Overview

Future Scope

3 of 15

HPE Performance Cluster Manager

All you need to manage your HPE clusters, keep them healthy, and running at peak performance

³

Flexible, easy to use system management solution offering system administrators all the tools they need to turn even the most complex hardware into easily manageable systems capable of accommodating a growing variety of workloads.

Powerful

Comprehensive set of tools you need to manage all aspects of your cluster.

Productive

Designed to maximize productivity of your cluster, automate actions, and optimize running costs.

Scalable

Manage systems from dozens of nodes up to Exascale.

Anywhere

Suitable for both on-premises as well as hybrid HPC deployments.

Flexible

Customize monitoring, alerts, and management actions to best suit your needs.

Proven

Used by hundreds of customers around the globe.

4 of 15

Monitoring Framework

⁴

Data Sources

HPE Cray EX, HPE Cray XD and HPE Apollo

Redfish End points & Sensor Data

Fabric—Slingshot & InfiniBand

Events and Telemetry

WLM—PBS Pro & SLURM

Events and Telemetry

Compute systems

CPU, GPU, Memory, Disk Telemetry and Events

CDU Events and Telemetry

Storage and Filesystems

GlusterFS

Power and Heartbeat Monitoring

Syslog

Data Pipeline

Persistent Storage

Monitoring Framework

Victoria Metrics

Service Infrastructure Monitoring

OpenSearch

Victoria Metrics

Mosquitto

subsmon

telegraf

pcim

Kafka

User Interfaces

Grafana

OpenSearch

Alertmanager

Dashboards

Alerts

Rest API

AIOps

Management hardware and software

Customer Integration

5 of 15

Limitations with the traditional approach to monitoring

⁵

Lack of Granularity: Standard dashboards often fail to provide detailed insights at a fine level (e.g., per-node or component-wise data), making it difficult to pinpoint issues.

Overwhelming Volume of Data: Exascale and pre-exascale systems generate massive amounts of telemetry data (temperature, power metrics, job status, etc.), leading to information overload in traditional dashboards.

Limited Scalability: Standard dashboards may struggle to scale up effectively for systems with tens of thousands of nodes, leading to performance bottlenecks or missing information.

Complexity in Monitoring Failures: Monitoring co-located failures in large clusters becomes difficult, as native panels may not show correlated data effectively.

Inadequate Node Status Tracking: Real-time tracking of node status during system bring-up (e.g., validation tests) is cumbersome without a focused view.

6 of 15

Challenges with the traditional approach to monitoring

⁶⁶

7 of 15

Challenges with the traditional approach to monitoring

⁷

8 of 15

HPE Clusterview Plugin

⁸

KEY FEATURES OF THE HPE CLUSTERVIEW PLUGIN

Hierarchical Data Grouping

Data is organized hierarchically, with each layer of the hierarchy (e.g., chassis, blades, nodes) to show detailed views.

Regex-based Data Parsing

It uses regular expressions (regex) to extract and format the data.

Custom Layouts

It supports various display layouts (e.g., horizontal, vertical, flow, grid), allowing for more complex arrangements that better reflect the structure of an HPC system.

Interactive and Clickable Nodes

Each node can be clickable with customizable URLs, allowing for a more interactive user experience.

The HPE Clusterview plugin for Grafana is designed specifically for high-performance computing (HPC) environments, offering a highly detailed view of various data points in a dense, hierarchical format.

It focuses on spatial representation and the operational hierarchy of the HPC infrastructure

It makes it ideal for large-scale systems where managing and visualizing complex data across multiple nodes is critical.

INTENT OF THE SLIDE: Demonstrate new AIOps feature now included in HPMC 1.5 and onwards�As the HPC systems are growing larger and more complex, the amount of data that operations teams need to analyze to perform their work is also growing exponentially. The Exascale systems (50k-100k servers) will generate 10-100M data points per second.
That is why HPE is deploying Artificial Intelligence to help customers simplify management of their IT operations, uncover issues and react to them faster - AIOps.
How can our customers benefit from using AIOps today?
AIOps are not just for the Exascale systems, you can use them to manage your current system already.
We first introduced AIOPs in HPCM 1.5 – about two years ago offering anomaly detection for CDUs which are notoriously hard to detect issues for as they are rather complex systems.

We evolved a lot since then – we now offer real-time anomaly detection for facility metrics (cooling devices) as well as IT metrics (CPUs, GPUs, memory…).

As a new functionality in HPCM 1.9 we offer anomaly forecasting for cooling devices – “predicting when anomaly in a device might occur 15 minutes ahead of time.

AIOps are only as good as the models used. Our models detect issues better than threshold-based monitoring. So far, we have not had any false positives detected by our methods and we see 50% more anomalies detected by our historical data.
This AIOps feature is just a first example of how HPE makes results of cutting-edge AI initiatives available to broad to all HPC customers as a standard.

9 of 15

Visualizing Slingshot Metrics

⁹

Switches are displayed in group orientation

Green represents nominal, red represents anomalous

HPE Clusterview Plugin

10 of 15

Visualizing Slingshot Metrics

¹⁰

Details of ports inside of a switch

Green represents normal�Red represents anomalous�Grey represents no data

HPE Clusterview Plugin

11 of 15

Visualizing Heartbeat of Aurora Compute Nodes

¹¹

HPE Clusterview Plugin

Nodes are displayed in rack orientation

12 of 15

Challenges with Clusterview Configuration

¹²

Manual Configuration Complexity: Configuring the Clusterview plugin involves �manually writing regex patterns for each group (rack, cabinet, chassis, slot, blade, node) �based on the cluster’s hardware.

Customization for Unique Hardware: Each cluster's configuration is unique, meaning �that regex patterns cannot be duplicated.

Scaling Issues for Large Systems: Large systems, like the Frontier exascale machine with �over 9,400 nodes, demand extensive manual configuration efforts, leading to inefficiencies �and potential delays in system bring-up.

Error-Prone Layout Configuration: Mistakes in the regex setup can lead to incorrect �representations of the system's physical layout. This misrepresentation can cause delays in �identifying genuine issues, particularly in environments with thousands of nodes, such as �exascale systems.

13 of 15

Automation of Clusterview Configuration

¹³

The automated solution addresses the issues by detecting the hardware type and location within the data center, enabling the real-time configuration of dashboards.

The key components of the automation of Clusterview configuration:

Command Line Interface: Users can execute the automation process with specific requirements using the command line tool. This targeted approach ensures that only the necessary dashboards are updated, providing flexibility and improving efficiency.

Data Collection: The framework collects metrics, identifying hardware types and configurations. Detailed data is gathered for nodes, racks, cabinets, chassis, slots, blades, and more, ensuring that the data presented in dashboards is accurate and meaningful.

Data Grouping: Once data is collected, the framework dynamically generates regular expressions (regex) to group nodes at different levels. It automatically creates visual elements like borders and labels, providing a clear and structured view of the node hierarchy.

14 of 15

Future Scope

¹⁴

Enhanced Visual Scaling

Plans to auto-scale node representations based on cluster size and screen orientation, improving readability and aesthetics across different devices.

Integration with Other HPE Solutions

Porting and integrating the solution with future versions of HPE cluster management software like Cray System Management (CSM) and System Monitoring Health.

Automation for Additional Components

The automation framework aims to extend beyond node layouts to cover additional cluster components, including network fabrics, CDUs, and PDUs, allowing for a more comprehensive and automated cluster configuration process.

Open-Source Offering

The goal is to make the automation framework available as an open-source tool, enhancing accessibility alongside the publicly available HPE Clusterview plugin.

15 of 15

Thank You!

Email: lavanya.l@hpe.com