2 of 21

HPCPerfStats (Formally known as TACC Stats)

A tool to collect HPC system-wide performance information from a variety of sources autonomously and with low-overheads

Open source and downloadable, configurable and compatible with general Linux-based computing platforms.

Integrates node-level statistics and job-level accounting to provide a job-centric view of performance and system information.

Texas Advanced Computing Center

The University of Texas at Austin

3 of 21

Performance Metrices Collection

System load and process statistics
FLOP counts of mixed precision
Socket-level memory usage
Swapping and page statistics
System and block device counters
IPC counters
Filesystems usage statistics
Interconnect fabric traffic
Core counters
Uncore counters
Energy counters

Texas Advanced Computing Center

The University of Texas at Austin

4 of 21

Usecases

Job Status and failure analysis
User job history and support context
Application behavior and runtime analysis
Memory usage and performance diagnostics

Texas Advanced Computing Center

The University of Texas at Austin

1. Memory Usage and Performance Diagnostics

Identify Memory high watermark and verify if the application crash with OOM error

Looking at high and low water marks for a job's memory usage so that I can see where issues are being caused/where the job is hanging

Identify memory leak - a line diagonally up

tracked the memory usage per socket throughout a memory-intensive simulation to identify the memory footprint bottleneck that could be circumvented by shared-memory approaches, such as OpenMP

2. Job Status and Failure Analysis

what jobs have failed and succeeded due to node count, node type, runtime, etc.

user has successfully run a job after not being able to due to mfa or some other issue

3. Application Behavior and Runtime Analysis

Whether MPI code ran actually and analyze their resource footprint

Whether IO was happening less or more frequently

Whether code stay active in the entire run?

4. User Job History and Support Context

users history of scaling and how that impacts the job's runtime/success rate

Look at the user and group job history to identify trends in their recent job submissions / look up when they last ran the job successfully - gives us an idea on user's experience level + how to assist the user

FROM STEPHEN:

5 and 6 are not really things that we do, or do well. Especially 5, which is basically XDMOD. AND we aren’t paid on this grant for anything

5. Resource Usage Cost Estimation

Getting a good idea for the typical SU charge for a user's jobs

6. Log and Trace Analysis

data on I/O and memory usage. to go dig in the splunk logs if necessary, so having that link there is also great

5 of 21

Architecture

Rabbit MQ

Timescale DB

Web Server

Data Pipeline

HPCPerfStatsServer

Xalt

Job Scheduler

Data (SLURM)

Collector (Monitor)

Node

Compute

Node

HPC System

Compute

Node

Compute

Node

XDMoD Export

Texas Advanced Computing Center

The University of Texas at Austin

6 of 21

System Statistics : 2025-03

Texas Advanced Computing Center

The University of Texas at Austin

7 of 21

Job Histograms : user

Texas Advanced Computing Center

The University of Texas at Austin

8 of 21

Job Details

Texas Advanced Computing Center

The University of Texas at Austin

9 of 21

High Level Job metrices

Texas Advanced Computing Center

The University of Texas at Austin

10 of 21

System Plot

Texas Advanced Computing Center

The University of Texas at Austin

11 of 21

Host Level Plots

GFLOPS
DRAM (GB/s)
Memory usage (GB)
Fabric (MB/s)
Freq (GHz)
CPU usage (%)
Lustre (MB/s)

Texas Advanced Computing Center

The University of Texas at Austin

12 of 21

CPU Plots (/proc/stat)

Time Spent in

user space
system (kernel space)
nice (low priority process)
idle
iowait
irq (hardware interrupts)
softirq (software interrupts)

Texas Advanced Computing Center

The University of Texas at Austin

13 of 21

Memory

(/sys/devices/system/node/DEV/meminfo)

Physical Memory : MemTotal, MemFree, MemAvailable

Memory Activeness : Active, Inactive,

Writebacks : Dirty, Writeback

Mapped : File Pages, Mapped, AnonPages,

PageTables, NFS_Unstable, Bounce Buffers, Slab

Huge Pages : HugePages_Total/HugePages_Free/AnonHugePages

Texas Advanced Computing Center

The University of Texas at Austin

14 of 21

Lustre : MDC (/proc/fs/lustre/mdc/MOUNT/stats)

ldlm_cancel
mds_getattr
mds_getattr_lock
mds_getxattr
mds_readpage
mds_close
mds_statfs
mds_sync
reqs
wait,us

Texas Advanced Computing Center

The University of Texas at Austin

15 of 21

Network (/sys/class/net/DEV/statistics)

collisions
multicast
{rx, tx}_{bytes, packets, compressed, dropped, errors, fifo_errors}
rx_{frame, length, missed, over, crc}_errors
tx_{aborted, carrier, heartbeat, windows}_errors

Texas Advanced Computing Center

The University of Texas at Austin

16 of 21

Energy : Running Average Power Limit (RAPL)

MSR_PKG_ENERGY_STATUS,mJ (Entire CPU Socket including cores, GPU (uncore), and memory controller)

MSR_PP0_ENERGY_STATUS,mJ (CPU cores)

MSR_DRAM_ENERGY_STATUS,mJ (Memory/RAM sticks)

Texas Advanced Computing Center

The University of Texas at Austin

17 of 21

Detailed Metrices

Type	Recorded Performance Events
block	rd_ios rd_merges rd_sectors,B rd_ticks,ms wr_ios wr_merges wr_sectors,B wr_ticks,ms in_flight io_ticks, ms time_in_queue,ms
cpu	user,cs nice,cs system,cs idle,cs iowait,cs irq,cs softirq
ib	excessive_buffer_overrun_errors link_downed link_error_recovery local_link_integrity_errors port_rcv_constraint_errors port_rcv_data,B port_rcv_errors port_rcv_packets port_rcv_remote_physical_errors
ib_ext	port_select counter_select port_xmit_data,B port_rcv_data,B port_xmit_pkts port_rcv_pkts port_unicast_xmit_pkts port_unicast_rcv_pkts port_multicast_xmit_pkts port_multicast_rcv_pkts
ib_sw	rx_bytes,B rx_packets tx_bytes,B tx_packets
intel_8pmc3	FP_ARITH_INST_RETIRED_SCALAR_DOUBLE FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE FP_ARITH_INST_RETIRED_512B_PACKED_DOUBLE FP_ARITH_INST_RETIRED_SCALAR_SINGLE FP_AR
intel_rapl	MSR_PKG_ENERGY_STATUS,mJ MSR_PP0_ENERGY_STATUS,mJ MSR_DRAM_ENERGY_STATUS,mJ
intel_skx_cha	SF_EVICTIONS_MES LLC_LOOKUP_DATA_READ_LOCAL BYPASS_CHA_IMC_ALL LLC_LOOKUP_WRITE
intel_skx_imc	CAS_READS CAS_WRITES ACT_COUNT PRE_COUNT_MISS
llite	read_bytes, B write_bytes,B direct_read,B direct_write,B osc_read,B osc_write,B dirty_pages_hits, dirty_pages_misses, ioctl open close mmap seek fsync setattr truncate flock getattr statfs alloc_inode

Texas Advanced Computing Center

The University of Texas at Austin

18 of 21

Detailed Metrices

Type	Recorded Performance Events
mdc	ldlm_cancel mds_close mds_getattr mds_getattr_lock mds_getxattr mds_readpage mds_statfs mds_sync reqs wait,us
mem	MemTotal,B MemFree,B MemUsed,B Active,B Inactive,B Dirty,B Writeback,B FilePages,B Mapped,B AnonPages,B PageTables,B NFS_Unstable,B Bounce,B Slab,B AnonHugePages,B HugePages_Total HugePages_Free
net	collisions multicast rx_bytes,B rx_compressed rx_crc_errors rx_dropped rx_errors rx_fifo_errors rx_frame_errors rx_length_errors rx_missed_errors rx_over_errors rx_packets tx_aborted_errors tx_bytes,B
numa	numa_hit numa_miss numa_foreign interleave_hit local_node other_node
osc	read_bytes,B write_bytes,B ost_destroy ost_punch ost_read ost_setattr ost_statfs ost_write reqs wait,us
ps	ctxt processes load_1 load_5 load_15 nr_running nr_threads
sysv_shm	mem_used,B segs_used
tmpfs	bytes_used,B files_used
vfs	dentry_use file_use inode_use
vm	nr_anon_transparent_hugepages pgpgin,B pgpgout,B pswpin pswpout pgalloc_normal pgfree pgactivate pgdeactivate pgfault pgmajfault pgrefill_normal pgsteal_normal pgscan_kswapd_normal pgscan_direct_norma

Texas Advanced Computing Center

The University of Texas at Austin

19 of 21

Recent Developments

Architecture Support: Enabled compatibility with the latest CPU and GPU architectures, including Sapphire Rapids and Grace Hopper.
Containerization: Provided a containerized build of HPCPerfStats. This has reduced the steps to install this software dramatically. This new install method also provides an easily deployable development environment.
Authentication: Implemented secure authentication mechanisms for accessing the containerized version.
Software Upgrades: Updated all dependent packages, resolved warnings and errors from the upgrade, migrated to Python 3.12 within the container, and upgraded Django.
Visualization: Introduced Roofline Plot for performance analysis.
Optimization: Pipeline runs 5x as fast. We will continue to look for optimizations in preparation for the data amount from Horizon.
Queue Filtering: Allows for obscuration of queues based on the config file.
Documentation/Config File: Enhanced documentation to better explain performance metrics. Added many new settings/options to the config file with documentation.

Bug Fixes:

Multiple edge cases with corrupt data in the pipeline.
Race conditions in the parallization code.

Texas Advanced Computing Center

The University of Texas at Austin

20 of 21

Outreach

“An introduction to TACC HPCPerfStats” by Amit Ruhela, In 2025 Improving Scientific Software Conference is organized by the UCAR Software Engineering Assembly (SEA), Boulder, CO

CaRCC Researcher Facing call titled "Effective Compute Resource Utilization",

Panel: Paul Fischer & Jay McKay(University of Utah), Jon Halverson & Josko Plazonic(Princeton), Julia Mullen(MIT), Amit Ruhela(TACC for HPCPerfStats), Joe White(University at Buffalo for XDMod)

Conference Submission: HPC Monitoring Data Analysis: A Case Study of TACC Monitoring Data Volatility and Stability; Authors: Tianyi Cong, Stephen Lien Harrell, Dorian Arnold, Submitted in HPCSYSPROS25 - https://sighpc-syspros.org/workshops/2025

Texas Advanced Computing Center

The University of Texas at Austin

1 of 21

2 of 21

3 of 21

4 of 21

5 of 21

6 of 21

7 of 21

8 of 21

9 of 21

10 of 21

11 of 21

12 of 21

13 of 21

14 of 21

15 of 21

16 of 21

17 of 21

18 of 21

19 of 21

20 of 21

21 of 21