Workflow Anomaly Detection

with Graph Neural Networks

Hongwei Jin¹, Krishnan Raghavan¹, George Papadimitriou², Cong Wang³, Anirban Mandal³,

Patrycja Krawczuk², Loïc Pottier², Mariam Kiran⁴, Ewa Deelman², Prasanna Balaprakash¹

​

¹Argonne National Laboratory, ²University of Southern California,

³Rennaissance Computing Institute, ⁴Lawrence Berkeley National Laboratory

This work was funded by the US Department of Energy under contracts DE-SC0022328 and DE-AC02-06CH11357.

17th Workshop on Workflows in Support of Large-Scale Science, Dallas, TX, 2022

Anomaly Detection in Scientific Workflows

Motivation:

• Detecting and diagnosing anomalies are both import and challenge for reliable scientific workflows
• Model the scientific workflow as a directed acyclic graph (DAG)

​

​

​

​

​

​

Our proposal :

• Apply graph neural networks (GNNs) to detect anomalies
• Anomaly detection on entire graph (workflow)
• Anomaly detection on nodes (jobs)

*

SC22 | Dallas, TX | hpc accelerates.

2

Figure: an abstract view of workflow as DAG

DAG

G=(A, X)

A: job dependencies (graph structure)

X: job features (node features)

classification

Data Collection

• We use Pegasus ¹, an exemplar workflow management system, enables users to design workflows at a high level of abstraction of resources available to execute.
• Collect job features such as ready time, prescript delay, runtime, postscript delay, queue delay, stage-in delay, stage-out delay, and etc. And normalize features along jobs.
• Data
• Normal. No anomaly is introduced normal conditions.
• CPU. 2–3 cores are occupied by the stress tool on each worker node.
• HDD. 50–100 MB of data are continuously written by the stress tool on each worker node.
• Packet loss. The network connection between two ExoGENI regions is experiencing 0.1%–5.0% of packet loss.

​

*

SC22 | Dallas, TX | hpc accelerates.

3

1. Deelman, E., et al. The Evolution of the Pegasus Workflow Management Software. Computing in Science Engineering 2019.

Graph Neural Networks

• Two-layer graph convolutional layer (GCN) aggregate the job information by propagation from neighbors
• From the hidden representation (H), we take a two-layer MLP for further classification
• Objective function is measured from cross-entropy loss and optimized in Adam

*

SC22 | Dallas, TX | hpc accelerates.

4

Figure: GNN architecture

• One of the recent deep learning approaches to learn the representation of structured data
• Learn the node representation based on both node feature and local structural information

Experiments

*

SC22 | Dallas, TX | hpc accelerates.

5

Single model trains multiple workflows simultaneously

Tables: A collection of results on anomaly detection (classification)

Binary: normal vs anomaly

Multi labels: based on anomaly category

Experiments (cont’d)

Detailed anomaly detection by anomaly categories

Detailed anomaly detection by anomaly level

​

HDD anomaly level

(60, 100 MB data are continuously written by stress tool)

Effectiveness and Efficiency

Classical ML models

Computer vision DL models ¹

Measure on single GPU

Trustworthy prediction based on the confusion matrices

1. Krawczuk, P. et al. Anomaly Detection in Scientific Workflows using End-to-End Execution Gantt Charts and Convolutional Neural Networks. EARC’21.

Conclusion and Future Works

Conclusion

• Map the scientific workflow as directed acyclic graphs (DAG) and learn the graph-level and node-level information through graph neural networks.
• GNN approach surpasses the conventional machine learning approaches and computer vision approaches with computation efficiency and higher accuracy in both graph and node level classification task.

​

Future Works

• Investigate to analyze larger scientific workflows in terms of efficiency
• Generalize the model from one learned workflow to another (transfer learning)
• Synthetic generation for anomaly data (generative models)
• On-line detection as workflow computes

​

*

SC22 | Dallas, TX | hpc accelerates.

8

Q&A

9

Contact information

Hongwei Jin: jinh@anl.gov