1 of 19

LARGE-SCALE WORKFLOW

PROVENANCE DATA MANAGEMENT

IN THE AI LIFECYCLE USING FLOWCEPT

SEP 15TH, 2025

AMAL GUEROUDJI (ANL)

RENAN SOUZA (ORNL)

DANIEL ROSENDO (ORNL)

RAFAEL FERREIRA DA SILVA (ORNL)

MATTHIEU DORIER (ANL)

2 of 19

OUTLINE 

  • Part One:
    • Introduction: what is provenance and why is it important? 
    • Overview on Flowcept
      • Concepts
      • Architecture
      • Set up 
    • Hands-on: Simple examples on provenance capture and analysis
    • Break [30 minutes]

​

​

3 of 19

OUTLINE

  • Part Two:
    • Collecting Provenance and Resource Consumption  Data
      • Capturing data at workflow, task, model, and layer levels
      • Monitoring CPU, GPU, memory, and storage metrics
    • Provenance Data Processing and Analysis
      • Techniques for handling and querying collected data
      • Extracting insights for optimizing ML training
    • Provenance Querying 
      • Setting up environment
      • Simple CLI queries
      • Jupyter Notebooks
      • Grafana
    • Flowcept Agent Demonstration
    • Interactive Q&A and Closing Remarks (15 minutes)

4 of 19

WHAT IS PROVENANCE AND WHY IS IT IMPORTANT?

  • The word "Provenance" comes from the verb "Provenir" in French, which means "Come Forth" 
  • The first known use of the word "Provenance" was in 1785, and it referred to the origin or source of something

​

  • Workflow Provenance refers to records of the application's execution. It may contain information about the workflow/data creation, task performance at runtime, task inputs and outputs, the environment, etc...

What does provenance mean?

5 of 19

WHAT IS PROVENANCE AND WHY IS IT IMPORTANT?

Understandability and Explainability:

    • Where the data comes from?
    • Why is there performance variability
    • How the final results got computed?
    • What execution path was followed
    • Which agents/tasks got impacted by a hallucination?
    • Where the error started getting propagated?
    • …?

Why is it important?

6 of 19

WHAT IS PROVENANCE AND WHY IS IT IMPORTANT?

  •  The need for multiple tools to collect performance and provenance
  • The need to correlate data from multiple sources for deep understanding
  • No support for multi-workflow scenarios
  • Challenging integration of heterogeneous workflows (ML, Data analysis, AI Agents .. )

Provenance and Performance Collection Challenges

7 of 19

FLOWCEPT OVERVIEW

Definitions

  • Flowcept is a data-driven, multi-workflow provenance and performance tool 
  • Data 
  • Task
  • Workflow
  • Message Queue
  • Persistence

​

8 of 19

FLOWCEPT OVERVIEW

Architecture

  • Data collection:
    • Observability Adapter 
    • Code Instrumentation
  • Message Queues:
    • Redis
    • Kafka
    • Mofka
  • Persistence:
    • MongoDB
    • LMDB
  • Visualization and Queries:
    • Python API, CLI
    • Grafana
    • Flowcept Agent

​

9 of 19

FLOWCEPT SETUP AND QUICK START

Setup and quick Start

Goal: Get a working environment.

Follow these steps: 

    • Create Conda Env
    • Install Flowcept
    • Try out a small example

​

https://flowcept.readthedocs.io/en/latest/quick_start.html

10 of 19

COLLECTING PROVENANCE AND RESOURCE CONSUMPTION  DATA

Capturing data at workflow, task, model, and layer levels

Goal: Get a good understanding of the type of data to capture and how.

    • Capture methods:
      •  Decorators
      •  Loop instrumentation
    • Configuration
    • Telemetry Capture:
      • Workflow
      • Task
      • Model
      • Layer

11 of 19

HANDS-ON: COLLECTING PROVENANCE AND RESOURCE CONSUMPTION  DATA

Monitoring CPU, GPU, memory, and storage metrics

Goal: select the type of telemetry to capture:  

    •  Telemetry Capture:
      • CPU
      • GPU
      • Memory usage
      • Storage
      • Network

12 of 19

BREAK FOR 30 MINUTES 

13 of 19

INTRODUCTION TO THE TUTORIAL 

  • Part Two:
    • Provenance Data Processing and Analysis
      • Techniques for handling and querying collected data
      • Extracting insights for optimizing ML training
    • Interactive Analysis via Jupyter Notebook
      • Interpreting provenance for decision-making
    • Flowcept Agent Demonstration
    • Interactive Q&A and Closing Remarks (15 minutes)

​

​

​

​

​

14 of 19

PROVENANCE DATA PROCESSING AND ANALYSIS 

Techniques for handling and querying collected data

Goal: Get a good understanding of the data and how to process it.

15 of 19

PROVENANCE DATA PROCESSING AND ANALYSIS 

Goal: Play with a real life example

  • Instrument the code
  • change the yaml file
  • collect provenance and performance data

​

16 of 19

FLOWCEPT AGENT DEMONSTRATION

Using LLM Agents to query Flowcept Data 

Goal: Demonstration of using LLM agents to query Flowcept data

  • Workflow overview
  • Tested LLMs
  • Demo

17 of 19

INTERACTIVE Q&A AND CLOSING REMARKS

18 of 19

INTERACTIVE Q&A AND CONCLUDING REMARKS 

​

​

​

​

​

Thank you for joining our Tutorial, and please don’t forget to fill the evaluation form to help us improve next editions

19 of 19