1 of 36

Work Package 3

Pulsar Network: Distributed heterogeneous compute

Mira Kuntz (ALU-FR), Marco A. Tangaro (CNR)

11th January 2024

2022-12-31 by Name Surname

Grant agreement 101057388

2 of 36

Introduction and Motivation

National Cloud and HPC infrastructures have been established, with differences in

Hardware
Configuration
Software stack
Authentication and Authorization
Access typically targeted at local researchers.

WP3 goal: provide efficient and structured access to data, tools and workflows supported by suitable IT infrastructures.

Presentation title | Name Surname

3 of 36

Introduction and Motivation

WP3 goal: provide efficient and structured access to data, tools and workflows supported by suitable IT infrastructures.

Easily deployable for Providers

Open Source Software solutions
No need to force partners to change their Authentication and authorization system.
Automated (and documented) procedures for deploying and maintaining compute endpoints.

Presentation title | Name Surname

4 of 36

Introduction and Motivation

WP3 goal: provide efficient and structured access to data, tools and workflows supported by suitable IT infrastructures.

Open Source Software usage
Supports researchers needs, by supporting different Workflow Management Systems (Galaxy, but also Nextflow and CWL).
Transparent and user friendly access to compute and storage resources.
Improve the availability of tools, workflows and reference data.

Easily available to Users

Presentation title | Name Surname

5 of 36

(a bit of) Technical background…

A User friendly interface to workflows, tools, compute and storage resources:

-> The Galaxy Project and UseGalaxy.* servers.

Grant users access to Compute Infrastructures where they are available, regardless of their location and underlying architecture:

-> Pulsar application

the Galaxy Project’s remote job execution system.
Consist of a Python server application that accepts jobs from a Galaxy server, submitting them to a local resource and then sends the results back to the originating Galaxy server once processed.
Support for different resource managers (HTCondor, SLURM, K8s).

-> The Advanced Resource Connector

Grid compute middleware providing an interface for submission of computational task to different computing systems (HTCondor, SLURM, K8s on development).
Already adopted in different research field, e.g. High Energy Physics.
Galaxy support developed in WP4.

Presentation title | Name Surname

6 of 36

The Pulsar Network

The Pulsar Network is a distributed job execution system, allowing to scale the computing resources available to Galaxy instances over heterogeneous and distributed compute facilities.

6 national Galaxy instances will make use of the Pulsar Network.
At least 10 Pulsar endpoints, routing the incoming jobs from Galaxy and other workflow management systems to local compute resources.
Enable Nextflow and CWL job submission on the distributed computing network.

Presentation title | Name Surname

7 of 36

The Pulsar Network architecture

Pulsar is deployed on the compute infrastructure, fetching required data from the Galaxy server when a job is submitted and uploading results back once the job is successful -> this is transparent to the user, who just needs its Galaxy authentication and nothing else to run jobs on a remote Pulsar.

Software tools and Reference data (e.g. genomic sequences and annotations needed by many tools) are readily available at the Pulsar destination through the CVMFS repository.

CVMFS

Presentation title | Name Surname

8 of 36

The Pulsar Network architecture

Software tools and Reference data (e.g. genomic sequences and annotations needed by many tools) are readily available at the Pulsar destination through the CVMFS repository.

Presentation title | Name Surname

9 of 36

Local infrastructure

Remote HPC infrastructure with Pulsar

Remote Cloud infrastructure with Pulsar deployed with Open Infrastructure

Remote HPC and Cloud infrastructure with ARC

Pulsar queue

Other queue

CVMFS

10 of 36

An Open Infrastructure

Set of tools to have ready-to-go Pulsar endpoints and Galaxy servers easily deployable by Cloud infrastructure providers;

Enable consortium partners (and beyond) to deploy new Pulsar nodes to further extend the computing capacity of the network, and/or Galaxy public servers to exploit the network compute power.

Presentation title | Name Surname

11 of 36

An Open Infrastructure

A virtual machine image, named Virtual Galaxy Compute Nodes, provides everything is needed to run Galaxy jobs.
Containerized tools and Reference Data shared through read only repositories based on CERN-VM FileSystem.
Terraform for configuring Cloud infrastructures, including compute, network and storage services.
Ansible automation engine to perform Pulsar and/or Galaxy configuration and update routines.

HTCondor serves as default workload management system for dedicated resources

Presentation title | Name Surname

12 of 36

An Open Infrastructure

A virtual machine image, named Virtual Galaxy Compute Nodes, provides everything is needed to run Galaxy jobs.
Containerized tools and Reference Data shared through read only repositories based on CERN-VM FileSystem.
Terraform for configuring Cloud infrastructures, including compute, network and storage services.
Ansible automation engine to perform Pulsar and/or Galaxy configuration and update routines.

Tools are packaged with Conda and Docker and Singularity (Apptainer) containers.

CernVM File System, a distributed read-only file system, grants the possibility to made tools and reference data already available to the job destination.

Presentation title | Name Surname

13 of 36

An Open Infrastructure

A virtual machine image, named Virtual Galaxy Compute Nodes, provides everything is needed to run Galaxy jobs.
Containerized tools and Reference Data shared through read only repositories based on CERN-VM FileSystem.
Terraform for configuring Cloud infrastructures, including compute, network and storage services.
Ansible automation engine to perform Pulsar and/or Galaxy configuration and update routines.

Terraform is a software for creating and managing virtual infrastructures by exploiting machine-readable configuration files.

Presentation title | Name Surname

14 of 36

An Open Infrastructure

A virtual machine image, named Virtual Galaxy Compute Nodes, provides everything is needed to run Galaxy jobs.
Containerized tools and Reference Data shared through read only repositories based on CERN-VM FileSystem. .
Terraform for configuring Cloud infrastructures, including compute, network and storage services.
Ansible automation engine to perform Pulsar and/or Galaxy configuration and update routines.

Ansible is an open-source software that automates cloud configuration management, application deployment and service orchestration.

Presentation title | Name Surname

15 of 36

The Pulsar Network

Pulsar endpoints deployment is completely automated through the Open Infrastructure.

Requirements:

OpenStack tenant access
UseGalaxy.eu RabbitMQ credentials.

and the endpoint is created with just one command.

terraform apply -var "pvt_key=~/.ssh/<key>" -var "condor_pass=<condor-passord>" -var "mq_string=pyamqp://<pulsar>:<password>@mq.galaxyproject.eu:5671//pulsar/<pulsar>?ssl=1"

Presentation title | Name Surname

16 of 36

The Pulsar Network

Task 3.1 - Develop and maintain an Open Infrastructure based deployment model for Pulsar endpoints (M1-M36)

Open Infrastructure updates:

Update images to RockyLinux 9 (https://usegalaxy.eu/static/vgcn/)
Pulsar app updates.
HTCondor update to 10.4.3 (new authentication mechanism between compute nodes and the Central Manager).
Continuous configuration automation: https://github.com/usegalaxy-eu/pulsar-deployment
Just one command to setup a working Pulsar endpoint.
Documentation update

Presentation title | Name Surname

17 of 36

The Pulsar Network

Task 3.3 - Build an European-wide network of Pulsar sites (M7-M36)

Many endpoints have been already updated:

ALU-FR (DE) Pulsar deployments for testing
CNR (IT) - IT02 and IT03
IISAS (Slovakia) - SK01
CNRS - GenOuest (FR) - FR01
CESNET (Czech Republic) - CZ01
EGI and INFN - EGI01
BSC (ES) - ES01
TUBITAK ULAKBIM setup is almost completed

… and counting.

Presentation title | Name Surname

18 of 36

UseGalaxy.* instances

Goal: ensure reproducible analyses across Galaxy instances, exploiting National and cross countries computing resources.

National Galaxy instances:

Fr, Be, No, Es, Cz and It

Presentation title | Name Surname

19 of 36

UseGalaxy.* instances

The Open Infrastructure allows to deploy full fledged usegalaxy.eu replica servers, thus allowing to instantiate new usegalaxy services easily, but also providing a robust framework for maintaining and updating running instances.

Started draft documentation (temporary repository):

https://usegalaxy-it.github.io/documentation/

Presentation title | Name Surname

20 of 36

UseGalaxy.* instances

Task 3.5 - Developing and maintaining national or domain-driven Galaxy servers (M1-M36)

Currently we have 7 endpoints: EU (ALU-FR), Be (VIB), Fr (CNRS), Es (BSC-CNS), No (ELIXIR-NO), Cz (CESNET) and It (CNR).

Align to Galaxy 23.0+ release
Switch to Total Perspective Vortex dynamic job destination tool (https://arxiv.org/abs/2312.02060)
Pulsar endpoints integration
OS image and PostgreSQL updates

(see Backup slides for full list of activities on UseGalaxy.* servers)

Presentation title | Name Surname

21 of 36

Other Workflow engines support

GOAL - Enable other workflow engines to leverage the European Pulsar Network.

HOW - Implement support for the GA4GH Task Execution Service, allowing other services to submit jobs via TES to Pulsar and therefore to the European Pulsar Network.

The Task Execution Service (TES) API is an effort to define a standardized schema and API for describing batch execution tasks. A task defines a set of input files, a set of containers and commands to run, a set of output files, and some other logging and metadata.

Open standard for describing how to run command line tools and connect them to create workflows.

Nextflow is a workflow system for creating scalable, portable, and reproducible workflows.

Presentation title | Name Surname

22 of 36

Other Workflow engines support

Task 3.2 - Add the GA4GH Task-Execution-Service (TES) API to Pulsar (M1-M12)

Task 3.4 - Add TES support to WfExS (Workflow Execution Service) (M18-M36)

WfExS is a high-level workflow execution service backend, developed within EOSC-Life as part of Demonstrator 7 (D7), which can manage workflows across different domains.

Fetches workflows from WorkflowHub.
identifies the workflow type and run it using its native workflow execution engine (currently CWL and NextFlow).
Identifies the containers needed by the workflow and fetches them.
Optionally describes the results with a RO-Crate and makes them available to users.

TESP (TES for Pulsar) is a separate microservice:

implemented the TES standard
distributing TES tasks using Pulsar REST API
tasks started in Docker containers
TESP implementation, usage via curl or Snakemake
three protocols for file transfer (S3, https, ftp)

Presentation title | Name Surname

23 of 36

Presentation title | Name Surname

24 of 36

Deliverables and Milestones

WP	Deliverable/Name	Lead	Due date
WP3	D3.1 Operations documentation on the Open Infrastructure deployment	INFN	M24
WP3	D3.2 Publication on the Pulsar Network, integrated in workflow management systems.	CNR	M36

WP	Milestone/Name	Means of verification	Due date
WP3	M3.1 Pulsar network is TRL-9: operational in environment	Service available	M36
WP3	M3.2 Demonstrated job submission via the WfExS on the Pulsar Network.	Service available	M36
WP3	M3.3 National Galaxy servers reaching TRL-9 (operational in environment)	Service available	M36

Presentation title | Name Surname

25 of 36

Conclusions & Future outlook

Short-term goals:

Release the documentation for both Pulsar and (Use)Galaxy endpoint deployment.
Complete the Pulsar endpoints deployment.
Connect more UseGalaxy.* to the Pulsar network.

Long-term objectives:

Use the Open Infrastructure with commercial Cloud providers.
Other workflow management systems will be enabled to submit jobs to this distributed compute network.

Presentation title | Name Surname

26 of 36

Thank you for your attention!

Marco Antonio Tangaro (CNR)

Federico Zambelli (CNR and UniMi)

Bjoern Gruening (ALU-FR)

Mira Kuntz (ALU-FR)

Sanjay Kumar Srikakulam (ALU-FR)

Stefano Nicotri (INFN)

María Chavero-Díez (BSC-CNS)

Josep Ll. Gelpi (UB)

Anthony Bretaudeau (CNRS)

Eva Mercier (CNRS)

Work package 3 contributors:

Hakan Bayindir (TUBITAK ULAKBIM)

Jan Astalos (IISAS)

Viet Tran (IISAS)

Sebastian Luna-valero (EGI)

Lukasz Opiola (AGH-UST)

Olivier Collin (CNRS)

Miroslav Ruda (CESNET)

Josef Handl (CESNET)

Presentation title | Name Surname

27 of 36

Backup

Presentation title | Name Surname

28 of 36

Work Package 3 - Task 3.2

TESP (TES for Pulsar) is a separate microservice, decoupled from the Pulsar

implementing the TES standard
distributing TES tasks to Pulsar applications
currently using Pulsar REST API
tasks started in Docker containers (+ Docker compose usage)

The current version (https://github.com/CESNET/tesp-api) provides

TESP implementation, usage via curl or Snakemake
three methods for file transfer (S3, https, ftp)
support for development version of Galaxy TES runner

Add the GA4GH Task-Execution-Service (TES) API to Pulsar (M1-M12)

Presentation title | Name Surname

29 of 36

Work Package 3 - Task 3.2

A large effort was dedicated to testing compatibility with the development effort on Galaxy TES Runner. (https://github.com/galaxyproject/galaxy/pull/14462).

Current issues:

due to the development status of this Runner, there are currently limitations to this approach - workflow from Galaxy can be started on Pulsar node, but workflow output is not transferred back to Galaxy.
main limitation of the current version of Galaxy runner is missing support for creation and specification of Docker image, which should contain all tools required by specified workload.

Add the GA4GH Task-Execution-Service (TES) API to Pulsar (M1-M12)

Presentation title | Name Surname

30 of 36

Work Package 3 - Task 3.4

WfExS is a high-level workflow execution service backend, developed within EOSC-Life as part of Demonstrator 7 (D7), which can manage workflows across different domains.

It has a strong focus on reproducible and replicable analysis by using digital objects like RO-Crate.

Fetches workflows from WorkflowHub.
identifies the workflow type and run it using its native workflow execution engine (currently CWL and NextFlow).
Identifies the containers needed by the workflow and fetches them.
Optionally describes the results with a RO-Crate and makes them available to users.

Task 3.4 - Add TES support to WfExS (Workflow Execution Service) (M18-M36)

Presentation title | Name Surname

31 of 36

Work Package 3 - Task 3.5

EU (ALU-FR):

Leads the development of the Open Infrastructure framework.
Upgraded from Galaxy 23.0 to Galaxy 23.1.
Switched from Sorting Hat to Total Perspective Vortex (TPV) meta-scheduler ( https://galaxyproject.org/news/2023-05-08-tpv-switch/).
Onboarded multiple Pulsar endpoints from diverse partners.
Added a ‘remote resources’ dropdown menu to the user preferences that lets users select a specific pulsar endpoint and integrate it with TPV, thus enabling the scheduling/distribution of jobs to the respective pulsar endpoints.
Enabled deployment of national and domain-driven Galaxy instances by knowledge transfer and support.

Developing and maintaining national or domain-driven Galaxy servers (M1-M36)

Presentation title | Name Surname

32 of 36

Work Package 3 - Task 3.5

IT (CNR):

Test instance has been deployed with the OI (Galaxy 23.0) with a pulsar endpoint.
The infrastructure automation framework and documentation are hosted on Github:

Usegalaxy-it test instance: https://usegalaxy-it.ext.cineca.it
Usegalaxy-it github repository: https://github.com/usegalaxy-it
Usegalaxy-it installation documentation: https://usegalaxy-it.github.io/documentation/

New hardware resources have been acquired in the context of the PON project CNR.BiOmics, while more will be acquired in the context of the ELIXIRxNextGenIT RRF project. Some of those resources will be dedicated to the UseGalaxy.it server, whose main production instance will be deployed at the ReCaS-Bari data center.

Developing and maintaining national or domain-driven Galaxy servers (M1-M36)

Presentation title | Name Surname

33 of 36

Work Package 3 - Task 3.5

Be (VIB):

The galaxy version was upgraded from 21.01 to 23.0
The postgres version was upgraded from 9 to 15
The OS was upgraded from CentOS 7 to Rocky Linux 8
Started migration to Total Perspective Vortex
Configured rabbitMQ
Added a Pulsar endpoint (still requires some more testing)

Developing and maintaining national or domain-driven Galaxy servers (M1-M36)

Presentation title | Name Surname

34 of 36

Work Package 3 - Task 3.5

Fr (CNRS)

Upgraded to version 23.0
Upgraded to latest TIaaS version
Started migration to Total Perspective Vortex
Installed and configured tools produced by the Biodiversity use case of WP5

Developing and maintaining national or domain-driven Galaxy servers (M1-M36)

Presentation title | Name Surname

35 of 36

Work Package 3 - Task 3.5

Cz (CESNET):

CESNET deployed a production version on https://usegalaxy.cz, as a collaboration of e-INFRA CZ and ELIXIR CZ.

Version 23.0, with TPV support, aiming to deploy the majority of tools installed on usegalaxy.eu.
Installation with Pulsar, connecting to e-infrastructructure PBSPro, which provides dedicated compute nodes (including GPU nodes) to Galaxy.
Currently several Pulsar nodes are installed, to support usegalaxy.cz, usegalaxy.eu and another installation specific to ELIXIR services, looking for better support in one installation.
Specific AAI setup, usegalaxy.cz provides both Lifescience AAI (to support ELIXIR users) and e-INFRA CZ AAI (users of national e-infrastructure).

Developing and maintaining national or domain-driven Galaxy servers (M1-M36)

Presentation title | Name Surname

36 of 36

Work Package 3 - Task 3.5

Es (BSC-CNS):

Instance currently implemented through Openstack, utilizing the cloud resources available at BSC.

Setup of a PostgreSQL database.
CVMFS for reference data.
Over 1000 different pre-installed tools.
Slurm as workload manager.
Working to expand its capacity and availability, through additional Slurm.
The deployment of a Pulsar endpoint is already scheduled

Developing and maintaining national or domain-driven Galaxy servers (M1-M36)

Presentation title | Name Surname