EuroScienceGateway Kick-Off Meeting

6th-7th October 2022, Freiburg

Work package 3 - Pulsar Network: Distributed heterogeneous compute

• National HPC and Cloud infrastructures have been established, with differences in
• Hardware
• Configuration
• Software stack
• Access typically targeted at local researchers.
• Different needs for researchers, depending on, for example:
• Local infrastructure availability and accessibility
• Sensitivity of the data
• Experience & skills
• The global pandemic has reshaped the way we look at biological data handling: prompt, straightforward, efficient and structured access to data, tools and workflows supported by suitable IT infrastructures is becoming increasingly critical for researchers.

1

EuroScienceGateway | Kick-Off Meeting

Work Package 3 - Pulsar Network

A User friendly interface to workflows, tools and compute and storage resources:�-> The Galaxy Project and UseGalaxy.eu

Grant users the access to Compute Infrastructures, regardless of the underlying infrastructure:

-> Pulsar

• the Galaxy Project’s remote job execution system.
• It is a Python server application that accepts jobs from a Galaxy server, submitting them to a local resource and then sending the results back to the originating Galaxy server.
• Support for different resource managers (HTCondor, SLURM, K8s).

Grant access to Tools and Reference data:

-> CernVM File System: distributed read-only file system.

2

EuroScienceGateway | Kick-Off Meeting

Work Package 3 - Pulsar Network

3

NO01, BE01, IT01...

Local infrastructure

Remote compute clusters

EuroScienceGateway | Kick-Off Meeting

Work Package 3 - Pulsar Network

4

https://pulsar-network.readthedocs.io

EuroScienceGateway | Kick-Off Meeting

Objectives

O3.1 - Build an European wide job-scheduling network

• T3.1, T3.3, T3.4 and T3.5

O3.2 - Make Pulsar endpoints conform to GA4GH Task Execution service standard

• T3.2

O3.3 - Deploying a TRL-9 web service to access the Pulsar Network

• T3.2 and T3.5

What the WP is planning to achieve?

5

EuroScienceGateway | Kick-Off Meeting

Objectives

• At least 10 Pulsar endpoints, routing the incoming jobs from Galaxy and other workflow management systems to local compute resources.
• 6 national Galaxy instances that will make use of the Pulsar Network

​

What the WP is planning to achieve?

​

6

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.1

Develop and maintain an Open Infrastructure based deployment model for Pulsar endpoints (M1-M36)

Task Lead: INFN

Task Members: ALU-FR, CESNET, CNR, IISAS

Goals:

• Extend the Open Infrastructure for the Pulsar Network deployment.
• Further extend to AWS, Azure and Google cloud and container orchestrator (k8s).
• Include EOSC-compliant AAI to facilitate integration with other services.

Status:

• documentation: https://pulsar-network.readthedocs.io
• github: https://github.com/usegalaxy-eu/pulsar-infrastructure
• Ansible roles, terraform recipes and documentation already available.

How are we planning to achieve the objectives?

7

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.1

Open infrastructure:

• set of tools to have a ready-to-go Pulsar environment easily deployable into a cloud infrastructure;
• enable consortium partners (and beyond) to deploy new pulsar nodes to further extend the computing capacity of the network.

How are we planning to achieve the objectives?

8

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.1

• A virtual machine image, named Virtual Galaxy Compute Nodes (VGCN), that provides everything is needed to run Galaxy jobs.
• Terraform scripts that take care of the infrastructure deployment over the Cloud resources
• Ansible scripts to complete the Pulsar's configuration and have then an easy mechanism for its update.

How are we planning to achieve the objectives?

9

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.1

• A virtual machine image, named Virtual Galaxy Compute Nodes (VGCN), that provides everything is needed to run Galaxy jobs.
• Terraform scripts that take care of the infrastructure deployment over the Cloud resources
• Ansible scripts to complete the Pulsar's configuration and have then an easy mechanism for its update.

How are we planning to achieve the objectives?

10

Terraform is a software for creating and managing virtual infrastructures by exploiting machine-readable configuration files.

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.1

• A virtual machine image, named Virtual Galaxy Compute Nodes (VGCN), that provides everything is needed to run Galaxy jobs.
• Terraform scripts that take care of the infrastructure deployment over the Cloud resources
• Ansible scripts to complete the Pulsar's configuration and have then an easy mechanism for its update.

How are we planning to achieve the objectives?

11

Ansible is an open-source software that automates cloud configuration management, application deployment and service orchestration.

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.1

How are we planning to achieve the objectives?

12

NFS

Central manager

(HTCondor + Pulsar)

Compute nodes

For a prototype setup, the minimum requirements are:

• Central manager and NFS server �each with 4 cores, 8 GB
• Computational workers�each with 4-8 cores, 16 GB
• >200 GB volume

but the more the better

​

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.1

How are we planning to achieve the objectives?

13

https://pulsar-network.readthedocs.io/en/latest/

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.2

Add the GA4GH Task-Execution-Service (TES) API to Pulsar (M1-M12)

Task Lead: CESNET

Task Members: ALU-FR, CNR

Goals:

• Implement support for the GA4GH Task Execution Service, allowing other services to submits jobs via TES to Pulsar and to the European Pulsar Network.

Status:

• TES spec: https://github.com/ga4gh/task-execution-schemas

How are we planning to achieve the objectives?

14

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.2

The Task Execution Service (TES) API is an effort to define a standardized schema and API for describing batch execution tasks. A task defines a set of input files, a set of (Docker) containers and commands to run, a set of output files, and some other logging and metadata.

https://ga4gh.github.io/task-execution-schemas/docs/

​

​

How are we planning to achieve the objectives?

15

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.3

Build an European-wide network of Pulsar sites (M7-M36)

Task Lead: CESNET

Task Members: ALU-FR, VIB, EPFL, CESNET, BSC, CNRS, CNR, INFN, UiO, AGH / AGH-UST. IIAS, TUBITAK

Goals:

• Deploy and maintain pulsar endpoints

Status:

• documentation: https://pulsar-network.readthedocs.io
• github: https://github.com/usegalaxy-eu/pulsar-infrastructure
• Several Pulsar endpoints are already online.

How are we planning to achieve the objectives?

16

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.4

Add TES support to WfExS (Workflow Execution Service) (M18-M36)

Task Lead: BSC

Task Members: UNIMAN

Goals:

• Extend WfExS to support ESG as compute platform
• execute task on the Pulsar Network using TES API developed in T3.2

Status:

• Github: https://github.com/inab/WfExS-backend

How are we planning to achieve the objectives?

17

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.4

WfExS is a high-level workflow execution service backend, developed within EOSC-Life as part of Demonstrator 7 (D7), which can manage workflows across different domains.

It has a strong focus on reproducible and replicable analysis by using digital objects like RO-Crate.

• Fetches workflows from WorkflowHub.
• identifies the workflow type and run it using its native workflow execution engine (currently CWL and NextFlow).
• Identifies the containers needed by the workflow and fetches them.
• Optionally describes the results with a RO-Crate and makes them available to users.

​

How are we planning to achieve the objectives?

18

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.5

Developing and maintaining national or domain-driven Galaxy servers (M1-M36)

Task Lead: VIB

Task Members: ALU-FR, UiO, UB, CNRS, CNR

Goals:

• Develop and maintain an Open Infrastructure for deploying National Galaxy instances.
• Deploy National Galaxy instances to access local infrastructure and the Pulsar Network.
• User support

Status:

• Github: https://github.com/usegalaxy-eu
• Ansible roles and terraform recipes available. Some useGalaxy national instances (Belgium, France) already up and running.

How are we planning to achieve the objectives?

19

EuroScienceGateway | Kick-Off Meeting

Objectives - Task 3.5

How are we planning to achieve the objectives?

20

• Continuous testing
• Continuous Deployment

EuroScienceGateway | Kick-Off Meeting

21

EuroScienceGateway | Kick-Off Meeting

Deliverables and Milestones

22

EuroScienceGateway | Kick-Off Meeting

Connection to the other Work Packages

​

​

Work Package 4

• BYOC Development -> usage of the Open Infrastructure to deploy new pulsar endpoint.
• BYOS Development -> mechanism for data locality development based on a caching layer, tracking which Pulsar endpoint has a specific dataset already available.
• Smart job-scheduling system development.

​

Work Package 5 -Use cases work package.

How can we work together?

23

EuroScienceGateway | Kick-Off Meeting

Conclusions & next steps

Open questions

• We plan to move the Pulsar Network from TRL-7 to TRL-9 by expanding the APIs, hardening the deployments already available and deploying new ones.
• The Pulsar Network will become a production-ready interface to European computing resources.
• National Galaxy instances across Europe and other workflow management systems will be enabled to submit jobs to this distributed compute network.

​

​

Work Package 3 planning meeting held on 6th of September.

Planning 1 WP3 monthly meeting.

Kick start Task meeting this month for T3.1, T3.2 and T3.3.

24

This project was funded by the European Union’s HORIZON-INFRA-2021-EOSC-01, under the Grant Agreement number 101057388.

EuroScienceGateway | Kick-Off Meeting

Backup

25

EuroScienceGateway | Kick-Off Meeting

Pulsar Network - current contributors

Pulsar endpoints:

• DE, de.NBI cloud
• IT, ReCaS-Bari
• BE, Vlaams Supercomputer Centrum (VSC)
• PT, Tecnico Lisboa
• ES, Barcelona Supercomp. Center (INB-BSC )
• NO, University of Bergen
• CZ, CESNET
• FI, CSC
• UK, Diamond Light Source
• FR, GenOuest

26

EuroScienceGateway | Kick-Off Meeting