Mini-Symposium: Reproducible Analytics & Infrastructure

Chair: Jonathan Tedds (Hub)

​

Speakers: Carole Goble (UK), Bjoern Groening (DE),

Eva Alloza (ES), Alex Kanitz (CH), Juha Törnroos (FI)

Wednesday, June 6th, 2023; 16:00-17:30 IST (local time) // 17:00 - 18:30 CEST

16:00 – 17:30 MS6: Reproducible Analytics & Infrastructure

​

Chair: Jonathan Tedds

Speakers: Carole Goble, Bjoern Groening, Eva Alloza, Alex Kanitz, Juha Törnroos

​

​

19:00 - 23:00 Social event and dinner in EPIC museum

​

​

Social event at the EPIC museum

​

Tonight at 19:00 - 23:00, please arrive promptly

​

Please bring your badge

​

https://epicchq.com

​

The chq Building,

Custom House Quay,

Dublin 1

​

​

​

​

Agenda

#ELIXIR23

ELIXIR Key Platform Plans for Enabling Reproducible Analytics in 2024-28

Jonathan Tedds (Hub)

Tools Platform Vision for 2024-2028

The ELIXIR Tools Platform aims to …

• Allow researchers (end-users) to find well-described and fit-for-purpose research software, providing alternatives when available, that can be used in heterogeneous computational setups from a laptop to a cloud installation to an HPC center
• Serve as reference point for individual research software engineers on how to develop better, more sustainable, and recognizable software by providing best practices and services supporting such endeavour
• Provide scientific-communities with a reference place for tackling common challenges deriving their own best practices for software development

The ELIXIR Tools Platform: In practical terms

The platform

WP1: Consolidating a comprehensive Tools Ecosystem

Individual tool repositories

Tools Ecosystem repository

Correction/modification PR

Validation

• direct contribution mechanisms for software and workflow developers
• full coverage of research software metadata directly from the source-code repositories
• contributing these from the Ecosystem
• use and drive the evolution of standards such as biotoolsSchema and EDAM
• become a reference resource for reproducible analytics of the software lifecycle in research workflows and computational scientific services
• contribute to semi-automated curation of information related to tools, and linked scientific literature

[WP3]

[WP4]

[WP2]

WP2: Software management, stewardship, and standards

• Guidelines, standards, and knowledge sharing dedicated to making research software FAIR:

​

• Standards for research software metadata:
• EDAM
• biotoolsSchema > BioSchemas / Codemeta.

​

• Software management and stewardship:
• software management portal linking knowledge, training materials, services, and providing quality metrics including FAIRness assessment
• Machine-actionable SMPs

​

Link/collaborate with other WPs, platforms, communities, and global initiatives: EOSC, RDA, NIH, Australian BioCommons, GA4GH.

WP3: Software for reproducible and distributed analysis

WP4: Democratised access to reusable complex workflows

Compute Platform 2024+ Goal and structure

To contribute to/build an ecosystem that supports reliable processing/analysis of sensitive data

○ Also covering environments for non-sensitive data with lower security/trust requirements

=> Deliver services to support federated data management and analytics in life science

through 4 complementary WPs:

• Advanced Service Access Control
• Protected access to sensitive data for analysis
• Multi Cloud Infrastructure Deployment
• Sustainability, Accounting and Provenance for Federated Analytics

WP1 – Advanced Service Access Control

• Federated distributed authorization is the next large step
• Trusted tokens as an aggregate representation of attributes – simplify the process
• Delegation – complex workflows running at federated distributed sites
• Long runs support – balance between protection (short time) and long running times
• GA4GH passports and visas – the first steps towards these goals
• Elevation of trust
• Complex access control specification evaluation delegated to the AAI – simplification for service providers
• No need for them to deal with the GDPR implications
• Higher privacy for users
• Connection to other tools
• Decisions taken by data access governing bodies translated into Access control executable specifications

WP1 – Advanced Service Access Control

• Integration with the infrastructure
• Workflow execution engines
• Data repositories with fine-grain access control
• Compute and data facilities – access to trusted environments
• Provenance and reproducibility
• Accounting – not only for the billing
• Audit trails and provenance reports
• ELSI and cybersecurity implications
• Standards
• International/global collaboration
• Major EOSC technical stakeholders, RDA, GA4GH, …

WP2 - Protected access to sensitive data for analysis

​

European regulation on sensitive human data management is tightening. We adapt to this by

• building the technology and processes for access rights management needed
• ensure proper level of assurance and right granularity of access rights in access rights management
• support European-wide interoperability of different Secure Processing Environments (SPE) on access rights management

=> From secure access through secure processing/analysis to publication

• Discovery tools for sensitive data – how to find it
• Trusted analytical/processing environment – where to process it
• Publication – what and how in terms of data

=> Evolution of tools and their secure interaction in a federated distributed world

• Secure storage and data transfer
• Secure computing environments
• Secure results delivery

WP3 - Multi Cloud Infrastructure Deployment

​

• We have explored hybrid cloud and containerization

techniques and developing services based on widely

adopted community standards (GA4GH Cloud APIs)

• Use cases increasingly require outscaling peak performance needs and bringing compute to where the data resides
• Now the various previous efforts will be bundled, hardened and - with the help of ELIXIR-internal Driver Projects - further matured
• Coordinate the development of an ELIXIR-wide federated hybrid- and multi cloud-ready platform consisting of:
• GA4GH-compatible backend microservices deployed across the ELIXIR Nodes
• micro-frontend-based web portal to operationalize the platform to end users

​

Image by vectorjuice on Freepik

WP3 - Multi Cloud Infrastructure Deployment

​

• users will be able to discover, fetch and execute a range of

workflow types (e.g., CWL, Nextflow, Galaxy) on HPC

clusters, native/on-premises cloud clusters and commercial

clouds, accessing data on commonly used storage solutions

(e.g., s3 compatible object storage)

• Access control management, the handling of sensitive data and compute-related provenance will be addressed
• The ELIXIR::GA4GH cloud platform will coordinate the bundling of backend and frontend components into a ELIXIR Cloud SDK, which will allow
• systems administrators to easily deploy on-premises, hybrid and multicloud solutions
• service developers to quickly adopt the necessary APIs and guidelines to interoperate with the ELIXIR cloud platform.

​

WP4 - Sustainability, Accounting and Provenance for Federated Analytics

• ELIXIR communities need to access to advanced, large scale and sustainable compute capacities which require appropriate regulatory compliance, accounting and provenance
• life scientists have additional demands for computing sensitive data which only increases the importance of appropriate audit trails which will fit with the relevant information governance and security requirements
• ELIXIR will identify best practices in our communities, Nodes and projects in the reporting of resources (facilities, services) used to lay the groundwork for billing or other financial accounting of services (within or between institutions, and eventually across borders)
• The Platform will identify automated, lightweight solutions which are FAIR in practice in order to track provenance
• can help address the integration of non automated systems such as electronic notebooks
• we need to see a way to establish a service where a researcher can execute their workflows in a Compute environment near the data

=> Service requires accounting and governance

EOSC Life WorkflowHub

Frederik Coppens (BE)

Carole Goble (UK)

EOSC-Life WorkflowHub

Frederik Coppens (BE)

Carole Goble (UK)

​

​

With thanks to Johan Gustafsson (Australian BioCommons), Beatriz Serrano-Solano (EuroBioimaging), Finn Bacall (ELIXIR-UK)

​

​

Mini-Symposium: Reproducible Analytics & Infrastructure

Where can I find these reproducible workflows?

Platforms / community repositories

Git Repos

Publications

Murigneux, V., Roberts, L.W., Forde, B.M. et al. MicroPIPE: validating an end-to-end workflow for high-quality complete bacterial genome construction. BMC Genomics 22, 474 (2021). https://doi.org/10.1186/s12864-021-07767-z

​

Lott, M. J., Wright, B. R., Neaves, L. E., Frankham, G. J., Dennison, S., Eldridge, M. D. B., Potter, S., Alquezar-Planas, D. E., Hogg, C. J., Belov, K., & Johnson, R. N. (2022). Future-proofing the koala: Synergising genomic and environmental data for effective species management. Molecular Ecology, 31, 3035– 3055. https://doi.org/10.1111/mec.16446

Data Repositories

IWC - Intergalactic Workflow Commission

Overcome a distributed, fragmented and variable world…

​

Use a registry.

http://workflowhub.eu

http://workflowhub.org

The WorkflowHub

https://workflowhub.eu

https://workflowhub.org

354 workflows

16 system types

481 contributors

165 teams

9 collections

426K+ accesses

5.8K+ downloads

Started April 2020, Beta Sept 2020, Launch Sept 2022

Popular platforms have dedicated support on both sides.

Galaxy: RO-Crate support, metadata extraction, search, execution, monitoring IWC.

Diverse platforms, spectrum of support.

Work with anything

WorkflowHub – Features�Doesn’t replace community and dev repositories, works with them

System agnostic

Native repository support

Multiple artefacts types

Git integration

Links to docs, test data, reference data

Versions, any stage of development

Metadata extraction & standards

Author credit, citation and workflow attribution

Bio.tools integration

DOIs & ORCID, integration with DataCite

Standardised Machine Processable Metadata�for reproducibility, metadata matters!

Common workflow description independent of platform

​

Abstract CWL

Common metadata about the workflow, tools & parameters

Digital Object format packaging workflows, metadata, companion data objects, logs

https://www.commonwl.org

https://bioschemas.org/

https://edamontology.org/

https://www.researchobject.org/ro-crate/

https://citation-file-format.github.io/

Run

Integration with

service ecosystem

APIs

Digital Objects

Search and execution by GA4GH TRS compliant platforms

LifeMonitor monitoring automated workflow tests and best practice adherence in Git repo using existing frameworks

Not all workflows are portable – maybe facility-bound

Integration with

scholarly ecosystem

APIs

Digital Objects

DOIs for workflows and other content. Integration with DataCite, ORCID

​

RO-Crate deposition in Zenodo

Contribution to PIDGraph, and hence OpenAIRE Research Graph

​

Early FAIR Signposting adopter

FAIR-IMPACT software metadata recommendations.

Work with anyone

`

WorkflowHub Club

As a registry and for best practices and guidelines for FAIR workflows.

Registering a workflow goes a long way to being FAIR.

Steps towards collaborating with Publishers.

Spaces & Teams: Communities of Practice

• Community controlled sharing permissions
• Curated Collections

Supporting Reproducibility in the Future

• Widening support for Workflow Management Systems 
• Improving support for workflow lifecycle, Git integration
• Handling multi-workflows
• Streamlining communities and workflow organisation
• FAIR workflow policies, guidelines and training

​

​

• Semi-automated metadata generation & curation
• Added value services.
• Integration with other services, e.g. Knowledge Graphs
• Alignment with emerging standards

Community

Knowledge

Workflow publishing – policy, recommend, support by selected publishers

https://www.eventbrite.co.uk/e/workflowhub-ask-me-anything-session-tickets-629927479047

https://workflowhub.eu/

WorkflowHub Club

WorkflowHub

https://workflowhub.eu/

RO-Crate

https://www.researchobject.org/ro-crate/

Galaxy Europe

https://galaxyproject.eu/

Bioschemas

https://bioschemas.org

CWL

https://www.commonwl.org/

LifeMonitor

https://lifemonitor.eu/

​

​

Workflow Community Initiative

https://workflows.community/about

​

​

https://about.workflowhub.eu

​

​

​

�

​

​

​

Goble, Carole, Soiland-Reyes, Stian, Bacall, Finn, Owen, Stuart, Pireddu, Luca, & Leo, Simone. (2023). EOSC-Life Implementation of a mechanism for publishing and sharing workflows across instances of the environment. Zenodo. https://doi.org/10.5281/zenodo.7886545

Galaxy Analytics Ecosystem

​

​

Bjoern Gruening (DE)

Galaxy as large scale compute infrastructure

• Multi-user environment (71,000 - 2023/06)
• Caching of reference data
• Caching of jobs
• Multi-tiered scheduling (workflows, “users” (limits), jobs) → EuroScienceGateway
• 3,000 tools (Assembly, Jupyter, Alphafold, …)
• ~2,000,000 containers a month
• 2.5 TB uploads per day

Open Infrastructure

https://github.com/usegalaxy-eu

#ELIXIR23

On-boarding: APIs, standards, logins

• Users login via LDAP, AD, PAM, CAS, OpenID, ORCID, LS-LOGIN
• Findability / Interoperability

#ELIXIR23

Mirco-services to handle requests

• Users interacting with the a) web frontend or b) a RESTful API
• Galaxy is a ASGI application

#ELIXIR23

Micro-services to handle requests

• Users interacting with the a) web frontend or b) a RESTful API
• Galaxy is a ASGI application

#ELIXIR23

Common theme: It’s actually a bit more complicated ;)

• Resilient and scalable services

web

#ELIXIR23

Galaxy as large scale compute infrastructure

• Jobs can be submitted to “destinations”
• All common and not so common DRMs
• dependency resolution

#ELIXIR23

Galaxy as large scale compute infrastructure

• Running on all resources

#ELIXIR23

Galaxy as large scale compute infrastructure

• Bridging to all resources at onces with Pulsar
• Meta-Scheduler → EuroScienceGateway

​

#ELIXIR23

Things that I left out … time and so …

• Uploads and any kind of IO is hard

​

​

• Monitoring is essential - it helps with user support, it helps your DevOPS, it helps everyone even your funders ;) https://stats.galaxyproject.eu

​

#ELIXIR23

Things that I left out … time and so …

• STORAGE and DATA
• Distributed storage
• Local storage
• User owned storage

#ELIXIR23

Open Infrastructure

https://github.com/usegalaxy-eu

https://eurosciencegateway.eu

#ELIXIR23

Open Infrastructure

https://github.com/usegalaxy-eu

https://eurosciencegateway.eu

#ELIXIR23

ELIXIR Strategic Implementation Study Use Cases on Container Deployment

Eva Alloza (ES)

Compute Platform

Tools

Platform

SIS Containers 2021-23 - Overview

Strategic Implementation Study (SIS)

• Led by Tools & Compute Platforms
• Build on the progress made through two previous SIS to enable adoption and deployment of protocols and services by the broader ELIXIR community at scale and to establish plans for sustainability.

​

Areas of Focus & Structure

• WP1 - User Community Engagement and Adoption
• WP2 - Technical Support for Users
• WP3 - Technical Support for Providers
• WP4 - Sustainable Service portfolio

#ELIXIR23

SIS Containers 2021-23 - Overview

Areas of Focus & Structure

• WP1 - User Community Engagement and Adoption
• Identify potential use-cases
• Promote software best practices and available services
• Existing training materials for specific target audiences
• WP2 - Technical Support for Users
• Providing (limited) users support for adopting the services and solutions
• WP3 - Technical Support for Providers
• Providing (limited) service providers support for adopting standards, protocols and developments
• WP4 - Sustainable Service portfolio

#ELIXIR23

SIS Containers 2021-23 - Bioinformatics Container Usage

Usage

• Bioinformaticians rely on 1,000’s of diverse tools
• Bio.tools (https://bio.tools/ ) - 28,189 tools (May 2023 count)

​

• These tools are generally chained together in a workflow to process data in repeatable and organized steps
• WorkflowHub (https://workflowhub.eu/ )

​

• Enabling FAIR, Open and Reproducible Science

→ these workflows are more important than ever

​

• To run these tool containing workflows

→ consistent and stable environments are required

• Containers offer this solution for environment control
• BioContainers (https://biocontainers.pro/)
• Compute Platform Task 3: Container Orchestration

​

​

​

​

​

​

​

User Community Engagement and Adoption

• 10 use-cases identified
• Survey to identify use-cases: 10 use-cases supplied information about maturity, technical expertise, usage/provision of container technologies, usage of software development best practices, adoption of standards/principles/formats.

#ELIXIR23

Use-cases entry point

• Use-case maturity level
• All use-cases already passed the PoC stage
• Basic questions seem to be answered by training

• Technical expertise regarding containers
• All different levels of expertise
• Must expect all levels of knowledge

• Container providers or users
• Most of them use general purpose containers, with a fair number using in HPC environments

#ELIXIR23

Use-cases entry point

• Software development best practises followed
• Version control and Issue tracking used
• Not too much CI/CD or testing used

• Community driven standards or methodologies
• FAIR principles widely adopted

#ELIXIR23

Use-cases identified

1. Automatic deployment of nextflow pipeline with singularity container
2. Datashield analysis. DNA variant interpretation pipelines
3. Sequence-based prediction (IDPs and IDRs). Cross-validation software for NMR data
4. Processing of cluster oriented experimental data
5. Proteomics community: benchmarking of proteomics workflows
6. MNXtools - genome-scale metabolic models
7. Automatic present/absent gene expression calls generation
8. Automatic integration of genome-scale metabolic models into deposition databases
9. Semi-automated workflow for the functional analysis of transcriptomic and metabolomic data
10. 1+MG, allow federated analysis of controlled access data across borders so that the analyses conform to GDPR and other ELSI requirements

#ELIXIR23

Use-cases engagement

• 10 use-cases identified
• Engagement
• 1^st stage - 1:1 meetings to understand use-cases’ challenges and needs, and the services, technologies and standards willing to adopt/deploy.
• Provision of solutions, hints and support documentation.
• 2^nd stage - Q&A workshop to introduce container technologies, promote software best practices and available services, and address questions and topics of interest pre-submitted by use cases.
• Dedicated Slack channel in the ELIXIR workspace to allow follow-up tech conversation.

#ELIXIR23

Use-cases at Q&A workshop

• Use-case services and technology interest
• Main interest in containerisation, container orchestration and workflow systems

#ELIXIR23

Some questions from use-cases…

#ELIXIR23

WP2 - Technical Support for Users

• Listing of relevant services and identification of typical use-cases
• User documentation on reproducible data analysis => will be integrated in RDMkit
• User documentation on executing containerized workflows based on GA4GH cloud standards�=> will be integrated in ELIXIR Cloud & AAI Documentation

​

Overview of services

WP3 - Technical Support for Providers

Documentation and materials

• Galaxy
• https://docs.galaxyproject.org (Admin / Dev)
• https://planemo.readthedocs.io (Tool / Workflow Dev)
• 48 Admin training courses: https://training.galaxyproject.org/training-material/topics/admin/
• Admin learning path: https://training.galaxyproject.org/training-material/learning-pathways/admin-training.html
• 21 Developer courses: https://training.galaxyproject.org/training-material/topics/dev/
• ELIXIR Cloud
• Documentation Hub (some content still in draft stage; user docs; admin docs)
• Jupyter notebook demos/tutorials

Publication

• Galaxy
• The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update
• The Planemo toolkit for developing, deploying, and executing scientific data analyses in Galaxy and beyond
• ELIXIR Cloud
• Publication describing federated execution of containerized tasks in preparation (with ELIXIR Compute Task 2 “Hybrid Cloud”)

User Community Engagement and Adoption

• Promote software best practices and available services
• Nodes to release their software under open source licenses while respecting existing licensing restrictions and institutional policies.
• Top 10 metrics for life science software good practices
• Four simple recommendations to encourage practices in research software
• 4 simple recommendations for Open Source Software
• ELIXIR Software Management Plan v1.0, with a section on Reproducibility
• First implementation - the first software management wizard

​

• Curated TeSS collection of training materials on reproducible

containers and workflows, providing a wide range of topics

from design to implementation and deployment.

#ELIXIR23

Special acknowledgements

Alvaro Gonzalez (FI)

Fotis Psomopoulos (GR), Beatriz Serrano-Solano (DE)

​

WP2 - users’ tech support: Frederik Coppens (BE), Sven Twardziok (DE)

​

WP3 - providers’ tech support: Alex Kanitz (CH), Björn Grüning (DE)

​

WP4 - sustainability: Justin Clark-Casey (EMBL-EBI), Martin Čech (CZ)

​

Hub support - Jonathan Tedds, Gavin Farrell

#ELIXIR23

ELIXIR::GA4GH Cloud and AAI Driver Project

​

​

Alex Kanitz (CH)

in a

“Collaborate on standards, compete on implementations!”

• Developing open, modular and interoperable community standards & guidelines,�driven by member institutions (include many companies & NGOs) & 24 Driver Projects
• Strategic Partnership between ELIXIR and GA4GH
• Organized in Work Streams:

#ELIXIR23

The relevant GA4GH standards

The ELIXIR::GA4GH Cloud

• Federated cloud environment for data analytics based on GA4GH standards, built by ELIXIR Cloud & AAI, a Driver Project of GA4GH (since 2020)
• Public deployment (for end users) as well as “SDK” for cloud adoption (for admins & devs)
• Microservice architecture, suitable for on premise, hybrid cloud & multicloud use cases

Compute federation

#ELIXIR23

Compute federation: Gateway

#ELIXIR23

Compute federation: CWL workflow

#ELIXIR23

Compute federation: Demo

A recording of a live demo can be found here

The demo itself can be found here

To run the demo yourself, you need to have:

• One or more TESK or Funnel instances deployed
• An FTP server

Alternatively, reach out to us privately to get the info/secrets to run the demo on the ELIXIR Cloud infrastructure

#ELIXIR23

Compute federation: Future directions

• More workflow engines�e.g., Nextflow, Galaxy, Cromwell
• More storage providers�e.g., S3, AZ, GS, Aspera
• Integration with commercial clouds for hybrid cloud use cases
• Integration with GA4GH WES API to abstract over workflow types�e.g., CWL, Nextflow, Galaxy, Snakemake, WDL

#ELIXIR23

Compute federation:�2023 integration goals

#ELIXIR23

Other activities

• Operationalise GA4GH-based clouds
• ELIXIR Cloud Components: Suite of reusable clients for GA4GH APIs, based on Web Components (repo)
• Krini: Web portal to access the ELIXIR::GA4GH Cloud, built with ELIXIR Cloud Components (repo)
• Data security
• Assembling group of experts around framework for “GA4GH TREs”, operationalised through GA4GH Cloud APIs
• Layered application of encryption and isolation techniques (Crypt4GH, homomorphic encryption, differential privacy, TEEs)
• Possible GIF project? ⇒ GA4GH Implementation Forum workshop tomorrow, 9am

#ELIXIR23

Acknowledgments

Jonathan�Tedds�ELIXIR Hub

Álvaro�González�ELIXIR-FI

Michael�Crusoe�ELIXIR-NL

Justin�Clark-Casey�EMBL/EBI

Thanasis�Vergoulis

ELIXIR-GR

ELIXIR Community

Collaborators

ELIXIR Cloud & AAI co-leads

Thank you!

Affiliations

ELIXIR Compute Applications in Human Data Projects including GDI

​

​

Juha Törnroos (FI)

​

​

Federated data and

federated analysis

​

Why?

​

​

ELIXIR Europe

​

​

• The goal of ELIXIR is to coordinate bioinformatics resources from across Europe so they form a single infrastructure.

​

​

​

Federated data

​

Why?

​

​

Federated data, some initiatives

​

​

​

​

​

​

Federated analysis

​

Why?

​

​

Federated analysis, some initiatives

​

​

​

​

​

​

How about the future?

Social event at the EPIC museum

​

Tonight at 19:00 - 23:00, please arrive promptly

​

Please bring your badge

​

https://epicchq.com

​

The chq Building,

Custom House Quay,

Dublin 1

​

​

​

​

Thank you for joining

Please consider contributing to the development of the new Programme in Platforms and future cross Platform and Community based activities

• Contact: Jonathan Tedds, Gavin Farrell, Manthos Pitoulias (Hub)
• jonathan.tedds@elixir-europe.org
• gavin.farrell@elixir-europe.org
• manthos.pitoulias@elixir-europe.org