1 of 13

Enabling the reuse, extension, scaling, and reproducibility of scientific workflows

​

Hervé Ménager - Institut Pasteur

ETP F2F Meeting 8-9 February 2018, Barcelona (ELIXIR-ES)

European Life Sciences Infrastructure for Biological Information

www.elixir-europe.org

European Life Sciences Infrastructure for Biological Information

www.elixir-europe.org

2 of 13

Goal of the Implementation Study

  • Aim: To develop analysis pipeline for the assembly and annotation of marine Eukaryotic transcriptome
  • Datasets: Tara Ocean and MMETSP (The Marine Microbial Eukaryote Transcriptome Sequencing Project)
  • Motivation:
    • MMETSP project, while published structured annotation never entered database of record (e.g. ENA)
    • Oceans are stuffed with eukaryotic organisms, that are not represented in reference database

3 of 13

Existing work

Building on experiences of ELIXIR Marine Metagenomics Use Case / Community in:

  • Cloud deployment
  • Workflow development using CWL
  • Many tools used in common

The EBI metagenomics CWL workflow using the UniMan visualisation tool (https://view.commonwl.org). The orange boxes indicate sub-workflows.

4 of 13

Datasets

Marine Microbial Eukaryotic Transcriptome Sequencing Project (MMETSP)

  • 678 Illumina RNA sequence datasets
  • Wide diversity spanning more than 40 phyla
  • Original assemblies by the National Center for Genome Resources (NCGR)
  • Priority will be the development of the annotation pipeline

5 of 13

Deployment

  • Goal is to deploy the CWL workflows to multiple environments
  • Currently such workflows run in production on command-line based environments (Toil, soon AWE (Argonne National Lab))
  • One axis will be its deployment on the ELIXIR Cloud
  • Another is Galaxy, because:
    • Galaxy is accessible through a web browser: no need to be familiar with command line syntax and job scheduling infrastructures
    • Galaxy is a very popular workflow environment in bioinformatics, including in the ELIXIR nodes
    • Creation of the European Galaxy Community
    • Many existing public and private servers

6 of 13

Galaxy vs. CWL tools

From https://toolshed.g2.bx.psu.edu/repos/lionelguy/seqprep and

https://github.com/EBI-Metagenomics/ebi-metagenomics-cwl/blob/master/tools/seqprep.cwl

The models for tools and workflows are not identical in CWL and Galaxy, e.g. in Galaxy:

  • all outputs are “File” types
  • ExpressionTools, record datatypes have no equivalent
  • record datatypes do not exist in Galaxy
  • UI-enabling features are more advanced (e.g. parameters grouping)

7 of 13

Galaxy vs. CWL workflows

​

From https://mississippi.snv.jussieu.fr/u/drosofff/w/metavisitor-workflow-for-use-case-2-2-imported-from-uploaded-file and https://github.com/EBI-Metagenomics/ebi-metagenomics-cwl/blob/master/workflows/emg-assembly.cwl

​

8 of 13

ToolDog (a.k.a. bio.tools Workbench Integration Enabler)

​

  1. Hillion KH, Kuzmin I, Khodak A et al. Using bio.tools to generate and annotate workbench tool descriptions [version 1; referees: 4 approved]. F1000Research2017, 6(ELIXIR):2074

(doi: 10.12688/f1000research.12974.1)

9 of 13

Approach 1: source-to-source translation

​

Tool

Tool

Workflow

Workflow

Translation software

(based on ToolDog)

deployment

Galaxy server

10 of 13

Approach 2: run native CWL in Galaxy

​

​

Tool

Workflow

Tool

Tool

Workflow

Workflow

Galaxy server

(CWL branch)

deployment

11 of 13

Galaxy CWL

https://github.com/common-workflow-language/galaxy

  • Developed by @jmchilton (co-founder of CWL and Galaxy core team developer) since 2015
  • Builds upon on the CWL reference implementation (cwltool, also developed in python) for some of the core CWL features

​

​

12 of 13

Galaxy CWL branch: current status

  • Will run simple CWL tools and workflows from the command line
  • UI needs heavy development for CWL compatibility (tools and workflows)
  • Still an experimental branch
  • In the roadmap of the Galaxy project
  • In the wishlist of the ELIXIR Galaxy community

​

​

13 of 13

Inclusion in the project plan

Task 5: Deployment on CWL on different compute platforms

  • M5.4 Development of a CWL translator to Galaxy XML (ELIXIR-FR, M8)
  • D5.3 Eukaryotic Transcriptome workflow in Galaxy (ELIXIR-FR, M9) where?
  • D5.4 Analysis of 5 datasets within Galaxy framework (ELIXIR-FR, M9) where?
  • D5.5 Production of online training materials for workflow within Galaxy (ELIXIR-FR, M11)