1 of 31

Building a Galaxy workflow community

Marius van den Beek, Matthias Bernt, Simon Bray, John Chilton, Ignacio Eguinoa, Wolfgang Maier, Aysam Guerler, Simone Leo

2 of 31

Workflows in Galaxy

Fastest and most reliable way to do an analysis
Encodes every parameter for every step
A lot of knowledge and trial & error in each workflow
Easy to get started
Can be generated from analysis history
Running a workflow is easier than running a tool

3 of 31

The status quo

Copy and paste for different applications

Repeatedly building the same workflow

Built-in workflow sharing limited to same Galaxy server

Is a workflow on different servers really the same ?

Hardcoded parameters and references
Very powerful, many advanced features

Collections, Subworkflows, Expression Tools, Reports underused

Few workflows are tested in continuous integration [CI]

https://github.com/usegalaxy-eu/workflow-testing

No “official” versions
No changelogs

4 of 31

How can we improve the situation ?

A (central) repository to collect workflows
Define and run workflow tests
Define workflow versions
Conventions and standards for metadata
Domain expert contributors
Workflows ! :)

5 of 31

Inspiration from tools-iuc

and many other community repositories!

https://github.com/galaxyproject/awesome-galaxy

A few years ago we were in a similar situation for tools. There were many tool wrappers out there in the toolshed and other source repositories, but the quality and security of these tools varied, and for admins that were security minded they had to scan the tool wrappers and see if they had any flaws, which is time consuming, inefficient, and not very easy. This did not scale well, especially if you want to install hundreds of tools, let alone thousands, as are available now on the big public Galaxy instances. Beyond security implications, various patterns and conventions, good or bad, were copy-pasted from one tool to another (we did not have an extensive tool schema and documentation that we have now), and collaborating to improve commonly used tool wrappers meant you had to send patches or bug reports by email.

In response to these issues the Intergalactic Utilities Commission, or IUC was founded. In practice that meant a repository (bitbucket ? github ?) was created where the IUC collects tool wrappers that follow the highest standards, and that those standards were documented. Anyone can submit or improve a tool wrapper, request a new wrapper, report bugs, at the IUC.

6 of 31

Inspiration from tools-iuc

and many other community repositories!

https://github.com/galaxyproject/awesome-galaxy

7 of 31

Inspiration from tools-iuc

IUC Goals:

Determine & refine best-practices
Guide authors in tool development
Provide training
Advocate for new tool-related features in Galaxy
Develop testing infrastructure

8 of 31

The IWC repository

https://github.com/galaxyproject/iwc

Adding a new workflow to the IWC

9 of 31

Workflow testing

Planemo is the Galaxy Tool and Workflow SDK

planemo test <tool.xml|workflow.ga>

Unify tool and workflow testing with galaxyproject/planemo-ci-action
Workflow tests run in github workflow

No need for extra architecture (Jenkins)
Everyone can run these tests
Not targeted to a specific Galaxy server
Access to all logs
All tools run in docker
Can target multiple Galaxy versions

Workflows have access to cvmfs for tools and data

Excerpt from github workflow

Test definition for Galaxy Workflow

So how do we do our testing ? Again this is very similar to testing tools. We use the planemo command line tool which uses Galaxy programmatically through Galaxy’s rest API. planemo runs the Galaxy workflow, waits until the workflow run is finished and compares the outputs to what is expected. On the top right you see the test definition for one workflow test. To write such a test you need to provide the inputs, called “Run accessions” in this case, which is a file with accessions, and some assertions about the expected output. In this case we provide a small snippet that should be contained in the `Single End Reads` output of the workflow.

We’ve chosen to run Galaxy workflow tests directly within GitHub workflow runners. This has a couple of advantages over other possible approaches. We don’t need to maintain a separate Jenkins instance, everyone can fork the repository and run tests on their fork.

Since planemo also starts the Galaxy instance we don’t have to rely on an external Galaxy server, which may not have all the tools installed that the workflow requires, and for which we can’t simply see the logs, as that would be a security problem. All jobs run within docker, so the tests can also be run locally and we can (reasonably) expect to produce identical results in local testing. And finally we can easily test against multiple versions of Galaxy, for instance when a new Galaxy release is being prepared, ensuring that the workflows continue to execute properly

One advantage of testing against external Galaxy servers is that they come with reference data, but we’ve set it up so that workflow tests can also use the same reference data that usegalaxy servers use via cvmfs.

And since the procedure of running workflow tests is not too different from running tool tests we have unified these within a script, and created a github action for these scripts called planemo-ci-action. So most of the logic is shared between tool tests and workflow tests. I’m showing the central part of the github CI workflow on the lower right side here, which uses planemo-ci-action.

This means we don’t have to update the workflow testing or tool testing in every repository that does workflow or tool testing, we can simply update the github action.

10 of 31

Periodic Workflow testing

LifeMonitor - Simone Leo

So how do we do our testing ? Again this is very similar to testing tools. We use the planemo command line tool which uses Galaxy programmatically through Galaxy’s rest API. planemo runs the Galaxy workflow, waits until the workflow run is finished and compares the outputs to what is expected. On the top right you see the test definition for one workflow test. To write such a test you need to provide the inputs, called “Run accessions” in this case, which is a file with accessions, and some assertions about the expected output. In this case we provide a small snippet that should be contained in the `Single End Reads` output of the workflow.

We’ve chosen to run Galaxy workflow tests directly within GitHub workflow runners. This has a couple of advantages over other possible approaches. We don’t need to maintain a separate Jenkins instance, everyone can fork the repository and run tests on their fork.

Since planemo also starts the Galaxy instance we don’t have to rely on an external Galaxy server, which may not have all the tools installed that the workflow requires, and for which we can’t simply see the logs, as that would be a security problem. All jobs run within docker, so the tests can also be run locally and we can (reasonably) expect to produce identical results in local testing. And finally we can easily test against multiple versions of Galaxy, for instance when a new Galaxy release is being prepared, ensuring that the workflows continue to execute properly

One advantage of testing against external Galaxy servers is that they come with reference data, but we’ve set it up so that workflow tests can also use the same reference data that usegalaxy servers use via cvmfs.

And since the procedure of running workflow tests is not too different from running tool tests we have unified these within a script, and created a github action for these scripts called planemo-ci-action. So most of the logic is shared between tool tests and workflow tests. I’m showing the central part of the github CI workflow on the lower right side here, which uses planemo-ci-action.

This means we don’t have to update the workflow testing or tool testing in every repository that does workflow or tool testing, we can simply update the github action.

11 of 31

Workflow testing against new releases

So how do we do our testing ? Again this is very similar to testing tools. We use the planemo command line tool which uses Galaxy programmatically through Galaxy’s rest API. planemo runs the Galaxy workflow, waits until the workflow run is finished and compares the outputs to what is expected. On the top right you see the test definition for one workflow test. To write such a test you need to provide the inputs, called “Run accessions” in this case, which is a file with accessions, and some assertions about the expected output. In this case we provide a small snippet that should be contained in the `Single End Reads` output of the workflow.

We’ve chosen to run Galaxy workflow tests directly within GitHub workflow runners. This has a couple of advantages over other possible approaches. We don’t need to maintain a separate Jenkins instance, everyone can fork the repository and run tests on their fork.

Since planemo also starts the Galaxy instance we don’t have to rely on an external Galaxy server, which may not have all the tools installed that the workflow requires, and for which we can’t simply see the logs, as that would be a security problem. All jobs run within docker, so the tests can also be run locally and we can (reasonably) expect to produce identical results in local testing. And finally we can easily test against multiple versions of Galaxy, for instance when a new Galaxy release is being prepared, ensuring that the workflows continue to execute properly

One advantage of testing against external Galaxy servers is that they come with reference data, but we’ve set it up so that workflow tests can also use the same reference data that usegalaxy servers use via cvmfs.

And since the procedure of running workflow tests is not too different from running tool tests we have unified these within a script, and created a github action for these scripts called planemo-ci-action. So most of the logic is shared between tool tests and workflow tests. I’m showing the central part of the github CI workflow on the lower right side here, which uses planemo-ci-action.

This means we don’t have to update the workflow testing or tool testing in every repository that does workflow or tool testing, we can simply update the github action.

12 of 31

Workflow testing in PRs

So how do we do our testing ? Again this is very similar to testing tools. We use the planemo command line tool which uses Galaxy programmatically through Galaxy’s rest API. planemo runs the Galaxy workflow, waits until the workflow run is finished and compares the outputs to what is expected. On the top right you see the test definition for one workflow test. To write such a test you need to provide the inputs, called “Run accessions” in this case, which is a file with accessions, and some assertions about the expected output. In this case we provide a small snippet that should be contained in the `Single End Reads` output of the workflow.

We’ve chosen to run Galaxy workflow tests directly within GitHub workflow runners. This has a couple of advantages over other possible approaches. We don’t need to maintain a separate Jenkins instance, everyone can fork the repository and run tests on their fork.

Since planemo also starts the Galaxy instance we don’t have to rely on an external Galaxy server, which may not have all the tools installed that the workflow requires, and for which we can’t simply see the logs, as that would be a security problem. All jobs run within docker, so the tests can also be run locally and we can (reasonably) expect to produce identical results in local testing. And finally we can easily test against multiple versions of Galaxy, for instance when a new Galaxy release is being prepared, ensuring that the workflows continue to execute properly

One advantage of testing against external Galaxy servers is that they come with reference data, but we’ve set it up so that workflow tests can also use the same reference data that usegalaxy servers use via cvmfs.

And since the procedure of running workflow tests is not too different from running tool tests we have unified these within a script, and created a github action for these scripts called planemo-ci-action. So most of the logic is shared between tool tests and workflow tests. I’m showing the central part of the github CI workflow on the lower right side here, which uses planemo-ci-action.

This means we don’t have to update the workflow testing or tool testing in every repository that does workflow or tool testing, we can simply update the github action.

13 of 31

Deposition of workflows

Using the GA4GH tool registry service standard (TRS)

The sars-cov-2 collection on Dockstore

IWC workflows on workflowhub.eu

14 of 31

Versioning of workflows

Workflows in the central IWC repository

15 of 31

Versioning of workflows

Workflows in the iwc-workflows organization

Release 0.3 of sars-cov-2-pe-illumina-artic-variant-calling workflow

16 of 31

Automated workflow updates

Simon Bray

17 of 31

Running an IWC workflow

18 of 31

Running an IWC workflow

19 of 31

Contributing

Adding Workflows

The goal is that contributing a new workflow (or a fix or enhancement) becomes as simple as possible.

You can find instructions in the readme of the iwc repository.

It basically boils down to starting with a workflow that you’ve run through the best practices panel in the workflow editor, which fixes a lot of common problems and alerts users of possible improvements.

You can then run planemo workflow_init on your workflow file, which creates a test template

that assumes you’re going to test all workflow outputs.

From this template you can delete the things you don’t need: In this case the paired end input because the accession is a single end accession, and fill in references to expected test data. You can then lint the workflow and tests to make sure all looks good, and then run planemo dockstore_init, which creates a file that dockstore scans for to find workflows on github.

The last thing you’ll need to do is to manually set a release in the workflow file while we haven’t implemented the release field within Galaxy.

We’ve just recently started this, so hopefully things will become more straightforward in the future.

20 of 31

What do you gain by publishing workflows on the IWC ?

Proper git releases of individual workflows

for each new version

Credit (in workflow metadata, dockstore + workflowhub, DOI)
Tests against new Galaxy releases
Automated updates
Review for common issues
A place to receive comments and updates

A static page for each workflow
Benchmarks and resource requirements
RO-crates with example invocations

Soon (tm)

Today

21 of 31

Future Plans

WIP: Graphical preview of workflow changes in PRs
Workflow invocation to github PR
Improved diff for reviews - gxformat2 + stable step IDs
Testing against public servers
A static page listing all of our workflows (using hub builds)
Benchmarks and resource requirements
Template for other organization

22 of 31

Join us! Thanks!

This is a community project, and we need you!

& everyone who has ever created a Galaxy tool!

I hope you will join us in creating many cool workflows, we hope this will become a community project.

tools-iuc in practice

Pull request

Review & Tests

Revision

Deployment