Tour of a WDL workflow

Stephanie Gogarten

Simulation and Benchmarking sub-WG

Sept 21, 2022

Outline

Where to learn WDL

Example workflow: submit job to imputation server

Docker

Dockstore

Advanced WDL: parallel jobs with scatter()

Where to learn WDL

https://primedconsortium.org/index.php/members/research/anvil-resources#developing-workflows

“Learn WDL” videos

Anatomy of a workflow

1. WDL version
2. Workflow name
3. Workflow inputs
4. Call to task
5. Task inputs
6. Workflow outputs
7. Author information

Workflow inputs

https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md

Calling a task

Workflow inputs map to task inputs

​

​

​

​

In this case there is only one task, but with multiple tasks each can have a different subset of inputs

Inside the task

1. Task inputs
2. Command block
3. Task outputs
4. Runtime (instructions for compute environment)

Task inputs

Doing stuff: the command block

Shell commands to be executed at runtime

​

​

​

​

Input references are enclosed in ${}

Transform inputs with arguments inside brackets (e.g. paste with separator)

Task output

Output type options are same as input: e.g. string, boolean, File

​

​

​

​

​

This example reads a temporary file to return a string

Workflow output

Output of workflow references task output

​

Displayed in AnVIL job manager:

Task name

Defining the compute environment

The “runtime” block specifies the docker image in which the command is executed

​

​

The docker image must contain all software needed to run the command

What is Docker?

• Custom compute environment that can be run anywhere
• A Docker image is:
• built containing all software necessary to run an application
• Usually built from a base image (e.g., ubuntu)
• Typically composed of multiple layers
• a read-only template with instructions for creating a Docker container
• A Docker container is:
• a runnable instance of an image on a local or host computer
• what the image becomes in memory when executed

​

13

Creating a docker image

1. Base image
2. Install curl and Java
3. Download imputationbot software

Save in a “Dockerfile”

Build image locally

Push to repository

• popular choices are https://hub.docker.com/ or https://quay.io/

Maintaining code on GitHub

Dockerfile

WDL file

Sync with Dockstore

Example inputs

Publishing the workflow on Dockstore

.dockstore.yml file

1. Name of workflow
2. Path to WDL
3. Path to example JSON

Follow instructions linked from PRIMED website to connect your GitHub repository to Dockstore

PRIMED WDL collections

https://dockstore.org/organizations/PRIMED

Workflow page on dockstore

Workflow must be published to see this page

Import workflow to an AnVIL workspace

Workflow on AnVIL

Edit example JSON and upload to AnVIL

Example JSON

Advanced WDL: scatter()

Example of a workflow with two levels of nested parallelization: over chromosomes and over populations

The output of a scattered task is an array

More info here

Questions?

#anvil channel on PRIMED Slack