Syntactic Sugar for CWL with the�POLUS Workflow Inference Compiler

Jake Fennick^1,2, Nazanin Donyapour^1,2, Brandon Walker^1,2,

Sam G. Michael², Nathan Hotaling^1,2, Hythem Sidky^1,2,*

1. Axle Research and Technologies, Rockville, MD, USA

federal contractor for

1. National Center for Advancing Translational Sciences, National Institutes of Health, Bethesda, MD, USA

label: Molecular Dynamics

steps:

- min.yml:

- equil.yml:

- prod.yml:

label: Minimization

steps:

- steep.yml:

- cg.yml:

label: Equilibration

steps:

- nvt.yml:

- npt.yml:

label: Steepest Descent

steps:

...

NCATS’ Mission and Workflow Inference Compiler

“to catalyze the generation of innovative methods and technologies that will enhance the development, testing and implementation of diagnostics and therapeutics across a wide range of human diseases and conditions”

Workflow Inference Compiler allows scientists to abstract out implementation details and focus their domain expertise into building pipelines for next-generation therapeutics

Who we are

Jake Fennick

Senior Data Scientist

Molecular modeling, software development,

workflow development

Hythem Sidky, PhD

Senior Data Scientist

Molecular modeling, statistics, ML/DL, software engineering, & clinical data analysis

Nathan Hotaling, PhD

Senior Data Scientist

Statistics, machine learning, deep learning, biomaterials, and tissue engineering

Sam Michael

CIO, NCATS

​

Nazanin Donyapour, PhD

Senior Data Scientist

Brandon Walker, PhD

Senior Data Scientist

​

Problem: Complex workflows

• Writing raw CWL can be verbose
• Target end users:
• Experts, but also undergraduates, wet lab scientists, etc
• We should not require the above users to learn a complex language
• Need something super simple!

Solution: POLUS Workflow Inference Compiler

• Simple YAML syntax
• Inline arguments
• Compiles / expands to CWL
• It’s really just CWL minus most of the syntax!

steps:

- echo:

in:

message: Hello World

$ wic --yaml docs/tutorials/helloworld.yml --run_local --quiet

…

Success! Output files should be in outdir/

$ cat outdir/helloworld/step\ 1\ echo/stdout/stdout

Hello World

#!/usr/bin/env cwl-runner

# This file was autogenerated using the Workflow Inference Compiler, version 0.post0.dev190

# https://github.com/PolusAI/workflow-inference-compiler

steps:

helloworld__step__1__echo:

in:

message:

source: helloworld__step__1__echo___message

run: ../cwl_adapters/echo.cwl

out:

- stdout

cwlVersion: v1.0

class: Workflow

$namespaces:

edam: https://edamontology.org/

$schemas:

- https://raw.githubusercontent.com/edamontology/edamontology/master/EDAM_dev.owl

inputs:

helloworld__step__1__echo___message:

type: string

outputs:

helloworld__step__1__echo___stdout:

type: File

outputSource: helloworld__step__1__echo/stdout

# This file was autogenerated using the Workflow Inference Compiler, version 0.post0.dev190

# https://github.com/PolusAI/workflow-inference-compiler

helloworld__step__1__echo___message: Hello World

Explicit edge syntax

• &var *var creates an edge
• Compiles to CWL dependencies
• Need quotes due to yml anchor/alias
• Tags should be globally unique
• Mimics bash scripts
• Still a little verbose

steps:

- pdb:

in:

config:

pdb_code: 1aki

output_pdb_path: '&protein.pdb'

- fix_amides:

in:

input_pdb_path: '*protein.pdb'

output_pdb_path: '&protein_fix_amides.pdb'

- fix_side_chain:

in:

input_pdb_path: '*protein_fix_amides.pdb'

output_pdb_path: '&protein_fix_side_chain.pdb'

- extract_model:

in:

config:

models: [1]

input_structure_path: '*protein_fix_side_chain.pdb'

output_structure_path: '&protein_model_1.pdb'

GraphViz DAGs automatically generated

Edge inference

• Often do not need explicit edges
• Algorithm:
• Match current input with most recent output, based on types and/or file formats
• Much less verbose!
• I am not aware of any other workflow languages with inference

steps:

- pdb:

in:

config:

pdb_code: 1aki

- fix_amides:

- fix_side_chain:

- extract_model:

in:

config:

models: [1]

output_structure_path: '&protein_model_1.pdb'

Composable reusable building blocks

• Subworkflows fully supported!
• Explicit edges and edge inference guaranteed to work the same across subworkflow boundaries!

steps:

- pdb:

in:

config:

pdb_code: 1aki

- fix_protein.yml: # subworkflow

- extract_model:

in:

config:

models: [1]

output_structure_path: '&protein_model_1.pdb'

steps:

- fix_amides:

- fix_side_chain:

Same DAG!

Edge inference

• Can infer arbitrary DAGs, not just linear pipelines!
• Caveats:
• Inference is not unique or guaranteed; always check the DAG!
• Use explicit edges as necessary

Label: System Setup

steps:

- pdb:

in:

config:

pdb_code: 1aki

- pdb2gmx:

in:

config:

water_type: spce

forcefield: amber99sb-ildn

- gmx_editconf:

in:

align_principal_axes: 0

box_type: cubic

distance_to_molecule: 1.2

- solvate:

- grompp:

in:

config:

mdp:

integrator: steep

- genion:

in:

config:

neutral: True

concentration: 0.05 # mol/liter

- gmx_trjconv_str:

in:

output_str_path: 'system.pdb'

Tooling support

• Formal JSON schema
• `wic --generate_schemas_only`
• VSCode code completion
• Tooltip documentation
• Including substructure within CWL inputs!
• e.g. gromacs mdp options

Advanced features

• Can submit workflows to remote Compute APIs
• See CWLCon 2022 talk “Running CWL across Computational Engines”
• Automated real-time analysis
• Can process outputs while a workflow step is still executing (cwltool only, localhost only)
• Can automatically insert missing workflow steps
• i.e. file format conversions
• Other compiler DAG transformations possible
• A KNIME-style GUI is in early development

Solutions Team!

Hythem

Andre

Samee

Nazanin

Jesse

Jake

Kristen

Lisa

Brandon

Sunny

Konstantin

Rachel

Aaron

Simo

Swazoo

Xi

Josue

Vlad

Nick

Najib

Nina

Hamdah

Ben

Camilo

Kevin

Nikita

Steve