1 of 19

Cloud Work Stream Roadmapping for 2022

Chairs: Brian O’Connor, David Glazer

​

Presenters:

Brian/Michael

Patrick/Lee/Sara

Denis

Kyle/Ania

2 of 19

Cloud Roadmapping - 2022

Our collective goals:

  • Clearly define/update our roadmap for the remainder of 2022
  • Target features for release in August ahead of September Plenary

​

September 19th Plenary

August 15th Releases of Cloud APIs

July

June

May

April

Prep work, documentation, presentations, coordination with Driver Projects and implementers, etc.

Think about how the PR/Issues will be addressed and merged in ~4 months

3 of 19

Cloud Roadmapping - 2022

API updates :

​

  • DRS - Brian/Michael
  • WES - Patrick/Lee/Sara
  • TRS - Denis
  • TES - Kyle/Ania

4 of 19

DRS�Brian/Michael

​

  • Key issues/PR before next release
  • Champions identified for each
  • Call to action/feedback on issues/PRs
  • Timeline for next release before Sep

​

5 of 19

GA4GH Data Repository Service (DRS) API

A cloud agnostic way to lookup data objects and read via signed URLs or other protocols

DRS Service

Data Browsing Portal

Index

Workflow Engines

AWS S3

Azure Bucket

Google Bucket

environment-

specific implementations

6 of 19

2022 DRS Roadmap

  • See our 2022 Roadmap document
  • Release 1.3.0 will need to be done by ~Aug 15th (~4 months)
  • DRS 1.3.0 - Overview
    • DRS+Passport, Auth/Passport info provided
    • Scaling/Batch Operations
    • Pagination or deprecation of Bundles
    • Extra metadata/Better Signed URL support
  • Other Items
    • Migrate to GitHub Actions → PR#379

7 of 19

2022 DRS Roadmap

1.3.0 updates

#

Feature

Issue/PR and Open Questions

Champion(s)

Comments Date /

Target Merge Date

1

Our overall "Epics"

  • Overall DRS scaling concerns [EPIC] #342
  • DRS + Passports Design Resolution [EPIC] #351
  • Extra metadata via DRS for Requester Pays and Egress [EPIC] #343

See below

August 2022

2

Batch Operations

  • Batch operations PR#365 (bulk) Issue #334
  • Deprecation Bundles, adapt PR#377 (no bundles)
  • Michael L → PRs for manifest best practice 2 PRs one for zip one for manifest in our spec
  • PR#367 (pagination of contents) -> abandon
  • Brian O.
  • Ian F.
  • Michael L
  • Jeremy A.
  • May 1st
  • Hold
  • May 15th
  • abandon

3

DRS + Passports

  • Add authorization information PR#381
  • Doug V.
  • ???

4

Extra metadata - Better Signed URL support

  • Issue(s): #343
    • Michael L to help, has example of this in Gen3
  • Brian O.
  • June 15th

8 of 19

DRS PR#365

  • See PR#365, leave comments there
  • See generated docs for a live view of this PR
  • See OpenAPI editor view
  • I need feedback on:
  • POST /objects: does this look right?
  • POST /objects/access: does the return look correct?

9 of 19

WES�Patrick/Lee/Sara

​

  • Key issues/PR before next release
  • Champions identified for each
  • Call to action/feedback on issues/PRs
  • Timeline for next release before Sept

​

10 of 19

2022 WES Roadmap

  • WES 1.1.0 - PR Schedule

Roadmap #

Feature

Issue/PR

Champion(s)

Comments Date /

Target Merge Date

1

Tasks endpoint

Patrick Magee

Lee Pang

2Q2022 / 2H2022

2

Workflow params endpoint

Lee Pang

2Q2022 / 2H2022

3

Run listing improvements

Patrick Magee

2Q2022 / 2H2022

11 of 19

2022 WES Roadmap

  • Triaged Issues

#

Feature

Issue/PR and Open Questions

Champion(s)

Comments Date /

Target Merge Date

1

Input and output format specification

Issue#176 (EPIC)

  • Issue#174 - workflow input validation
  • Issue #163 - specification of WES outputs structure
  • Issue #161 - standardize workflow_params format

Patrick Magee, Lee Pang

​

2

Pass TES endpoints in WES request

Issue#170

  • Should WES be agnostic of task exec implementation?

NEEDED

​

3

Standard workflow types for WES

Issue#173

  • Shouldn’t this be discoverable?

NEEDED

​

4

Use more standard paging for GET /runs

Issue#159

  • Persistent pages vs streaming?

NEEDED

​

5

Status callback

Issue#132

  • Open for design

NEEDED

​

12 of 19

TRS�Denis

​

  • Key issues/PR before next release
  • Champions identified for each
  • Call to action/feedback on issues/PRs
  • Timeline for next release before Sept

​

13 of 19

Future ideas

  • Part of 2.0.1?
    • Highlighted outstanding issues https://github.com/ga4gh/tool-registry-service-schemas/issues
      • 210
        • Switch to content negotiation for json vs plain text (potential breaking change)
        • Represent multiple language versions (ex: CWL 1.0 vs 1.2)
      • 202
        • Elaborate on URIs for versioning

14 of 19

Future ideas

  • Part of 2.1?
    • Larger TRS requests from dockstore.org collaborators
      • Fleshing out metadata about accrediting workflows
        • Unstructured/structured metadata passed from workflow author through registry to execution platforms that can help the platform decide whether to execute
        • GA4GH DUO?
          • If exploring this, might be good use-case to explore in FASP
      • Fleshing out checksums for signing workflows
        • Any changes for cryptographic signatures vs current hashcode-like checksums
      • Pagination TASC discussion - applicable to paging through versions of workflows
    • Ideas from yesterday’s workshop
      • Additional properties to support a UI - ex: timestamps

15 of 19

Unofficial extensions

  • These are potential or existing differences that can be discussed as part of leading TRS into different areas or needs
  • Services
    • Dockstore.org has been experimenting with services, leading to a different set of toolClasses (1 vs 2) (API definition is open-ended)
  • New languages
    • Workflow languages can change between implementations
      • (1 vs API definition which is hard-coded but trivially extended)

​

​

16 of 19

Unofficial extensions

  • Adding write
    • Adding/mirror new workflows
    • Workflow hub has an extension for accepting RO-crate workflows
  • Optional authenticated behaviour

​

17 of 19

Schedule - 2.0.1

Roadmap #

Feature

Issue/PR

Champion(s)

Comments Date /

Target Merge Date

1

Describe versioned URIs

Alex Kanitz

2Q2022 / 2H2022

18 of 19

TES�Kyle/Ania

​

  • Key issues/PR before next release
  • Champions identified for each
  • Call to action/feedback on issues/PRs
  • Timeline for next release before Sept

​

​

19 of 19

2022 TES Roadmap

  • 1.1 updates

#

Feature

Issue/PR and Open Questions

Champion(s)

Comments Date /

Target Merge Date

1

backend_parameters

Issue #127

PR #154

Cromwell for Azure developers

Done

2

Making input/output type flag optional

Issue #76

PR# 155

CWL/Snakemake developers

PR ready

3

Ignore executor error

Issue #158

PR #159

Galaxy developers

PR ready

4

File Globbing

Issue #77

CWL/Nextflow developers

​

5

Tag based list filtering

Issue #104

PR #170

Funnel team/Kyle

PR ready

6

GA4GH Passport support

Issue #151

PR #162

​

Need to validate additions as compared to other APIs and Passports 1.2 and 2.0 (work order tokens). May need DRS support in TES.

Not for TES 1.1

7

URL Callback

Issue #121

Elixir Driver

GSoC may lead to a PR. Not 1.1.