1 of 24

Exchanging workflow provenance �as FAIR Digital Objects�using RO-Crate

European Galaxy Days 2023�Freiburg, Germany

2023-10--05

This work is licensed under a �Creative Commons Attribution 4.0 International License.

Stian Soiland-Reyes

The University of Manchester�RO-Crate co-lead

soiland-reyes@manchester.ac.uk�https://orcid.org/0000-0001-9842-9718

2 of 24

Describe and package data collections, datasets, software etc. with their metadata

Platform-independent object exchange between repositories and services

Support reproducibility and analysis: link data with codes and workflows

Transfer of sensitive/large distributed datasets with persistent identifiers

Aggregate citations and persistent identifiers

Propagate provenance and existing metadata

Publish and archive mixed objects and references

Reuse existing standards, but hide their complexity

Aims of FAIR Research Objects

3 of 24

Realizing FAIR Digital Objects with RO-Crate

3

RO-Crate Metadata file

id

type

description

datePublished

…

license

author

organisation

https://github.com/o/script

files

By reference (PID, URL)

RO-Crate Content

directories

id

type

description

datePublished

creator

size

format

…

https://doi.org/10.5281/zenodo.5841615

Structured metadata about the RO-Crate and content

Reference existing repositories

Re-use Web standards (JSON-LD, schema.org)�Persistent identifiers w/FAIR Signposting

Add context: people, projects, etc.

https://www.researchobject.org/ro-crate/specification.html

4 of 24

Techie deep-dive!

Warning: JSON ahead

4

5 of 24

6 of 24

RO-Crate Metadata File

{ "@id": "cp7glop.ai",

"@type": "File",� "name": "Diagram showing trend to increase",

…

},

…

{ "@type": "CreativeWork",

"@id": "ro-crate-metadata.json",

"conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"},

"about": { "@id": "./" }

}

{ "@context": "https://w3id.org/ro/crate/1.1/context",

"@graph": [

RO-Crate metadata file descriptor

RO-Crate root dataset

..collection of Data entities

..described w/ contextual entities

{ "@id": "./",

"identifier": "https://doi.org/10.5281/zenodo.1009240",

"@type": "Dataset",�

"hasPart": [

{ "@id": "cp7glop.ai" },

{ "@id": "lots_of_little_files/" },

{ "@id": "communities-2018.csv" },

{ "@id": "https://doi.org/10.4225/59/59672c09f4a4b" },

{ "@id": "SciDataCon Presentations/AAA_Pilot_Project_Abstract.html" }

],�

"author": { "@id": "https://orcid.org/0000-0002-8367-6908" },

"publisher": { "@id": "https://ror.org/03f0f6041" },

"citation": { "@id": "https://doi.org/10.1109/TCYB.2014.2386282"},

"name": "Presentation of user survey 2018"

},

Flat list of metadata per entity

JSON-LD preamble

"hasPart": [

{ "@id": "cp7glop.ai" },

{ "@id": "lots_of_little_files/" },

{ "@id": "communities-2018.csv" },

{ "@id": "https://doi.org/10.4225/59/59672c09f4a4b" },

{ "@id": "SciDataCon-Presentations/AAA_Pilot_Abstract.html"}

],

That context is basically the magic bit we insert on the top of the JSON. The way we have documented this for RO-Crate you shouldn't normally care too much about the Linked Data aspect; instead that provide the links if you have them, but it does mean we can process the crate as JSON-LD and make knowledge graphs if we want to.

RO-Crates always self-declare, because we want that if you just find one in a wild, you can know that it is an RO-Crate, and this particular version of specifications, so you know which conventions we are following.

Then we have the metadata about the crate itself, this is the outer box you saw before. Maybe it has some identifier, author and so on. You see these identifiers can be DOIs, re-use existing registries for people and organizations.

The content is listed as parts, same hasPart as before, but they can be mixed. There can be local files as well as handles and so on in here. As I said, we are flexible in how you store the RO-Crate, the base assumption is you have a folder and you put the RO-Crate metadata file into it.

Now you can describe those files, and then you can deposit this wherever you want. It could be that all of these are persistent identifiers and there are no relative file paths.

7 of 24

Metadata

Data and Contextual entities �described within RO-Crate Metadata File

Base vocabulary & types: schema.org �

Cross-references to further contextual entities

RO-Crate principle:

Reuse existing PIDs and URLs

� ..but always describe entities which lack a � human-readable resolution

{

"@id": "https://orcid.org/0000-0002-8367-6908",

"@type": "Person",

"affiliation": { "@id": "https://ror.org/03f0f6041" },

"name": "J. Xuan"

}

{

"@id": "https://ror.org/03f0f6041",

"@type": "Organization",

"name": "University of Technology Sydney",

"url": "https://www.uts.edu.au/"

}

{

"@id": "figure.png",

"@type": ["File", "ImageObject"],

"name": "XXL-CT-scan of an XXL Tyrannosaurus rex skull",

"identifier": "https://doi.org/10.5281/zenodo.3479743",

"author": {"@id": "https://orcid.org/0000-0002-8367-6908"},

"encodingFormat": "image/png"

}

Metadata file describe each object (briefly)

Then we have additional entities to describe each of these. Each entity gets “unrolled” below, so you can include core metadata for each of the items; you don't have to follow the links to see what that individual item is described as. This is important because many of them don't have any FAIR Digital Objects presence, like I said they could be in all kinds of repositories, they might just have a web page, or it could be files that are fresh and don't have anywhere to go.

Each of the entities can be described in the same way, and then you keep adding nested links. In this case this is from the contextual entity, so it's an entity that you basically can't download, like you can't download a person, it could be digital twins for that, these are linked onwards. This may seem basic for anyone who knows Linked Data, but the thing is you shouldn't need to know Linked Data to do RO-Crate.

Now because we only have this flat list approach, so we have a profile of JSON-LD that is more guided than having all kind of possibilities. The first one here is a good example, because it's a file that has been localized, bason a repository entry with a persistent identifier, but if you go to that Zenodo page, you just get the landing page, you don't get the actual image, and so we have to kind of unroll it in this way.

8 of 24

Using common vocabularies

.. extending only when needed

9 of 24

10 of 24

https://www.npmjs.com/package/ro-crate-html

11 of 24

RO-Crate�in practice

RO-Crate is used by multiple international projects
Applied across research domains – from life sciences to cultural heritage

https://www.researchobject.org/ro-crate/in-use/

12 of 24

Collecting corpora for a �Language Data Commons

Peter Sefton et al: https://ptsefton.com/2022/11/25/ldaca-metadata-ecosystem-eresearch-2022/index.html

https://www.researchobject.org/ro-crate/in-use/LDaCA.html

https://youtu.be/p-GZbe-Kzww

13 of 24

Adding rich metadata to existing data platforms�

https://doi.org/10.5281/zenodo.7310739

https://doi.org/10.3897/rio.8.e95972

https://cs3mesh4eosc.eu/

https://arkisto-platform.github.io/tools/description/describo-online/

The CS3MESH4EOSC project combines major data services into the federated ScienceMesh

Users can collaborate across established data repositories and data science services.

FAIR Description Service (based on Describo Online) to annotate data using RO-Crate

Domain-specific profiles for additional metadata requirements

14 of 24

Building an �EOSC ecosystem of�FAIR Workflows

14

EOSC projects BY-COVID, EOSC-Life, EuroScienceGateway, BioDT exchange rich �Workflow RO-Crates within an emerging �EOSC ecosystem of workflow services
Workflow Crates transfer

identifiers, authors, license, workflow system
executable workflows in their native format (e.g. Galaxy)
interoperable CWL description of the workflow
software citations (e.g. tools used)
required data sources
test suites
workflow execution provenance

https://workflowhub.eu/

https://w3id.org/workflowhub/workflow-ro-crate/

15 of 24

Provenance traces of �computational executions

“Just enough” provenance model using schema.org Actions:

input1.txt is the object Alice used to create result1.txt with instrument matlab

Provenance chain of connected actions�→ implicit workflow: can it be automated?

Layered profiles:

Process Run Crate – some tool was executed
Workflow Run Crate – the tool was a workflow
Provenance Run Crate – we know which tools the workflow ran

https://www.researchobject.org/workflow-run-crate/

During execution time we try to track the provenance, but as you see this light trace, follow the red arrow, you can see the provenance chain emerging from the final results, going back to the individual steps of the workflow.

It might be there is no actual workflow, that it’s just someone running tools manually, and you can describe this in exactly the same way. So we have this gradual approach to describing how something came to be. This is some optional thing, that you can add when you have done computation analysis, and particularly when you have reused data from other digital objects, to connect up those identifiers. The provenance may be through some indirections like in this case.

…

It is important not just to record the different parts of the research objects as just showed, but also how they are connected. �For instance, if some physical instruments and computational tools have been used to produced a dataset, then we can represent that provenance in the RO-Crate by recording a series of actions that happened. This is a lightweight provenance model where you can start with almost nothing!�But, if we do collect enough information of each step, this can even form a complete chain or what we can call an implicit workflow. This is one step away from actual computational reproducibility! You could formalize it further using Common Workflow Language and anyone re-run the computational steps - but often we see that it is more important just to document what happened and why. And so we build a gentle road towards reproducible data management which also forms the basis for more detailed Workflow Run Crate which I’ll come back to.�Q: Why not W3C PROV?�A: We see PROV as important for detailed provenance, like generated from a workflow system, but it can have so many levels of granularity you need a profile of what kind of provenance. Common Provenance Model is being developed by EOSC-Life and BY-COVID, and indeed will use RO-Crate as a higher-level wrapping to contain PROV alongside the data it describes. But here we look at what minimal provenance can we embed directly in the Crate, to help document the data, rather than be most truthful to the process.

16 of 24

Five Safes RO-Crate Profile

User making the request

Data Access agreement

Pre-approved workflows and containers on TRE

Metadata of dataset and references to dataset (e.g. HDR-UK Gateway)

Outputs approved for release & provenance

Secure workflow execution �in federated Trusted Research Environments

https://trefx.uk/5s-crate/

17 of 24

PID Profile

Collection

FDO

PID�20.301/a

Metadata

Operation

�Attributes

20.123: “Alice” 20.789: <http://...>

20.456: 10.1234/ab

PID Record

Bytes

FDO

FDO Type

FAIR Digital Object (FDO) – conceptual view

Rigid Persistent Identifiers

Self-describing digital objects

Distributed architecture

Machine actionable

Encapsulation of operations

CRUD
Extensible operations

Data/metadata abstraction

Several types of metadata

Predictable implementation of FAIR �for active objects, not just static data

https://fairdo.org/

http://shorturl.at/klrw2

In case you were new to FDO this is a quick intro. We know we have a persistent identifier to some PID Record, some record of what is inside, and then you can get to the data and the metadata. The collection is just a type of these objects.

A FAIR Digital Object, conceptually speaking, is a data structure that describes a. A Persistent Identifier (PID) can be looked up to a PID Record, where we find a list of attributes of the FDO. These can be references, the most important one is the type, as that tells us the purpose of the FDO and what attributes to expect.

The PID Profile tells us how conceptual FDOs are realized with particular technologies, for instance using DOIs and a PID record in JSON.

For Data objects there are usually some bytes that can be requested through Operations like the familiar CRUD (Create-Read-Update-Delete), but also

Attributes can reference other FDOs, so you don’t need to squeeze it all into the PID Record but can have separate Metadata in different formats that describe this FDO further.

Now crucially all of these are FDOs, it’s turtles all the way down, so what I just said also apply to types, profiles, metadata and operations, and these are therefore can be distributed and re-used. A Collection is likewise just another FDO that points to the PIDs of other FDOs.

18 of 24

Resolving RO-Crate FDOs using FAIR Signposting

Profile:�RO-Crate

w3id

�<…schema.org/dataset>; rel=type

<https://doi…>; rel=cite-as

<…crate.zip>; rel=item�<ro-crat…>; rel=describedby;� profile=…ro-crate

PID Record

zip

Type

Dataset

FAIR Signposting

...

Type

ComputationalWorkflow

Metadata FDO

ro-crate-metadata.json

PID�https://doi.org/...

w3id

RO-Crate profile

HEAD https://workflowhub.eu/workflows/255?version=1 HTTP/1.1

…

Link: <https://workflowhub.eu/workflows/255?version=1> ;� rel="describedby"; � type="application/vnd.datacite.datacite+xml", �<https://workflowhub.eu/workflows/255?version=1> ; � rel="describedby" ; type="application/ld+json", �<https://doi.org/10.48546/workflowhub.workflow.255.1> ; � rel="cite-as", �<https://workflowhub.eu/workflows/255/ro_crate?version=1> ; � rel="item" ; type="application/zip" ;� profile="https://w3id.org/ro/crate"

HTTP Link headers for machines

HTML landing page for humans

https://doi.org/10.3897/rio.8.e93937

In RO-Crate we kind of realized FDO in very much the same way, but we use FAIR Signposting and URLs as the entry point, so it doesn't have to be in the handle system, and you will know the DOIs are technically in the handle system, but you can be more flexible and using Signposting over here to follow that same navigation.

Here you see how you can find it is an RO-Crate, not just any other crate, but a Workflow RO-Crate, and you can also find a direct downloads. If you take away nothing else from this, I hope that you just add these headers to your web server, because then we can programmatically follow the digital object. I would not have needed to that indirection for the Zenodo image, because if Zenodo did include that kind of download signposting, I could already follow it programmatically.

...

Now let’s look at how we realize FDO using RO-Crate and the existing Web.

I’m looking now at an FDO in our registry WorkflowHub, so when you resolve the DOI you get a nice HTML landing page. This is what regular users normally expect from identifiers, not some mystic JSON. We’ll keep those under the hood for now.

But to be an FDO we need machine readability, right? So here we use the Web standards.

FAIR Signposting allows a computer program to look up related links, which we in FDO terms can call the PID Record.

This gives the high level FDO aspects we saw before: We have a type, from schema.org and bioschemas

We have the PID itself, a downloadable ZIP, and a metadata file that we can parse.

This is then the PID Profile for RO-Crate, the program can get straight at the download if it so wants, using regular HTTP operations.

But we want to know more! And that is why we use JSON-LD in the RO-Crate Metadata file, where you can extend to your heart’s delight. For instance in this case we can find out it’s a Workflow RO-Crate, a profile of RO-Crate that describes workflows, a Galaxy workflow, some ORCID identifiers of people who contributed.

So here we see that we don’t need to invent new Digital Object protocols to implement FDO, the Web can do it perfectly, as long as we put in the right links for signposting.

Now some of you may be Linked Data experts already and say, hey, we always did this, but with some magic content-negotiation or embedded statements. I don’t want to open that can of worms now, but suffice to say that is very fragile for consumers. Here we have declared explicitly what metadata you can get to which profile – and you can have many of these. In fact in WorkflowHub we also link to Datacite XML as alternative metadata. So that is realization of the FDO principles to allow full predictable machine actionability. Next for us now, in projects like EuroScienceGateway, we will expand the green bits on operations, how to do other active things with the object. We use GA4GH APIs for instance to launch a workflow in UseGalaxy, these operations to “wake up” the object we will also add to the FDO with RO-Crate.

19 of 24

RO-Crate in Galaxy�

20 of 24

21 of 24

https://doi.org/10.5281/zenodo.7785861

(rocrate) stain@xena:~/Downloads/41$ �runcrate report .

action: #d1131123-46a9-4b08-94f5-f57166758c62

instrument: workflows/a899d403c7447c52.gxwf.yml (['File', 'SoftwareSourceCode', 'ComputationalWorkflow'])

started: 2021-11-18T01:35:41.811075

ended: 2021-11-18T01:35:41.811085

inputs:

datasets/Pasted_Entry_4.txt <- #3ec11a41-a7b1-46fc-82c2-69d4d71d5298

outputs:

datasets/tac_on_data_4_7.txt <- #341540ad-2cc0-42a1-98f2-e845926ff184

datasets/Select_first_on_data_7_8.txt <- #5e2c955c-a96c-49ba-a77f-91f06e1e8452

(rocrate) stain@xena:~/Downloads/41$ �rochtml ro-crate-metadata.json

22 of 24

RO-Crate training

https://gallantries.github.io/video-library/modules/ro-crate

https://www.researchobject.org/runcrate/

23 of 24

More �RO-Crate in Galaxy

Import history

Export to InvenioRDM

Import as data set

24 of 24

Thank you!

24