1 of 53

12-14 November 2024, Costa Rica and Online

RDA 23rd Plenary Meeting (RDA P23) | Sustainable Science

FAIR4ML IG Activity Updates and Feedback

RDA 23rd Plenary Meeting

www.rd-alliance.org

2 of 53

Housekeeping

Collaborative notes: https://tinyurl.com/RDA-P23-FAIR4ML-notes

These slides: https://tinyurl.com/RDA-P23-FAIR4ML-slides

Meeting etiquette

RDA Code of Conduct
Add your questions in the chat
Raise your hand if you wish to speak
Add comments to the collaborative doc

RDA 23rd Plenary Meeting

www.rd-alliance.org

3 of 53

Agenda (in Costa Rica time, GMT-6)

Time	Description
10.00 - 10:03	Welcome and introduction	Daniel S. Katz
10:03 - 10:08	Intro to FAIR4ML IG, the objectives, structure, how to join	Daniel S. Katz
Part 1 - Updates
10:08 - 10:18 10:18 - 10:38	Task Force 1 Overview Feedback	Curtis Sharma All
10:38 - 10:48 10:48 - 11:08	Task Force 2 Overview Feedback	Daniel S. Katz All
Part 2 - Invited talks
11:08 - 11:28	up to 4 flash talks	speakers
11:28 - 11:30	Wrap-up

RDA 23rd Plenary Meeting

www.rd-alliance.org

4 of 53

Welcome and Introduction

10.00 - 10:03	Welcome and introduction	Daniel S. Katz
10:03 - 10:08	Intro to FAIR 4 ML IG, the objectives, structure, how to join	Daniel S. Katz

Collaborative notes: https://tinyurl.com/RDA-P23-FAIR4ML-notes

These slides: https://tinyurl.com/RDA-P23-FAIR4ML-slides

RDA 23rd Plenary Meeting

www.rd-alliance.org

5 of 53

FAIR4ML IG at a glance

Discussions and task forces around FAIR4ML → e.g., FAIR elements in the ML lifecycle, FAIR principles for ML, metadata for ML models and FAIR lifecycle

Some possible outcomes

Refined use cases relevant to FAIR4ML
Definitions of FAIR4ML
Recommendation and guidance documents for FAIR4ML
Technical specifications in support of FAIR4ML
Produce and offer training of the supporting outcomes on FAIR4ML (connection with RDA groups as well as external networks such as Pistoia Alliance and ELIXIR)

All outcomes to be published through the RDA platforms, and communicated across all relevant channels.

RDA 23rd Plenary Meeting

www.rd-alliance.org

6 of 53

Relevant projects & stakeholders

Platforms

Gardens AI - Garden is an ecosystem of machine learning models and datasets—made by researchers to simplify ML
Kipoi - API & repository of ready-to-use trained models for genomics
OpenML - Build open source tools to discover (and share) open data, draw them into machine learning environments, build models, analyse results, get advice on better models
Hugging Face - A platform where the machine learning community collaborates on models, datasets, and applications.

Communities

Pistoia Alliance - a global, not-for-profit members’ organization working to lower barriers to innovation in life science and healthcare R&D through pre-competitive collaboration
ELIXIR - An intergovernmental organisation that brings together life science resources (including databases, software tools, training materials, cloud storage and supercomputers) from across Europe
CLAIRE - Confederation of Laboratories for Artificial Intelligence Research in Europe
NFDI4DataScience - German national consortium working towards increasing the transparency, reproducibility and fairness of Data Science and Artificial Intelligence projects

Projects

FAIR4HEP - Using high-energy physics (HEP) as the science driver, developing a FAIR framework to advance understanding of AI, applying AI techniques, and exploring approaches to AI
HPC-FAIR - Providing a generic HPC data management framework to make both training data and AI models of scientific applications FAIR, focusing on the domain of program analyses/optimizations using AI/ML
RECUP - Framework and tools to natively extract, make FAIR HPC datasets, and prepare them for AI/ML analysis

Others? Please contact us

RDA 23rd Plenary Meeting

www.rd-alliance.org

7 of 53

How to join

RDA Mailing list → join the group and you will get the emails
Join monthly meetings on Zoom

Even months at 08:00 UTC
Odd months at 20:00 UTC
Reminder sent before each meeting to mailing list

Join any of the task forces

Task Force 1 - White paper on FAIR things aligned to ML lifecycle
Task Force 2 - Core metadata schema for ML models

Join online asynchronous discussion

Contribution to documents created by the task forces
Task Force 2 - Open consultation
GitHub space

RDA 23rd Plenary Meeting

www.rd-alliance.org

8 of 53

Goals for today

Objectives

Review and discuss the effort around the ML Lifecycle, for example, publishing of the report, challenges and opportunities in applying FAIR principles to stages of the lifecycle and adoption of the lifecycle in the Skills4EOSC project evaluating and recommending FAIR practices for AI research
Review and discuss the effort around metadata for ML
Updates from other relevant communities

Progress work towards FAIR4ML

ML lifecycle as a step towards mapping FAIR
Metadata for ML

RDA 23rd Plenary Meeting

www.rd-alliance.org

9 of 53

Task Force 1: Presentation and discussion on the ML Lifecycle for FAIR4ML

Collaborative notes: https://tinyurl.com/RDA-P23-FAIR4ML-notes

These slides: https://tinyurl.com/RDA-P23-FAIR4ML-slides

10:08 - 10:18 10:18 - 10:38	Task Force 1 Overview ML Life Cycle Mapping of FAIR across the life cycle Structure of White paper Feedback, how to get involved, who else we should �reach out to	Curtis Sharma All

RDA 23rd Plenary Meeting

www.rd-alliance.org

10 of 53

Overview of TF1

Ambition�How are the FAIR Principles applied to ML

FAIR for Research Software work and FAIR Workflows focused on how to translate/interpret the principles for research software & workflows
What about machine learning (ML) models?

Are they data?

E.g., a set of parameters and options for a particular framework

Are they software?

E.g., an executable object that takes input and provides output

Are they a combination of data + software + workflows?
Are they something else?

RDA 23rd Plenary Meeting

www.rd-alliance.org

11 of 53

Process so far

Goal: develop a white paper on this subject

Steps:

Literature review - in progress
Define ML Lifecycle - done
Define FAIR and how it applies to each part of the lifecycle in progress - in progress

RDA 23rd Plenary Meeting

www.rd-alliance.org

12 of 53

1. Literature review

An ongoing process

Gdoc capturing the state-of-the-art: FAIR4ML-Literature-Review
Zotero collection (Zotero | Groups > FAIR4ML)
Review and notes in progress

RDA 23rd Plenary Meeting

www.rd-alliance.org

13 of 53

2. Define ML Lifecycle

Mini-hackathon organized/run by ZB Med (21-24 Nov 2023)

(notes here)

Outputs:

Consensus on a simplified structure for the ML Lifecycle
First attempt at FAIR aspects of the Lifecycle

Preprint: Lifecycle for FAIR Machine Learning (zenodo.org)

RDA 23rd Plenary Meeting

www.rd-alliance.org

14 of 53

2. Define ML Lifecycle

Comprised of 10 broadly defined steps

Overall simplified process, to assist in the mapping across the FAIR principles

Things that we ignored (as part of the ML Lifecycle)

Publication
Documentation
Continuous monitoring of the process
Lifecycle of data and software

RDA 23rd Plenary Meeting

www.rd-alliance.org

15 of 53

3 FAIRness in the ML Lifecycle

Started effort here: ML Lifecycle and FAIRness

Planned activity

please go to the Collaborative notes: https://tinyurl.com/RDA-P22-FAIR4ML-notes

Three columns: FAIR Data, FAIR Software, FAIR AI Models
Ten rows: the ten steps of the lifecycle

Questions:

Which of the FAIR principles are relevant in each step?
What is/could be missing?

RDA 23rd Plenary Meeting

www.rd-alliance.org

While we have published our report on the ML Lifecycle, the White Paper is still an ongoing effort, and we invite all members of the IG to to the link here and provide your thoughts. This is where we are at the moment with aligning the FAIR principles to each step. There is a table summarising how the different stages in the life cycle map to the individual principles, or more specifically, which principles fit those stages. There is a second table as well, where a model is presented in a more detailed manner, based on the lifecycle created during the hackathon. What we want to do is “reapply” the identified principles.. We are determining for each step in the lifecycle, the final elements we need to identify. This itself happens across various steps, consolidating through the progressive work since 2023 in plenaries. The Zenodo publication presents a better way to first and foremost, identify and document in detail which elements are applicable in each step. This represents a start on aligning ML and FAIR. So, all contributions from the community, through the previous plenaries and the hackathon are represented here in structured notes, and we invite all to contribute, comment, etc.The next step is to consolidate all of this into a single report - a preprint possibly or first outlne - and feed this back into the community for consultation. We had hoped to have a draft topresent at this plenary, but this looks more likely to be ready in early 2025. In the meantime, please do continue to contribute

16 of 53

Collaboration with Skills4EOSC

P20 (March 2023)

ML Lifecycle →

“Top 10 FAIR Data Things for AI”

Delphi Survey

FAIR4ML community contributed to finalising items
Participate in survey and final focused sessions
Round two now closing

May 2025

→ FAIR4ML White Paper

RDA 23rd Plenary Meeting

www.rd-alliance.org

17 of 53

Task Force 2: Presentation and discussion of a ML model metadata based on schema.org

Collaborative notes: https://tinyurl.com/RDA-P23-FAIR4ML-notes

These slides: https://tinyurl.com/RDA-P23-FAIR4ML-slides

10:38 - 10:48 10:48 - 11:08	Task Force 1 Overview of produced output Vocabulary (https://w3id.org/fair4ml) Cross-walks Feedback, how to get involved, who else we should �reach out to	Daniel S. Katz All

RDA 23rd Plenary Meeting

www.rd-alliance.org

18 of 53

Metadata for ML related artifacts

Metadata related to ML

Training Datasets → Croissant ML (part of ML Commons)

Software to train and optimize the model → CodeMeta, Bioschemas, NFDI4DS

ML models
Extrinsic cross-validation/evaluation
Deployment
Others?

RDA 23rd Plenary Meeting

www.rd-alliance.org

19 of 53

Metadata for ML models

Approach

Literature review (including controlled vocabularies)
Initial reference point: ML model cards

Crosswalks from ML/dataset cards to platforms, e.g., hackathons

ML models: Schema.org, Hugging Face, Bioimage.io, ONNX, DO M E recommendations
Datasets (using Croissant): Hugging Face, Bioimage.io, D OME recommendations

Vr0.0.1 (schema:Thing > schema:CreativeWork > fair4ml:MLModel) �reuses the Croissant Dataset data model
Community feedback from RDA Plenary 22 → Vr0.1.0

RDA 23rd Plenary Meeting

www.rd-alliance.org

20 of 53

Metadata for ML working documents

Literature review
Crosswalks
Metadata schema

Contributed by partners: INESData (UPM) and ZB MED MLEntory
Joint schema → First consultation/feedback concluded → v0.1.0: https://w3id.org/fair4ml/

RDA 23rd Plenary Meeting

www.rd-alliance.org

21 of 53

FAIR4ML metadata in a nutshell

RDA 23rd Plenary Meeting

www.rd-alliance.org

22 of 53

Annotated example

Similar to HF Model Cards!

work in progress; more details in GitHub

identifier: https://huggingface.co/oeg/software_benchmark_bio

name: Software SciBERT

version: 0.0.1

keywords:

- software classification

- software mention

license: https://www.apache.org/licenses/LICENSE-2.0

developmentStatus: Active

dateCreated: 20-03-2023

dateModified: 20-10-2023

…

RDA 23rd Plenary Meeting

www.rd-alliance.org

23 of 53

Next steps

Continue work on crosswalks → Join online hackathon to create more crosswalks, running until the end of the month
Launch new consultation/feedback period
Incorporate feedback from crosswalks hackathon and community consultation
Update v0.1.0 schema based on feedback
Documentation, website, examples, adoption
Early adopters: INESData (incubator for AI models and data spaces in Spain) by the Ontology Engineering Group at UPM and MLentory (metadata aggregator) by ZB MED/NFDI4DataScience

RDA 23rd Plenary Meeting

www.rd-alliance.org

24 of 53

How to get involved:

Contribute to the community consultation/feedback

Join online hackathon
General comments and suggestions on RDA page
Comments about properties on the spreadsheet (v0.1.0)
Comments about missing properties on the spreadsheet
Public repository: https://github.com/RDA-FAIR4ML/FAIR4ML-schema

Issues are welcome!

RDA 23rd Plenary Meeting

www.rd-alliance.org

25 of 53

Flash Talks

Collaborative notes: https://tinyurl.com/RDA-P23-FAIR4ML-notes

These slides: https://tinyurl.com/RDA-P23-FAIR4ML-slides

11:08 - 11:28	TBD	TBD

RDA 23rd Plenary Meeting

www.rd-alliance.org

26 of 53

Wrap-up

Collaborative notes: https://tinyurl.com/RDA-P23-FAIR4ML-notes

These slides: https://tinyurl.com/RDA-P23-FAIR4ML-slides

RDA 23rd Plenary Meeting

www.rd-alliance.org

27 of 53

Any new task forces?

Please add ideas to the collaborative notes with some possible outcomes for the new task force
If you are willing to co-lead the task force, please add your name as well

RDA 23rd Plenary Meeting

www.rd-alliance.org

28 of 53

How to join

RDA Mailing list → join the group and you will get the emails
Join monthly meetings on Zoom

Even months at 08:00 UTC
Odd months at 20:00 UTC
Reminder sent before each meeting to mailing list

Join any of the task forces

Task Force 1 - White paper on FAIR things aligned to ML lifecycle
Task Force 2 - Core metadata schema for ML models

Join online asynchronous discussion

Contribution to documents created by the task forces
Task Force 2 - Open consultation
GitHub space

RDA 23rd Plenary Meeting

www.rd-alliance.org

29 of 53

THANK� YOU

RDA 23rd Plenary Meeting

www.rd-alliance.org

30 of 53

How to use these slides

Add your Group and Session Title to the front page.
Navigate to Slide > New Slide to add a blank template.
Apply customised layouts to blank slides to suit your content (see image for how to do this in Google Slides)
The RDA Comms Kit has extra visual elements you can use to bolster the design.
Remember to delete this slide before your presentation!

RDA 23rd Plenary Meeting

www.rd-alliance.org

31 of 53

RDA 23rd Plenary Meeting

www.rd-alliance.org

32 of 53

RDA 23rd Plenary Meeting

www.rd-alliance.org

33 of 53

RDA 23rd Plenary Meeting

www.rd-alliance.org

34 of 53

RDA 23rd Plenary Meeting

www.rd-alliance.org

35 of 53

Slides from previous sessions

35

RDA 23rd Plenary Meeting

www.rd-alliance.org

36 of 53

What is FAIR?

36

RDA 23rd Plenary Meeting

www.rd-alliance.org

37 of 53

What is FAIR

Findable

F1. (meta)data are assigned a globally unique and eternally persistent identifier.

F2. data are described with rich metadata.

F3. (meta)data are registered or indexed in a searchable resource.

F4. metadata specify the data identifier.

Accessible

A1 (meta)data are retrievable by their identifier using a standardized communications protocol.

A1.1 the protocol is open, free, and universally implementable.

A1.2 the protocol allows for an authentication and authorization procedure, where necessary.

A2 metadata are accessible, even when the data are no longer available.

Interoperable

I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.

I2. (meta)data use vocabularies that follow FAIR principles.

I3. (meta)data include qualified references to other (meta)data.

Reusable

R1. (meta)data have a plurality of accurate and

relevant attributes.

R1.1. (meta)data are released with a clear and accessible data usage license.

R1.2. (meta)data are associated with their provenance.

R1.3. (meta)data meet domain-relevant community standards.

A set of principles, to ensure that data are shared in a way that enables and enhances reuse by humans and machines

RDA 23rd Plenary Meeting

www.rd-alliance.org

38 of 53

FAIR for non-data objects: some context

FAIR Principles, at a high level, are intended to apply to all research objects; both those used in research and those that are research outputs
Text in principles often includes "(Meta)data …"

Shorthand for "metadata and data …"

Principles applied via dataset creators and repositories, collectively responsible for creating, annotating, indexing, preserving, sharing the datasets and their metadata
What about non-data objects?

While they can often be stored as data, they are not just data

While high level goals (F, A, I, R) are mostly the same, the details and how they are implemented depend on

How objects are created and used
How/where the objects are stored and shared
How/where metadata is stored and indexed

Work needed to define, then implement, then adopt principles

RDA 23rd Plenary Meeting

www.rd-alliance.org

39 of 53

FAIR4ML: External support

FAIR Principles, are intended to apply to all digital objects (Wilkinson et al. 2016)

Here we focus on the adaptation and adoption of the FAIR principles to machine learning

Recommendation n°5 :

Recognise that FAIR guidelines will require translation for other digital objects and support such efforts.

2020: ‘Six Recommendations for Implementation of FAIR Practice’

(FAIR Practice Task Force EOSC , 2020):

RDA 23rd Plenary Meeting

www.rd-alliance.org

40 of 53

FAIR for non-data objects: some efforts

RDA 23rd Plenary Meeting

www.rd-alliance.org

41 of 53

FAIR Principles for Research Software

A joint RDA Working Group, FORCE11 Working Group, and Research Software Alliance (ReSA) Taskforce.
Coordinating a range of existing community-led discussions on:

How to define and effectively apply FAIR principles to research software
How to achieve adoption of these principles

Output approved by RDA: May 24th 2022:

Chue Hong, N. P., Katz, D. S., Barker, M., Lamprecht, A-L, Martinez, C., Psomopoulos, F. E., Harrow, J., Castro, L. J., Gruenpeter, M., Martinez, P. A., Honeyman, T., et al. (2022). FAIR Principles for Research Software version 1.0. (FAIR4RS Principles v1.0). Research Data Alliance. DOI: 10.15497/RDA00068

RDA 23rd Plenary Meeting

www.rd-alliance.org

42 of 53

What does FAIR apply to?

As previously stated*, original FAIR principles

Claim to apply to "scholarly digital research objects"
But actually focus on metadata and data

FAIR for Research Software work and FAIR Workflows focused on how to translate/interpret the principles for research software & workflows
What about machine learning (ML) models?

Are they data?

E.g., a set of parameters and options for a particular framework

Are they software?

E.g., an executable object that takes input and provides output

Are they a combination of data + software + workflows?
Are they something else?

* see Wilkinson et al. 2016 and Katz & Barker 2021

RDA 23rd Plenary Meeting

www.rd-alliance.org

43 of 53

How does FAIR apply?

Large elements of FAIR for data are dependent on archival repositories (e.g. Zenodo, re3data.org)

Hold data and/or metadata, provide search and access capabilities

Software is different, since it typically isn't shared via archival repositories but instead via social coding platform (e.g., GitHub) and package management systems (e.g. PyPI, CRAN)
What about ML models?

Searched and shared via repositories?
Searched and shared via executable platforms?
Searched and shared via something else? (e.g., DLHub, OpenML, …)

Models and training data are linked - should they be shared together?

RDA 23rd Plenary Meeting

www.rd-alliance.org

44 of 53

Work so far, getting to the FAIR4ML IG

Poster at RDA VP16 (Nov 2020):
BoF at RDA VP17 (April 2021):
FAIR for Machine Learning Models (June 2021), FAIR Festival
1st Community call (July 2021)
DaMaLOS talk (24 Oct 2021)
BoF at RDA VP18 (4 & 9 Nov 2021)

Discuss status and (possibly) form new working group

Find co-chairs & general contributors

Discuss a potential white paper on FAIR4ML

BoF at SC21 (18 Nov 2021)

RDA 23rd Plenary Meeting

www.rd-alliance.org

45 of 53

Defining FAIR in ML

As already mentioned, ML is in a grey spot when discussing FAIR

FAIR Data
FAIR for Research Software work
FAIR Workflows
FAIR digital objects?

What about machine learning (ML)? Is it just models or other components as well?

RDA 23rd Plenary Meeting

www.rd-alliance.org

46 of 53

Defining FAIR in ML: Task Force 1

Key points to be addressed:

What is already in the literature about this?
What are domain-specific constraints
Which are the different components in ML that we need to be looking into?

Discussions started May 2023, 5 monthly calls since then.

RDA 23rd Plenary Meeting

www.rd-alliance.org

47 of 53

Defining FAIR in ML: Activities so far

Literature review

Working document FAIR4ML-Literature-Review
List of papers on FAIR ML/AI -> transferred to a Zotero Group (https://www.zotero.org/groups/5157406/fair4ml)

Outlining a white paper structure

Introduction
Literature review
Define ML Lifecycle
Define FAIR with respect to ML lifecycle
How FAIR applies to each part of the lifecycle

RDA 23rd Plenary Meeting

www.rd-alliance.org

48 of 53

Metadata for ML

Superpowers!

Research artifacts as open as possible as close as necessary

But… we still need some minimum information on that that is closed (same on that that is open) → Metadata

Metadata help us describe things: models, publications, data, software

Metadata enabling bridges but also *ilities

Structured metadata plays a key role in FAIR

and makes things easier for machines and humans

Big community effort, cultural change

RDA 23rd Plenary Meeting

www.rd-alliance.org

49 of 53

Metadata for Linked Open Science

Linked Open Data → RDF standards, common ground, openness, linking

FAIR → minimum metadata, enriched metadata, community agreement

Open Science → sharing as much as possible

Linked Open Science → (lightweight) LOD + FAIR + Open Science → Knowledge graph for research artifacts

Image taken from https://www.w3.org/DesignIssues/LinkedData

RDA 23rd Plenary Meeting

www.rd-alliance.org

50 of 53

Metadata for ML models

Metadata for ML

Training Datasets
Software to train and optimize the model
ML models
Evaluation

Approach

Basis: ML model and dataset cards
Literature review (including controlled vocabularies)
Mapping from cards to schema.org

RDA 23rd Plenary Meeting

www.rd-alliance.org

51 of 53

Connecting the dots

Software Management Plans

SoftwareSourceCode, SoftwareApplication, SoftwareTestingAction (new type), �SMP (new type)

Machine Learning

ML models (new type), training Datasets, SoftwareSourceCode, SoftwareApplication

Scholarly publications

Research Digital Objects

Sources

RDA 23rd Plenary Meeting

www.rd-alliance.org

52 of 53

RDA 23rd Plenary Meeting

www.rd-alliance.org

53 of 53

RDA 23rd Plenary Meeting

www.rd-alliance.org