www.osipi.org


OSIPI Roadmap 2020 - 2022


By the OSIPI Executive & Management Board 2020

Steven Sourbron (OSIPI chair)

Laura Bell (OSIPI co-chair)

Henk-Jan Mutsaerts (OSIPI secretary)

Charlotte Debus (OSIPI past chair)

Jan Petr (Task Force 1.1 Lead)

Sudipto Dolui (Task Force 1.1 Co-lead)

Kathleen Schmainda (Task Force 1.2 Lead)

Melissa Prah (Task Force 1.2 Co-lead)

Matthias Schabel (Task Force 2.1 Lead)

Simon Levy (Task Force 2.1 Co-lead)

Ananth Madhuranthakam (Task Force 2.2 Lead)

Li Zhao (Task Force 2.2 Co-lead)

Michael Thrippleton (Task Force 2.3 Lead)

Petra van Houdt (Task Force 2.3 Co-lead)

Jim Holmes (Task Force 3.1 Lead)

Chad Quarles (Task Force 3.1 Co-lead)

Andrey Fedorov (Task Force 3.2 Lead)

Greg Cron (Task Force 3.2 Co-lead)

David Thomas (Task Force 4.1 Lead)

Yuriko Suzuki (Task Force 4.1 Co-lead)

Ina Kompan (Task Force 4.2 Lead)

David Buckley (Task Force 4.2 Co-lead)

Paula Croal (Task Force 6.1 Lead)

Udunna Anazodo (Task Force 6.1 Co-lead)

Anahita Fathi Kazerooni (Task Force 6.2 Lead)

Hamidreza Saligheh Rad (Task Force 6.2 Co-lead)


Table of Contents

Introduction        3

OSIPI deliverables (2020 - 2022)        4

OSIPI roadmap overview (2020 - 2022)        8

Task force 1.1: ASL inventory        9

Task force 1.2: DCE/DSC inventory        12

Task force 2.1: Library structure and management        15

Task force 2.2: ASL code contributions        19

Task force 2.3: DSC/DCE code contributions        22

Task force 3.1: Digital reference objects and phantoms        26

Task force 3.2: Clinical and preclinical data        30

Task force 4.1: ASL lexicon        33

Task force 4.2: DCE/DSC lexicon        36

Task force 4.3: DICOM standards        39

Task force 6.1: ASL challenges & benchmarks        40

Task force 6.2: DSC/DCE challenges        44


Introduction

The Open Source Initiative for Perfusion Imaging (OSIPI, www.osipi.org) is an initiative of the ISMRM perfusion study group, founded in 2019 after survey of the membership, and with a mission to:

“promote the sharing of perfusion imaging software in order to eliminate the practice of duplicate development, improve the reproducibility of perfusion imaging research, and speed up the translation into tools for discovery science, drug development, and clinical practice”

OSIPI is endorsed by the parent society ISMRM (www.ismrm.org) and was officially launched with a member-initiated symposium and a workshop on the annual meeting in Montreal, 2019. The first year in the run-up to the cancelled ISMRM in Sydney (May 2020) was a planning phase where a strategic plan for OSIPI was drawn up (see here) and a governance structure was defined (here).

OSIPI is led by a management team of four appointed by the ISMRM perfusion study group and consisting of a chair, vice-chair, secretary and past chair. The current strategic plan foresees 6 different aims which are implemented by 12 independent task forces, each led by a task force lead and co-lead. The task force leads together with the management team form the Executive & Management Board (EMB) of OSIPI, the main body responsible for timely delivery against the aims. The EMB meets every 3 months to check on progress.

The implementation is organised around two-year delivery cycles, the first of which starts in May 2020 and will end in May 2022. The delivery plan for each task force is referred to as the OSIPI roadmap, and is detailed in this document.

The roadmap defines 1-3 main deliverables for each task force, which represent the main results of this delivery cycle. In order to ensure timely delivery, 3-monthly milestones are defined which serve as checkpoints to determine whether or not a deliverable is on track. The roadmap is a living document, revised at each EMB meeting, and updated with the same frequency as needed to adapt to changing circumstances.


OSIPI deliverables (2020 - 2022)

For this first delivery cycle (2020 - 2022), OSIPI has 24 main deliverables including 13 peer-reviewed journal articles, 6 sustainable online resources, 1 manual, 3 software releases, and 1 standard amendment (DICOM). An estimated 16 abstracts will be submitted to ISMRM 2021 and 2022, and each task force will present at the perfusion study group workshop planned around February 2022, as well as on the study group sessions at the annual meetings.

The deliverables are listed in Table 1 and explained in more detail in the roadmaps of each individual task force below. They will build the foundations for a long-term sustainable open-source infrastructure to support perfusion research and address the strategic aims of OSIPI as laid out in the current Strategic Plan:

Aim 1: Software Inventories

OSIPI will deliver two inventories of software pipelines, one for ASL and one for DCE and DSC (jointly referred to as DC - dynamic contrast). The inventories are aimed at end-users looking for a suitable tool to analyze their data. They will be publicly available in February 2021 and will list all currently available pipelines for perfusion image analysis that are within scope. They will initially be listed with links and a basic characterisation in terms of key features and properties. In a second step a more in-depth scoring system will be developed and applied to the existing solutions. The data will be added to the inventory and the results will be submitted for publication (May 2022).

Aim 2: Code library

OSIPI will deliver an integrated library of code snippets and scripts for ASL, DCE and DSC analysis, aimed at researchers looking for source code to support their research, and for those looking to share their code with others. Existing code snippets will be sourced from the community through a public call for contributions. The code will be collected, verified, harmonised and documented by experts in the field in collaboration with the contributors. The code will then be integrated in an OSIPI library, developed and maintained. A first collection of code will be completed in Feb-May 2021, and a first version of the library will be formally released in May 2022. The structure will be scalable to allow future extension. The results of the code comparisons will be written up as manuscripts focusing on the effect of implementation differences on outcomes.

Aim 3: Reference Objects and Data

OSIPI will deliver an inventory of existing digital and physical perfusion phantoms, and of publicly available clinical and preclinical data. In the first instance the inventory will list only currently available phantoms and data, along with some metadata to help a user select the most appropriate tool for their needs. These first versions of the inventories will be delivered in Feb 2021. In a second stage, good practice guidelines will be developed for the sharing of digital/physical phantoms and data. These guidelines will be applied to existing but not yet available phantoms and data, and aim to increase the number of datasets publicly available by May 2022.

Aim 4: Reporting Guidelines

OSIPI will deliver reporting guidelines for ASL, DCE and DSC in the form of two separate lexicons tabulating and defining all the different variables in the acquisition and analysis of perfusion MRI. The aim is to improve reproducibility of research by a more standardised and well-defined way of reporting methods used to measure perfusion values. The lexicons will form the basis for a DICOM standard amendment that will be submitted by May 2022. A first version of the lexicons will be publicly available in May 2021 and will be open for public consultation. As an example application, the DC lexicon will be applied to describe the analysis performed in the software tools listed in the inventories.

Aim 6: Benchmarking

OSIPI will deliver 3 challenges, one on each of ASL, DCE and DSC. The ASL challenge will be based on a DRO with hidden ground truth and will require submission of results as well as instructions to replicate the analysis independently. The DCE/DSC challenges will be based in clinical data and will revolve around repeatability. Both challenges will formally launch before Feb 2021 and winners will be announced at the perfusion workshop in Feb 2022.


Deliverable

Strategic objective

Due date

Type

ASL pipeline inventory

1. Software inventories

1 Feb 2021

Resource

ASL pipeline comparison

1. Software inventories

1 May 2022

Manuscript

DC software inventory

1. Software inventories

1 Feb 2021

Resource

DC software comparison

1. Software inventories

1 May 2022

Manuscript

ASL code collection

2. Toolboxes

1 May 2021

Software

ASL library paper

2. Toolboxes

1 May 2022

Manuscript

DC sample code

2. Toolboxes

1 May 2021

Software

DC library paper

2. Toolboxes

1 May 2022

Manuscript

DRO & code inventory

3. Reference objects & data

1 Feb 2021

Resource

DRO sharing guidelines

3. Reference objects & data

1 May 2022

Manuscript

Perfusion data inventory

3. Reference objects & data

1 Feb 2021

Resource

Data sharing recommendations

3. Reference objects & data

1 May 2022

Manuscript

ASL Lexicon

4. Reporting

1 May 2021

Resource

ASL Lexicon paper

4. Reporting

1 May 2022

Manuscript

DC Lexicon paper

4. Reporting

1 Aug 2021

Manuscript

DC Lexicon v2.0

4. Reporting

1 May 2022

Resource

DICOM change proposal

4. Reporting

1 May 2022

Standard

ASL challenge launch

6. Benchmarking

1 Feb 2021

Manuscript

ASL challenge manuscript

6. Benchmarking

1 May 2022

Manuscript

ASL/PET software validation

6. Benchmarking

1 May 2022

Manuscript

DC challenge launch

6. Benchmarking

1 Feb 2021

Manuscript

DC challenge manuscript

6. Benchmarking

1 May 2022

Manuscript

Table 1: List of OSIPI deliverables for the first delivery cycle 2020 - 2022, showing the name of the deliverable (first column, link to more detailed description), the task force responsible (second column, link to task force roadmap), the delivery date (third column) and the type of deliverable (fourth column).


OSIPI roadmap overview (2020 - 2022)

The OSIPI roadmap for the delivery cycle 2020 - 2022 maps out the trajectory that each task force will follow to ensure the deliverables are completed in time. The progression of the tasks is split up into 3-monthly milestones with a delivery date that coincides with the meetings of the EMB. The timing of the deliverables is synchronised with the ISMRM calendar (abstract submission, annual meeting, workshops) to maximise opportunities for dissemination.

The OSIPI roadmap is a living document that will be updated at each EMB meeting depending on changing realities as needed. More detailed descriptions of the milestones and deliverables for each task force are given in the next sections.


Task force 1.1: ASL inventory

Leads: Jan Petr, Sudipto Dolui

Members: Udunna Anazodo, David Thomas, Henk-Jan Mutsaerts, Paula Croal

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) Call for ASL pipeline submission

ISMRM Submission deadline

1-Nov-2020

(M2) First ASL pipeline summary

1-Feb-2021

(D1) ASL pipeline Inventory

ISMRM 2021 (Vancouver)

1-May-2021

(M3) ASL pipeline scoring system

1-Aug-2021

(M4) Launch ASL pipeline comparison

ISMRM Submission deadline

1-Nov-2021

(M5) First ASL pipeline comparison

Perfusion workshop

1-Feb-2022

(M6) ASL pipeline comparison finalised

ISMRM 2022 (London)

1-May-2022

(D2) ASL pipeline comparison manuscript

Summary

A wide range of tools for ASL image processing exist but for end-users looking for a tool to process their ASL images, it can be a difficult process to identify one that is suitable for their needs. TF1.1 will produce an inventory of ASL pipelines aimed specifically at such end-users. The inventory will list available tools, providing information that will help users to select a suitable tool, such as scope of application, methodology, level of validation, licence policy, transparency, user-friendliness, and reviews by other users. Only freely-accessible full pipelines will be listed - smaller code snippets are available through TF2.2. The work can build on a recent review of possible ASL processing steps (ExploreASL: an image processing pipeline for multi-center ASL perfusion MRI studies).

The aim will be progressed in two steps. In a first phase, a list of available tools will be compiled after a public call for contributions and and inventory will be created of all submitted tools that are within scope (deliverable 1, Feb 2021). In a second step, an ASL pipeline scoring system will be developed consisting both of tests performed by expert users, as of comments and scores provided by the wider public. The scoring system will be applied to the pipelines in the inventory and results will be be made available online and in a publication to further assist end-users in the choice of a suitable tool (Deliverable 2, May 2022).

Milestone #1: Call for ASL pipeline submissions

Define the basic requirements for ASL pipelines to be considered for ASL inventory. Work together with TF2.2 and TF6.1 and send out a call for submissions. Let authors fill a basic questionnaire to self-assess their pipelines.

Milestone #2: First ASL pipeline summary

Close of pipeline submission site (1 Oct), summarizing the basic properties of the submitted pipelines and preparing an abstract about the summary to submitted to the ISMRM if appropriate.

Deliverable #1: ASL pipeline inventory

An overview of available software in co-operation with TF6.1, including key properties and features.

Milestone #3: ASL pipeline scoring system

Prepare a comprehensive grading system that will allow the pipeline users to grade pipelines based on their features, performance, but also ease of use and availability of support. Open the document to pipeline developers and users for comments, and to allow the developers to prepare for the comparison.

Milestone #4: Launch ASL pipeline comparison

Launch both public and expert comparison of pipelines. Both comparisons will use the same grading system. The public comparison will be driven by the users and will start soon after the scoring system is defined (1st July). The expert comparison will be coordinated and all pipelines will be assessed on the same pre-selected set of data.

Milestone #5: First ASL pipeline comparison

Publish the preliminary results from the expert comparison and the public comparison so far. All expert and public reviews submitted until 15 Oct will be considered for abstract and paper publication. The public comparison will stay open after the deadline but will not be considered.

Milestone #6: ASL pipeline comparison online

Summarize the results from both the public and expert testing in #5. Public testing will remain open for new reviews with new reviews being available for viewing continuously.

Deliverable #2: ASL pipeline comparison manuscript

Write a journal publication on the pipeline comparison.


Task force 1.2: DCE/DSC inventory

Leads: Kathleen M. Schmainda & Melissa A. Prah

Members: Andrey Fedorov, Charlotte Debus, Ives Levesque, Sudarshan Ragunathan, Laura Bell, Ina Kompan, and Rianne van der Heijden.

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) List of available DC software tools

ISMRM Submission deadline

1-Nov-2020

(M2) DC software descriptors

1-Feb-2021

(D1) DC software Inventory v1.0

ISMRM 2021 (Vancouver)

1-May-2021

(M3) DC software review process (draft)

1-Aug-2021

(M4) DC software review process v1.0

ISMRM Submission deadline

1-Nov-2021

(M5) First DC software reviews available

Perfusion workshop

1-Feb-2022

(M6) Extended number of DC software reviews available

ISMRM 2022 (London)

1-May-2022

(D2) Software inventory manuscript

Summary

The DC (dynamic contrast perfusion) inventory is aimed at end-users looking for a suitable tool to process their dynamic contrast-enhanced (DCE) and dynamic susceptibility contrast (DSC) images. The inventory will list available tools, providing factual information that will help users to select a suitable tool based on perfusion parameters generated, algorithms used, level of validation, licensing policy or status, transparency, publications using the software, and contact information when available. A first version of the inventory with this information included will be finalised in Feb 2021 (Deliverable D1).

        In a second phase TF1.2 will aim to provide expert reviews of software tools in the published inventory, and more comprehensive factual information such as more detailed lists of publications where each software tool has been used. The expert reviews can include a brief summary of software and its usage in publications. First or last authors of the publications will be contacted for additional input regarding ease of use and overall assessment of the tool. The expert review can also include an application of the tool to a set of standard data and report of the reviewer’s experiences.

        The DC software review process will be drafted and tested, and will be open for public consultation, before finalising it in Aug 2021. First reviews will become available throughout Nov 2021 - Feb 2022, and a manuscript describing methods and first results of comparisons will be submitted for publication in May 2022 (Deliverable 2).

 

Milestone #1: List of existing DC software

A list of available DC software known to the task force members, including both commercial and open-source tools, with links. The list will be made public and a call will be sent out to check for missing contributions.

Milestone #2: DC software descriptors

An excel spreadsheet with column headings that describe the descriptive/factual information that will be obtained for each software tool (e.g. functionality, number of publications, level of support, applications, etc).

Deliverable #1: DC inventory v1.0

A first complete overview of available software, with all descriptors defined in Milestone 2 completed. The document will be made available publicly for feedback from other OSIPI members and the wider community.

 

Milestone #3: DC software review process (draft)

Detailed definition of the expert review process and criteria to ensure a consistent and transparent review process, including number of expert reviewers per tool, review criteria, means of reporting, anonymity, author response processes, etc. The review process ideally also contains some quantitative metrics that can be used to compare software tools objectively. Reviews can be done by TF members, or by ad-hoc external reviewers.

Milestone #4: DC software review process v1.0

One or two DCE/DSC software tools will be selected for a first test review. Results of this review phase will not be made public. The draft review process will be updated and finalised as needed based on the experience. Additionally any feedback received on the first draft of the review process will be incorporated.

Milestone #5: First formal DC software reviews available

A small number of software tools will be prioritised for formal review, reviewers will be identified and contacted, and reviews + author responses will be concluded. Results will be made available publicly.

Milestone #6: Extended number of DC software reviews available

An extended number of software tools will be targeted for review, aiming for balance across open source / commercial and/or coverage of functionality and scope. Results will be made available publicly.

Deliverable #2:  DC software evaluation manuscript

A manuscript describing the approach and first results of DC software evaluation.


Task force 2.1: Library structure and management

Leads: Simon Lévy, Matthias Schabel

Members: Michael Berks, Martin Craig, Rebecca Echeverria, Joao Sousa

Summary 

The main mission of TF2.1 is:

TF2.2 and TF2.3 will create and maintain their own repositories but organise the code in these repositories based on specifications defined by TF2.1. When the code has passed independent compliance testing by TF2.1 it will then be integrated in a common library and formally released by TF2.1.

TF 2.1 will be dormant until May 2021 when TF2.2 and TF2.3 have built up the foundations of their library. TF2.1 will review the available code and propose a harmonized approach to organisation and management of code. Detailed milestones and deliverables will be defined from may 2021 onwards.


Task force 2.2: ASL code contributions

Leads: Ananth Madhuranthakam (Lead), Li Zhao (Co-Lead)

Members: Patrick Hales, Maria Guadalupe Mora Alvarez, Limin Zhou, Yiming Wang, Sudipto Dolui, Rebeca Echeverria Chasco.

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) ASL code collection - scope defined

ISMRM Submission deadline

1-Nov-2020

(M2) Targeted call for ASL code

1-Feb-2021

(M3) ASL code v1.0 - closed for contributions

ISMRM 2021 (Vancouver)

1-May-2021

(D1) ASL code v1.0 - verified

1-Aug-2021

(M4) ASL code v1.0 - harmonized

ISMRM Submission deadline

1-Nov-2021

(M5) ASL code v1.0 - tested

Perfusion workshop

1-Feb-2022

(M6) ASL code v1.0 - completed

ISMRM 2022 (London)

1-May-2022

(D2) ASL library manuscript

Summary

TF2.2 will develop a code collection of open-source functions, scripts and in-house pipelines for ASL perfusion imaging analysis. This is aimed at developers of ASL perfusion methods looking for specific functionality or development templates, or who want to share their own in-house code with others. Contributions will be sourced from the community, and may include individual functions and more complete pipelines in various programming languages. TF2.2 will organise these in a coherent and well-documented structure.

For this delivery cycle, the code collection will be focused on brain applications. Beginning with images of vendors’ products would be the priority, including PASL and pCASL for brain imaging, and eventually incorporate reconstruction from k-space. Any functionality beyond that is reserved for future updates.

The code will be collected and organised in two main phases. In the first phase (May 2020 - May 2021), a complete collection of code will be sourced from the community through a public call for contributions. The code will be classified and organised according to functionality and will undergo a minimal verification, but will not otherwise be modified, harmonised, documented or tested. The verified collection of ASL code will be made available in May 2021 (Deliverable D1).

In a second phase (May 2021 - May 2022) the code will be harmonised, tested more rigourously and documented according to the guidance set out by TF2.1. After independent formal testing any code that has passed the tests will be integrated in the OSIPI library v1.0. This process will conclude in Feb 2022 and the results will be announced during the perfusion workshop (if planning aligns). A manuscript will be submitted in May 2022 (Deliverable D2) and the code will be made available as part of the v1.0 library release by TF2.1 at the same time.

Milestone #1: ASL code collection - scope defined

Define the scope of the ASL code collection, including functionality to be targeted for release v1.0, and any specific requirements on code contributions. A public call for contributions will be launched and incoming code will be verified in an ongoing process following the approach laid out in the scope document.

Milestone #2: Targeted call for ASL code

Code collected up to this point will be deposited in a google drive folder and organised. The code origin, functionality, sample data from the owner and details will be documented. Gaps will be identified to prepare a more targeted approach to other developers around missing functionality to fill the gaps.

Milestone #3: ASL code v1.0 - closed for contributions

All code submitted at this point will be organised and documented - any code submitted after this date will be collected but may not be incorporated into version 1.0 of the repository. At this point the contributed code should cover a fairly comprehensive range of relevant ASL functionality.

Deliverable #1: ASL code v1.0 - verified

A collection of ASL code in the “verified code” repository within the scope set out for the v1.0 library in Milestone 1. At this point the code will be organised according to functionality, but not otherwise modified, harmonised, or documented. A verification will have been performed to test that the code runs and produces expected output, and a comparison between the outputs of code snippets that perform similar functions:

Milestone #4: ASL code v1.0 - harmonized

The code contributions in the library will be harmonised according to the approach set out in the TF2.3 scope document. Relevant structures will be built for individual data processing that can be incorporated into TF2.1 pipelines as appropriate. Harmonised versions will be tested against the original version and moved to the “harmonized code” section of the ASL repository after passing the test. The performance of code contributions with the same functionality (e.g. different implementations of the same model) will be compared.  

Milestone #5: ASL code v1.0 - tested

Formal unit tests for the code contributions in the library will be defined and executed. Results will be collected and feedback will be provided to developers. The performance of code contributions with the same functionality will be compared. Formal guidance for this stage of the process will be provided by TF2.1 (coding style, versioning etc). Results of comparisons may be submitted to ISMRM.

Milestone #6: ASL code v1.0 - completed

Code contributions that have successfully passed testing will be fully documented according the guidance laid out by TF2.1. Subsequently they will be moved to the “completed code” repository of the ASL code collection, ready for independent testing and verification by TF2.1 prior to integration in the OSIPI library. Results will be presented at the perfusion workshop.

Deliverable #2: ASL library manuscript

Manuscript detailing the ASL library, including the approach to development and organisation.


Task force 2.3: DSC/DCE code contributions

Leads: Petra van Houdt, Michael Thrippleton

Members: Jonathan Arvidsson, Samual Barnes, Martin Craig, Ingomar Gutmann, Zaki Ahmed, Michael Berks, Ben Ellingson, Jayashree Kalpathy-Cramer, Federico Pineda, Matthias Schabel, Sudarshan Ragunathan, Luis Torres, Laura Bell, Sirisha Tadimalla

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) DC collection - scope defined

ISMRM Submission deadline

1-Nov-2020

(M2) DC sample code - verified

1-Feb-2021

(M3) Public call for DC code

ISMRM 2021 (Vancouver)

1-May-2021

(D1) DC sample code - completed

1-Aug-2021

(M4) DC code v1.0 - integration underway

ISMRM Submission deadline

1-Nov-2021

(M5) DC code v1.0 - integration complete

Perfusion workshop

1-Feb-2022

(M6) DC code v1.0 - completed

ISMRM 2022 (London)

1-May-2022

(D2) DC library manuscript

Summary 

TF2.3 will build a code collection consisting of open-source functions, scripts and pipelines for DCE/DSC perfusion imaging analysis (Dynamic Contrast or “DC” perfusion analysis for short). This is aimed at developers of DC perfusion methods looking for specific functionality or development templates, or who want to share their own in-house developments with others. Contributions will be sourced from the community, and may include individual functions and more complete pipelines in various programming languages. TF2.3 will organise these in a coherent and well-documented code collection as defined by TF2.1. After formal testing by TF2.1 this code collection will be integrated in the OSIPI library developed by TF2.1.

Development of the code collection will take place in two phases. In the first phase (May 2020 - Feb 2021) a collection of code will be developed (including unit testing, documentation, etc) with a limited scope, using sample code contributed by TF2.3 members (Deliverable D1). This phase will serve to generate experience with the process, inform the development by TF2.1 of more prescriptive guidance on the library’s organisation, and also to generate interest in the wider community.

In a second phase (Feb 2021 - May 2022) a more extensive collection of code will be collected from the wider community. This code will be organised, tested, harmonised and documented based on guidance provided by TF2.1. This process will be completed in Feb 2022 and a version 1.0 of the DC code collection will be passed on to TF2.1 to be formally tested and integrated in the OSIPI library v1.0, released on May 2022. A manuscript describing the process and results will be submitted in May 2022 (Deliverable D2).

Milestone #1: DC code collection - scope defined

Determine the functionality to be targeted for the sample code and OSIPI library release v1.0, the target software languages, strategy for verification of submitted code, strategy for harmonization and nomenclature (liaising with TF2.1, TF2.2, TF4.2), approach to authorship of publications.

Milestone #2: DC sample code - verified

Sample code will be collected among TF2.3 members, tests will be defined and results will be compared. At this point the code itself is not altered and is organised in a Github repository. At present the library contains code for T1 mapping, including automated unit testing and test coverage monitoring.

Milestone #3: Targeted call for DC code

Code collected up to this point will be organised in categories. The code origin, functionality and any sample data from the owner and details will be documented in a database linked to the github repository.. The contributions will be summarised, gaps will be identified and a targeted call will be launched. We will set up instructions for new contributions such that the code can be uploaded directly to the github repository. In the meantime the TF will continue to process incoming contributions, including organising according to functionality and verification of validity.

Deliverable #1: DC code collection v1.0 - tested

At this point the “tested” section of the DC repository should contain a complete collection of code with functionality targeted for v1.0. Unit tests will have been developed for all functionality defined in the scope document for v1.0. The test results of the contributions will be available and summarised.

Milestone #4: Code integration into OSIPI library - underway

Code contributions that passed testing under milestone #2 will be selected, harmonised, integrated into the OSIPI library structure (TF2.1), verified against the original versions, and documented based on the library specifications of TF2.1. At this point a validated and documented library of code will be available, albeit with limited functionality.

Milestone #5: DC code integration with OSIPI library - completed

After a first round of code integration to the OSIPI library in the previous milestone, all code for the targeted functionality for v1.0 will be integrated into the OSIPI library. Formal guidance for this stage of the process will be provided by TF2.1 (coding style, versioning etc). The harmonised code and their tests will be documented following guidance provided by TF2.1.The performance of code contributions with the same functionality (e.g. different implementations of the same model) will be compared. Results of comparisons may be submitted as an abstract to ISMRM.

Milestone #6: DC code v1.0 - completed

This milestone will focus on the integration tests to build pipelines from the individual code modules. In this way we can test the modularity and flexibility of the library. The basic pipeline structure that was described in the scientific pipeline specifications document will be used to construct such a pipeline. In addition, a few use cases will be selected from the ‘use cases’ section of the lexicon by TF4.2.The results may also be presented at the perfusion workshop.

Deliverable #2: DC library manuscript

Manuscript describing DC code collection including details on approach to harmonisation and results of testing and comparisons. Can also include examples of application to clinical or synthetic data.


Task force 3.1: Digital reference objects and phantoms

Leads: Jim Holmes and Chad Quarles

Members: Jochen Hirsch and Leah Henze-Bancroft

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) Scope definition & call for contributions

ISMRM Submission deadline

1-Nov-2020

(M2) Define inventory structure

1-Feb-2021

(D1) Digital & physical phantom inventory v1.0

ISMRM 2021 (Vancouver)

1-May-2021

(M3) Draft recommendations of DRO sharing

1-Aug-2021

(M4) First DROs available

ISMRM Submission deadline

1-Nov-2021

(M5) All DROs available

Perfusion workshop

1-Feb-2022

(M6) Digital & physical phantom Inventory v2.0

ISMRM 2022 (London)

1-May-2022

(D2) Recommendations for DRO sharing

Summary 

TF3.1 will develop an inventory of DRO’s/synthetic data and corresponding code (source or compiled), and of physical phantoms/phantom data aimed at researchers looking for reference objects to test their perfusion methods, and at data/DRO/phantom owners who want to share those resources for secondary research. The inventory will include a curated list of publicly available digital and physical phantoms and/or associated code and data that can serve as ground truth for perfusion (analysis) methods.

The inventory is not intended to replace or compete with existing code repositories such as Github or data warehouses such as the NIH Cancer Imaging Archive. As such, the inventory and collaboration list will consist of a list of projects and publications from the scientific community as well as descriptive details about these works.

TF3.1 will initially focus on identifying, listing and characterising existing phantoms (digital or physical) in a curated and scaleable inventory (Deliverable D1, Feb 2021). In a second phase the TF will seek to extend the number of DRO’s that are freely available by developing guidelines for sharing DRO’s (Deliverable D2, May 2022). During the course of this delivery cycle, a number of existing DRO’s that are not yet freely available will be identified. An early draft of these sharing guidelines will be stress-tested by making those DRO’s available, leading to an upgraded inventory v2.0 (Feb 2022) which will include a set of new DRO’s.

Milestone #1: Call for contributions

Define the scope and content of the Inventory of DROs & code and send out a call for contributions. For DROs, the team will determine if the inventory will house actual data or be a listing of links to other data hosts. This scope and format should be synchronized with formats used by other task forces to enable continuity.

 

Milestone #2: Summarise responses to call

Compile the initial responses to the call and use these to create a draft inventory. The draft inventory will include developing a mock-up of the inventory and could include a webpage with descriptions and reference links or a Github page as potential examples. The draft inventory will be externally accessible and linked from the TF home webpage.

Deliverable #1: Digital & physical phantom inventory v1.0

Define inventory structure, form, and function - possibly synchronized with information inventories from other task forces. Build a DRO & Code inventory v1.0 including a first draft of the collaboration inventory with the contributions from Milestone #2 as well as a literature inventory with several examples of entries describing DROs, code, and physical phantoms. The goal is to test the inventory structure and confirm there are no unexpected issues with supporting various forms of content. The team will develop a process for integrating updates from contributors.  

 

Milestone #3: Draft of sharing guidelines

The team will prepare a draft of guidelines describing how researchers can contribute their work to the literature and/or sharing inventories. This will include developing a plan for long-term sustainability of the inventories including adding new content in the future. The draft will be open for public feedback and this will be encouraged and promoted via several channels including email to Perfusion study group, presentation on ISMRM, dissemination to other bodies and societies. It will also include a call for researchers with in-house DRO’s that are not yet available to come forward with an intention to share those.

 

Milestone #4: Broad request for contributions and first new DRO available

A broad request for contributions will be sent including the latest sharing guidelines. Volunteers from the TF with existing DRO’s that are not yet publicly available will follow the sharing guidelines from Milestone #4 to make these available. This will be a limited subset at this stage (one or two) but should provide researchers with enough information to understand the scope of the effort and how they can contribute. The experience and challenges encountered will be documented and used to update the draft recommendations if necessary. The new DRO’s will initially be only available internally within OSIPI until the formal public release of v2.0 of the inventory in Feb 2022.

 

Milestone #5: Extended set of new DRO(s) available

Additional contributions and lessons learned following M4 will be integrated into the inventory structure. A second subset of existing DROs will be targeted for sharing and the steps outlined in the guidance will be followed to make those available and add to the inventory. Sharing recommendations will continue to be updated if necessary. The new DRO’s will initially be only available internally within OSIPI until the formal public release of v2.0 of the inventory in Feb 2022.

Milestone #6: Digital & physical phantom inventory v2.0

Release an updated inventory including latest contributions and additions from the community challenge. All comments received since version 1.0 has been made public will be reviewed and analysed, and the structure of the inventory will be updated accordingly if needed. Additionally the new datasets made available in Milestone #4 and #5 will be added to the inventory. A formal public release v2.0 of the inventory will be created and presented at the perfusion workshop.

Deliverable #2: Recommendations for DRO sharing

Comments obtained from the community after release of the first draft in Milestone 3 will be summarised and used to update the draft of the recommendations, alongside any changes made by the TF during the development of the use cases in milestone #4 and #5. This will now also include a Sustainability Plan outlining a process and resources needed to maintain and add new content to the inventory in the future with a goal that only limited resources will be necessary. Depending on the content and perceived value of the document, this could be published as a preprint or as a peer-reviewed manuscript.

 


Task force 3.2: Clinical and preclinical data

Leads: Andrey Fedorov and Greg Cron

Members: Zaki Ahmed, Beatriz Asenjo Garcia, Greg Cron, Pauline Hall Barrientos, Jochen Hirsch, Thomas Lindner, Felix Navarro, Federico Pineda, Annette van der Toorn, Lutz Ludemann, Anahita Fathi

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) Inventory of repositories

ISMRM Submission deadline

1-Nov-2020

(M2) Structure of perfusion data inventory

1-Feb-2021

(D1) Perfusion data inventory v1.0

ISMRM 2021 (Vancouver)

1-May-2021

(M3) Draft recommendations for sharing

1-Aug-2021

(M4) First dataset available

ISMRM Submission deadline

1-Nov-2021

(M5) All datasets available

Perfusion workshop

1-Feb-2022

(M6) Perfusion data inventory v2.0

ISMRM 2022 (London)

1-May-2022

(D2) Recommendations for data sharing

Summary

The overall aim of TF3.2 is to support activities of the community related to sharing of clinical and preclinical perfusion data. The specific deliverables for this two year cycle are (D1) an inventory of publicly available perfusion datasets, and (D2) recommendations for best practices in sharing perfusion data.

A version 1.0 of the inventory of perfusion data will be delivered in February 2021, listing currently available perfusion datasets identified by the group along with metadata describing their most pertinent characteristics. Since perfusion datasets are commonly offered as part of a larger repository that also contains other types of (imaging) data, the TF will separately maintain an inventory of such larger repositories that contain perfusion data.

The recommendations for perfusion data sharing will be developed after delivery of the first version of the inventory, and will be finalised at the end of the current delivery cycle (May 2022). As part of their development, the recommendations will be applied to a number of test datasets currently held by members of the task force. These applications will serve to inform the further development and refinement of the recommendations and will also produce a second extended version of the perfusion dataset inventory with these new datasets added (February 2022).

Milestone #1: Inventory of repositories that contain perfusion data

A publicly available list of data repositories that contain perfusion data. The inventory will contain URLs to the repositories, and the most important pertinent characteristics of the individual repositories, such as the review process, types of data that are supported by the repository, etc.

Milestone #2: Structure of the inventory of perfusion datasets

A list of attributes that should be captured for individual perfusion datasets, which will be populated in Deliverable #1. The attributes should capture the properties of a perfusion dataset that are important to the individual researcher. Examples could include: peer-reviewed publications describing the dataset, access restrictions (if any), data format, types of data included (e.g., annotations or analysis results, in addition to the imaging data). The attributes will be tested on example datasets and refined if needed. If appropriate, an ISMRM abstract describing the process and results will be considered.

Deliverable #1: Version 1.0 of the inventory of perfusion datasets

A publicly available inventory (list) of publicly available perfusion datasets described using the descriptors defined in Milestone #2. Perfusion datasets that are not part of a wider repository will also be included at this stage. The inventory will be made available either as a webpage linked from the OSIPI website, or as a publicly available linked spreadsheet. Public feedback will be invited and encouraged, and collected systematically.

Milestone #3: Draft recommendations for best practices of sharing perfusion data

A first draft of best practices and actionable guidelines for a researcher willing to share a perfusion dataset publicly. This will include aspects such as recommended repositories, format for image data, format for annotations and analysis results, metadata accompanying the dataset, requirements that are not met in some of the existing repositories. The draft will be open for public feedback and this will be encouraged and promoted via several channels including email to Perfusion study group, presentation on ISMRM, dissemination to other bodies and societies.

Milestone #4: Test implementation of the data sharing best practices (first dataset available)

Volunteers from the TF with the actual needs to share datasets will follow the best practices recommendations developed in Milestone #4. In this first step the datasets to be shared will be identified and one or two will be prioritised to process first. This will be shared and the experience and challenges encountered will be documented and used to update the draft recommendations if necessary.

Milestone #5: Test implementation of the data sharing best practices (other datasets available)

Volunteers from the TF with the actual needs to share datasets will follow the best practice recommendations developed in Milestone #3 and #4. The remaining datasets will be shared following the updated recommendations and the experience and challenges encountered will be documented and used to further update the draft recommendations if necessary.

Milestone #6: Version 2.0 of the inventory of perfusion datasets

All comments received since version 1.0 has been made public will be reviewed and analysed, and the structure of the inventory will be updated accordingly if needed. Additionally the new datasets made available in Milestone #4 and #5 will be added to the inventory.

Deliverable #2: Recommendations for best practices of sharing perfusion data

Comments obtained from the community after release of the first draft in Milestone 3 will be summarised and used to update the draft of the recommendations, alongside any changes made by the TF during the development of the use cases in milestone #4 and #5. Depending on the content and perceived value of the document, this could be published as a preprint or as a peer-reviewed manuscript.

Task force 4.1: ASL lexicon

Leads: David Thomas (UCL, UK), Yuriko Suzuki (Oxford, UK)

Members: Patricia Clement (Ghent, Belgium), Thomas Lindner (Hamburg, Germany), Jan Petr (Dresden, Germany), Xingfeng Shao (USC, USA), Henk-Jan Mutsaerts (Amsterdam, Netherlands), Sudipto Dolui (UPenn, USA).

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) Standard ASL parameters

ISMRM Submission deadline

1-Nov-2020

(M2) Non-standard ASL parameters

1-Feb-2021

(M3) Physiological constants

ISMRM 2021 (Vancouver)

1-May-2021

(D1) ASL Lexicon v1.0 (public draft)

1-Aug-2021

(M4) Codes assigned

ISMRM Submission deadline

1-Nov-2021

(M5) Deadline for public comment

Perfusion workshop

1-Feb-2022

(M6) ASL Lexicon v1.0 (final)

ISMRM 2022 (London)

1-May-2022

(D2) ASL Lexicon v1.0 (manuscript)

Summary

The lexicon developed by TF 4.1 will contain guidelines for reporting of ASL image acquisition, analysis and artefacts in scientific papers and articles. These will enable interoperability and facilitate comparison of results produced by different ASL analysis tools, studies or sites. We aim to harmonize and increase the detail in the description of parameters and configurations, enable encoding of the complete perfusion imaging workflow, and pave the way for consensus building. An important application will be the amendment of the DICOM standard for ASL perfusion parametric maps, as well as BIDS extensions and development of demonstrations and use cases. The 2015 ASL ‘consensus recommendations’ contain a lot of the standard terminology that will be required (doi: 10.1002/mrm.25197). The Lexicon will also be consistent with the ASL-BIDS Extension Proposal currently under development (led by Henk-Jan Mutsaerts and Patricia Clement).

A first draft of the Lexicon ready for public consultation will be produced in May 2021 to be disseminated on the ISMRM in Vancouver. After allowing for a period of public feedback a final version 1.0 will be released in February 2022 along with a covering manuscript to be submitted before May 2022.

Milestone #1: Standard ASL parameters

Compile comprehensive lists of standard ASL acquisition protocol parameters and output parameters (derivatives). Acquisition parameters include those relating to standard labelling schemes (pulsed, continuous, pCASL) and standard readouts i.e. those available as product sequences on each vendor (EPI, 3D GRASE, stack of spiral). Output parameters relating to standard ASL techniques include CBF, ATT, aBV.

Milestone #2: Non-standard ASL parameters

Compile lists of ASL acquisition protocol parameters and output parameters (derivatives) for non-standard techniques such as velocity-selective ASL, acceleration-selective ASL, QUASAR, TRUST, QUIXOTIC (and derivatives), WEP-CAST, diffusion-weighted and T2-weighted ASL, vessel-selective. Expand the lists to areas outside of the brain, e.g. kidney (see recent recommendations).

Milestone #3: Physiological constants

Compile a list of physiological constants required for ASL quantification, such as haematocrit, arterial blood T1/T2 etc, their definition and standard values. The parameters, their definitions or values may be organ-specific (e.g. brain, kidney, heart, placenta etc.) and/or disease-specific (stroke, tumour…)

Deliverable #1: ASL Lexicon v1.0 (public draft)

Draft version of the ASL lexicon that will act as a basis for a public consultation round. The documents from milestone #1-3 will be combined and along with introduction and discussion submitted as v1.0 preprint. Announce at ASL WhatsApp community (before ISMRM) and ask for the feedback. A Slack channel or other mechanism will be created for feedback and discussion. Discussions will be organised at ISMRM

Milestone #4: Codes assigned

Determine which parameters need to be included in DICOM fields. Liaise with TF 4.3 to identify codes for each of those fields.

Milestone #5: Deadline for public comments

Public consultation round will be closed and feedback received will be summarised. At regular intervals prior to the deadline, emails and notifications will be sent out with a reminder of the deadline and to continue raising awareness. Specific experts will also be contacted individually with a personal request to feed back. 

Milestone #6: ASL Lexicon v1.0 (final)

Public comments received will be incorporated to create a final version 1.0 of the lexicon, which will be formally released with a new DOI allocated.

Deliverable #2: ASL Lexicon v1.0 (manuscript)

Write a conceptual article about the rationale and process leading up to the final lexicon with references to the details in the online resource.


Task force 4.2: DCE/DSC lexicon

Leads: Ina Kompan & David Buckley

Members: Laura Bell, Charlotte Debus, Ralf Floca. Andrey Fedorov, David Clunie, Steven Sourbron, Chad Quarles, Petra van Houdt, Ingomar Gutmann, Patrick Thurner, Ben Dickie, Rianne van der Heijden

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) DC Lexicon v0.2 (first draft)

ISMRM Submission deadline

1-Nov-2020

(M2) DC Lexicon v0.3 (revised structure)

1-Feb-2021

(M3) DC Lexicon v0.4 (public draft)

ISMRM 2021 (Vancouver)

1-May-2021

(M4) Deadline for public comments

1-Aug-2021

(D1) DC Lexicon v1.0 (manuscript)

ISMRM Submission deadline

1-Nov-2021

(M5) Software applications (phase 1)

Perfusion workshop

1-Feb-2022

(M6) Software applications (phase 2)

ISMRM 2022 (London)

1-May-2022

(D2) DC Lexicon v2.0

Summary

The Lexicon developed by TF4.2 will constitute a set of guidelines for DCE/DSC to enable interoperability and facilitate the comparison of results produced by different analysis tools, studies or sites. We aim to harmonize and increase the detail in the description of parameters and configurations, enable encoding of the complete perfusion imaging workflow, and pave the way for reproducibility and consensus building. An important application will be the amendment of the DICOM standard for by TF4.3.

The Lexicon was already in development by the start of the OSIPI delivery cycle in May 2022 and available here: http://bit.ly/perfusion-reporting. After collecting comments internally, public comments will be invited in november 2020 and version 1.0 of the Lexicon will be published in may 2021 along with a manuscript describing the process and rationale. At this point TF4.3 will start preparing a DICOM amendment and TF4.2 will apply the Lexicon to the software tools listed in the inventory of TF1.2. This process will conclude with a release of version 2.0 of the Lexicon in May 2022 with these cases studies added.

Milestone 1: DCE/DSC Lexicon v0.2 (draft for internal OSIPI review)

A complete version of the Lexicon will be produced for internal consultation including an optimised structure, update of introductions/discussion and references, and completing the missing sections.

Milestone 2:  DCE/DSC Lexicon v0.3 (after internal OSIPI review)

Comments received from the internal consultation will be incorporated into version v0.3.  A review strategy and discussion channel will be created to receive feedback. A summary of the current lexicon will also be submitted as ISMRM abstract on 16th December 2020

Milestone 3: DCE/DSC Lexicon v0.4 (draft for public review)

A new lexicon draft (v.0.4) for public consultation including an encoding mechanism to produce descriptive pipelines of the analysis will be prepared. A feedback request is sent to the perfusion study group mailing list, including instructions for feedback submission.

Milestone 4: Clean version

Prioritisation of supplementary material to include in version 1.0. A clean version of the covering manuscript and of version 1.0 supplementary material. Dissemination during the ISMRM annual meeting.

Milestone 5: Deadline for public comments

Public consultation round will be closed and feedback received will be summarised and incorporated. At regular intervals prior to the deadline, emails and notifications will be sent out with a reminder of the deadline and to continue raising awareness. Specific experts will also be contacted individually with a personal request to feed back.

Deliverable 1: DCE/DSC Lexicon v1.0 (manuscript)

Create version 1.0 of the lexicon, which will be formally released with a DOI allocated. MRM submission of conceptual lexicon manuscript linking to v1.0. The manuscript will not contain the detailed elements of the lexicon but focus on rationale for the choices that were made and the process leading up to the result.

Milestone 6: Software applications

Apply the Lexicon to describe workflows of open-source perfusion software tools from the DCE/DSC software inventory of TF 1.2. In this first phase a limited number of software tools will be selected as test cases to help develop the process. This process will also be used to pinpoint any reporting gaps of v1.0. The results will also be submitted as to the ISMRM.

 

Deliverable 2: DCE/DSC Lexicon v1.1

First update of the Lexicon including an extra section on applications to DCE/DSC software tools.


Task force 4.3: DICOM standards

Lead: David Clunie & Andrey Fedorov

ISMRM 2020 (Virtual)

1-Aug-2020

ISMRM Submission deadline

1-Nov-2020

1-Feb-2021

ISMRM 2021 (Vancouver)

1-May-2021

1-Aug-2021

(M1) Codes assigned

ISMRM Submission deadline

1-Nov-2021

Perfusion workshop

1-Feb-2022

ISMRM 2022 (London)

1-May-2022

(D1) DICOM change proposal

Summary

Task forces 4.1 and 4.2 will develop consensus guidelines for reporting image acquisition and analysis of ASL and DCE/DSC perfusion imaging, respectively. Task force 4.3 will integrate these into a DICOM amendment. This will enable interoperability and facilitate the comparison of results produced by different analysis tools, studies or sites. Task force 4.3 is dormant until Aug 2021 when the DCE/DSC Lexicon is completed and translation into a DICOM standard amendment will begin. The aim is to deliver a first DICOM amendment submission by the end of this first cycle in may 2022. Presumably this will focus on DCE/DSC as the respective lexicon was already mature in 2020. DICOM amendments for ASL will in that case be referred to the next OSIPI delivery cycle.

Milestone 1: Codes assigned

Structured codes will be assigned to the elements of the Lexicon produced by TF 4.2 to form the basis for a DICOM amendment. Provisional structured codes for the Lexicon of TF4.3 will also be assigned.

Deliverable 1: DICOM change proposal (DCE/DSC)

Proposal for a DICOM amendment incorporating the fields from the DCE/DSC Lexicon v2.0

Task force 6.1: ASL challenges & benchmarks

Leads: Paula Croal (Lead) Udunna Anazodo (Co-Lead)

Members: Henk-Jan Mutsaerts, Jan Petr, Moss Zhao, Flor, Josepha Kennedy McConnell, Joana Pinto, Cassandra Gould van Praag Woods, Andre Paschoal, Maria-Eleni Dounavi, Diego Pineda-Ordóñez. 

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) Challenge design drafted

ISMRM Submission deadline

1-Nov-2020

(M2) DRO developed

1-Feb-2021

(D1) ASL challenge launch

ISMRM 2021 (Vancouver)

1-May-2021

(M3) ISMRM event

(M7) ASL/PET comparison protocol

1-Aug-2021

(M4) Challenge closed

(M8) ASL/PET datasets identified and verified

ISMRM Submission deadline

1-Nov-2021

(M5) Submissions verified

(M9) Pipelines implemented and tested

Perfusion workshop

1-Feb-2022

(M6) Winner announced

(M10) Analyses completed

ISMRM 2022 (London)

1-May-2022

(D2) ASL challenge manuscript

(D3) ASL/PET comparison manuscript

Summary

TF 6.1 will deliver a challenge on ASL image analysis based on a publicly available DRO. Entries will consist of resulting CBF and variance maps, summary of performance metrics, and documented analysis pipeline. The current plan is to focus on single PLD methods but expansion to include multi-PLD will be considered in the early discussions. The challenge will be launched in February 2021 with a 6-month submission window, and results will be published at the end of the current 2-year cycle in may 2022.

A parallel activity in TF 6.1 will be to score pipelines in the inventory produced by TF1.1 by application to healthy and patient data with paired ASL and 15H2O-labelled PET. This activity will be launched in February 2021 after TF1.1 has concluded the v1.0 pipeline inventory, and will conclude with a publication in may 2022.

First objective: ASL challenge

Milestone #1: Challenge design drafted

A draft paper describing the challenge in detail, including a brief literature review on current practices, and the metrics that the challenge outcome will be measured against. These metrics will guide both design of challenge data and determining challenge ‘winners’, and will be included in challenge documentation. The draft paper will be distributed internally in OSIPI to collect feedback before finalising the design.

Milestone #2: Mature Challenge Design  developed

Mature draft of the challenge design developed, to be shared internally/externally for consultation purposes. Milestone report will be in the form of a draft ISMRM abstract, with restricted access, prior to submission in December.  

Deliverable #1: ASL challenge launch

The challenge will be launched by a public announcement and a simultaneous release of the paper describing the challenge (rationale, design, data), the DRO, online documentation and procedures for data submission including anonymisation and GDPR compliance as needed. The challenge will be announced through channels such as the perfusion study group mailing list, ASL network, social media, whatsapp group, OSIPI website, and others.

Milestone #3: ISMRM event

The challenge will be presented during an event on the ISMRM, which could be the study group meeting, a member-initiated symposium, ASL network event, educational series, oral presentation, etc (as appropriate). The event and the run-up to it will form an occasion to advertise and maintain challenge presence while it is open, including reminders of the deadline.

Milestone #4: Challenge closed

The challenge will no longer accept new submissions. This deadline is fixed up front and will not be flexible to avoid creating unfair advantages. Reminders of the deadline will be circulated with increasing frequency in the run-up to the deadline. Some descriptive parameters will be released on the website immediately after closure (nr of submissions, geographical distribution, etc).

Milestone #5: All submissions verified

All entries will be verified according to preset criteria and procedures which will include an implementation and rerun of the pipeline as validation, check of proper anonymisation, summary of performance metrics.

Milestone #6: Challenge winner announced

After verification and validation the results will be analysed statistically and interpreted, and the submissions will be ranked. The ranking and winner will be publicly announced, ideally during an event on the ISMRM perfusion workshop planned around this time.

Deliverable #2: ASL challenge Manuscript

The design and results of the challenge will be written up for publication in a peer-reviewed journal.

Second objective: ASL/PET pipeline comparison

Milestone #7: ASL/PET comparison protocol written

The approach to the ASL/PET comparison will be written out in detail and published before starting the work. This will include detail on the approach to selecting and implementing pipelines, collecting and curation of data to be analysed, approach to data sharing and quality assurance, performance metrics to be collected, independent verification of the results.

Milestone #8: ASL/PET datasets collected and verified

The patient and volunteer data to be used for the validation will be identified, access rights will be secured and preliminary analyses will be performed on all datasets to test suitability for use in the challenge.

Milestone #9: ASL/PET comparison pipelines implemented and tested

All pipelines to be compared will be implemented and applied to a small selection of training data to verify that they are running as expected.

Milestone #10: ASL/PET comparison analysis completed

All pipelines will be applied to all datasets and results will be summarised and analysed statistically.

Deliverable #3: ASL/PET comparison manuscript

Paper describing the methods, results and main conclusions of the ASL/PET comparison.


Task force 6.2: DSC/DCE challenges

Leads: Anahita Fathi Kazerooni (Lead), Hamidreza Saligheh Rad (Co-Lead)

Members: Laura Bell, Mohammadreza Alviri, Salman Rezaei, Moss Zhao, Xinze Zhou, Hanieh Mobarak Salari, Fardin Samadi, Kaveh Daneshmand, Hamed Hamedian

ISMRM 2020 (Virtual)

1-Aug-2020

(M1) Challenge design drafted

ISMRM Submission deadline

1-Nov-2020

(M2) Challenge tested & draft released

1-Feb-2021

(D1) position paper & challenge launch

ISMRM 2021 (Vancouver)

1-May-2021

(M3) ISMRM event

1-Aug-2021

(M4) First phase closed

ISMRM Submission deadline

1-Nov-2021

(M5) Second phase closed

Perfusion workshop

1-Feb-2022

(M6) Data processed and analysed

ISMRM 2022 (London)

1-May-2022

(D2) DC challenge manuscript

Summary

TF 6.2 will deliver a challenge to compare quantification pipelines for DSC/DCE-MRI in clinical cancer imaging applications. Two separate challenges will be progressed in parallel, one on DSC and one on DCE. Both challenges will be based on a recently released dataset in brain cancer patients, available here, including DCE and DSC in each study participant, along with repeatability data. The data contains “double baseline” multi-parametric MRI images collected on patients with newly diagnosed glioblastoma.  The patients were scanned after surgery but prior to the start of therapy, typically 2-5 days apart. The double baseline scans will allow us to test the reproducibility of CBV and CBF quantification. A combined gradient-echo (GE) and spin-echo (SE) 2D EPI sequence has been performed to acquire DSC-MRI data.

The data will be released in two stages, the first stage allowing sufficient time to set up a dedicated pipeline, the second time with a quick turnaround time for an analysis with the existing pipeline. The challenge will be launched in February 2021 with a position paper, and the final results will be written up in a manuscript to be submitted for publication in may 2022 at the end of the current 2-year cycle for OSIPI. Winners will be announced during the planned perfusion workshop in February 2022.

Milestone #1: DCE challenge format outline

Submission formats and specific guidelines for both challenges, defining evaluation metrics, means of announcing the challenges, means of accepting the submissions, etc.

Milestone #2: DSC/DCE challenge procedures tested

Preparation of in-vivo data for both challenges, including any necessary co-registrations and segmentations. Preparation and testing of a mechanism to download the data and upload the results of the challenge, including documentation and instructions for challenge participants.

Milestone #3: Challenge design drafted

Mature draft of the challenge design including processes, metrics, prepared datasets ready for download, and a design for the DRO to be used.

Deliverable #1: DCE challenge launch

The challenge design (guidelines) will be released publicly and the challenge will open for submissions. The challenge launch will be disseminated through channels such as the perfusion study group mailing list, social media, OSIPI website, and ISMRM annual meeting. We will present the challenge at ISMRM during our e-poster session.

Milestone #4: DCE-MRI challenge scoring of submissions

We will create a task force of evaluators to reproduce the received submissions shortly after receiving the SOPs of challengers for the reproducibility metric. The two neuroradiologists in the task force will choose regions of interest on all the scans independently to be used for scoring purposes. Furthermore, the task force will score the submissions for their accuracy and repeatability within a month of receiving the results by the challengers. The task force will continue dissemination of the challenge during this time.

Milestone #5: DCE-MRI challenge closed

The DCE challenge will no longer accept submissions. The most recent submissions will be scored and rated. A summary of earlier submissions will be prepared. .

Milestone #6: Data processed and analysed

All submitted data have been verified and validated, metrics are extracted and any statistical analysis will be performed.

Deliverable #2: DSC/DCE-MRI challenge manuscripts

One or two manuscripts for DSC and DCE challenges summarizing the methods and results of the challenges. Top 3 contestants can also present at the ISMRM conference.