NWB U24: User Selection and Prioritization Plan

Overview

The purpose of this document is to define the process by which users, developers, issues, and tools are selected for support for NWB training and integration as part of the NIH project “Advancing Standardization of Neurophysiology Data Through Dissemination of NWB” (FAIN: U24NS120057). The main aims of the project are:

Resource Dissemination

All NWB source code, software, documentation, and training materials are open source and will be accessible to anyone with an internet connection. As such, no prioritization is required for resource distribution. All NWB online resources will be accessible via the nwb.org website and using other appropriate distribution channels, e.g., GitHub or ReadTheDocs. Primary online resources include:

Prioritization of access to in-person training

The general aim is to make in-person training as broadly accessible as possible. As such, the annual researcher and developer training workshops will be advertised as broadly as possible via mailing lists, conferences, and social media. All workshop resources and project pages will be open-source via GitHub and other distribution channels as appropriate (e.g., via Google Drive). If we receive more applicants than we can accommodate, we will prioritize researchers and developers based on the likely impact on adoption of NWB and availability of alternate training resources for the applicants. Applicants who do not have the opportunity to learn about NWB locally have the greatest potential to benefit from a workshop. Once high-priority applicants have been assigned, further selection will be based on a first-come basis.

For the developer workshop, we will prioritize those who have already made concrete contributions to NWB based on their GitHub activity, as well as new developers with the promise for high impact on NWB (e.g., developers seeking to integrate a new tool important to the NWB ecosystem). Our goal is to encourage as many developers as possible to maintain a high level of involvement and facilitate growth and long-term sustainability of the NWB software ecosystem. Once high-priority applicants have been assigned, further selection will be based on a first-come basis.

For virtual training, the team has so far been able to accommodate all participants. We anticipate that prioritization may be necessary once in-person training options are again available. If applicants have to be rejected, the team will aim to make alternate training resources (e.g., virtual attendance options, recordings of talks from the event, or appropriate existing training materials) accessible to those that could not attend.  

Prioritizations of engagements, issues, and needs

Given our experience developing NWB for the last three years, we anticipate that the main resource requiring prioritization will be personnel hours required for direct engagement to support neuroscience labs and developers. NOT-MH-19-010 has outlined clear goals for adoption of NWB and DANDI that will guide our user prioritization of one-on-one consultations with users. The notice requires neurophysiology labs with BRAIN project proposals after March 1, 2020 to publish their data to DANDI in NWB upon publication or at the end of the grant funding period. It also establishes an expectation that labs deposit their data on DANDI in embargo mode during the course of the projects, at 6-month intervals. The proposals under this new requirement just started being funded in about January 2021. If we estimate that the projects are being set up for the first 6 months, then we might expect the first data from projects to be deposited in January 2022. This is a dramatic shift in data management that will require substantial effort from the data producers as well as the NWB and DANDI development teams, and we are committed to ensuring that BRAIN neurophysiology researchers have the support to meet this ambitious goal.

The sequence of funded projects creates for us a prioritization plan for who to help in individual consultations. We plan to reach out to each funded project within 1 year of their initial funding date, urging them to start preparing for this requirement and attempt to schedule individual consultations. During these consultations, we will learn about their data management needs, help them decide if NWB and DANDI are appropriate for them and point them, and direct them to tutorials and existing software that might help them meet the new requirement.

While we are committed to conducting consultations with each new neurophysiology BRAIN project, in our experience the amount of support needed for each group can vary greatly. Some groups are able to accomplish their goals from simple generalization of existing tutorials, and some require cross-group collaboration, in-depth extensions, and infrastructural development that can take months. Also, as the number of projects/labs requiring support grows under NOT-MH-19-010, the issues and support needs that arise may outpace the available personnel resources available as part of the U24NS120057 project.

Therefore, we need an additional prioritization plan to help us evaluate the cost/benefit of these projects in order to devote our development efforts in directions that would most benefit the neurophysiology community as a whole. Fig. 1 outlines the principal process for prioritization of issues and needs as part of the support for neuroscience labs and tool developers (Task 1B, 2B) and the integration of NWB with new neuroscience technologies and community tools (Task 1C, 2C). In this process, new or more specific issues and needs may be identified at any stage of the process (arrows not shown in Fig. 1 to improve readability). For example, even if an issue is closed after step <1>, it may still result in the identification of a more specific issue/need relevant to NWB, restarting the decision process for the new issue/need.

We strive to keep the prioritization process as open and transparent as possible. To achieve this goal we manage issues on GitHub so that they are openly accessible to the community and retained for future reference. As part of the U24 project, the team seeks to further refine this process by adding support for GitHub project boards and more rigorous policies for assigning labels, milestones, and projects to issues.

For development of new data types and changes to the schema, we will follow our policies and community-driven processes for integration of new neuroscience technologies (available in the “Policies” section of the nwb.org website) using open working groups and the Neurodata Extensions (NDX) Catalog to evaluate extensions and proposals in an open and transparent process.

The team will prioritize issues and needs in a timely manner as new issues are submitted or identified by team members. Initial prioritization will typically be performed by individual members of the technical development team. Issues requiring further discussion (e.g., due to the need for significant resources (>2 days) or issues with broader impact on users) will be reviewed with the U24 leadership team during regular team meetings or via GitHub issues, as appropriate. The team will review priorities regularly (at least every 6 month). For prioritization and selection of tools, the team may also proactively engage with development teams of tools of high interest to the NWB community to proactively fill critical gaps in the NWB community software ecosystem.

Engagement and Decision Process

Evaluation and Prioritization Criteria (Aim 1 and 2)

  • Is the issue/need relevant to NWB?

  • Is the issue/need specific to NWB?
  • Can the NWB team resolve the issue/need?

 Issue prioritization strategy:

  • Critical: Impacts core function of NWB
  • High: Impacts feature important to most users
  • Medium: Addresses non-critical problem and/or affects only a small set of NWB users
  • Low: Alternative solution already working; Issue relevant to only a specific user

  • Is the issue/need related to the NWB software/standard?
  • Does the issue/need add value to the NWB mission? Does it:

(a) Make new/improved resources available to the NWB community, e.g., release of data or software?

(b) Enable new scientific insights?

(c) Facilitate NWB adoption?

(d) Further the NIH BRAIN Initiative mission?

  • Do multiple viable solutions exist?
  • Is further planning required?
  • Does the plan affect core NWB technologies and/or the NWB community?