Strategies for sharing NWB extensions (NDX)

1. Overview

The purpose of this document is to define standard practices and strategies for sharing format extensions for NWB, so called Neurodata Extensions (NDX).

1.1 Definitions

1.2 Summary

NWB defines a set of Versioning Guidelines for versioning extension namespaces and Sharing Guidelines that public NDX MUST follow and internal NDXs are highly RECOMMENDED to follow. In order to help ensure compliance with these rules, NWB provides a number of tools, templates, and guidelines.

2. Methods and tools for sharing extensions

2.1 nwb-extensions: Extensions GitHub Organization

NDXs are shared via a dedicated nwb-extensions GitHub organization for the NDX Catalog. The NDX Catalog provides the ndx-template and staged-extensions extension template repositories (see below) as well as all public ndx-custom-record repositories. In this way, all public extensions are managed in one place and MUST follow the same basic internal organization. The public ndx-custom-record repositories contain only metadata about the extensions, such as installation instructions and URLs to the extension sources. This strategy allows for labs, universities, and independent groups to maintain the source code for their extensions in their own Git space. The ability to do so can be important in particular when the developers of the extension are funded by their own grants and/or are applying for funding.

2.1.1 ndx-template: Extension template

The ndx-template provides a standard Git template repository with a standard setup for creating and sharing extensions and instructions for how to use the template. Using the cookiecutter tool, users can easily modify the template to create the basic setup for their ndx-template repository. See the ndx-template repository for details on the organization of the template.

2.1.2 staged-extensions: Creating custom ndx records

The staged-extensions repository is a place to submit new NDX records for registration with the NDX Catalog, i.e., before a ndx-custom-record repo exists for the extension. The repository holds an example record which should be copied and modified for a new extension to be registered. See the staged-extensions repository for details on how to use the repo for registering extensions with the NDX Catalog.

2.2.3 ndx-custom-record: Custom extension catalog record

The NDX Catalog contains a collection of ndx-custom-record repositories created by the creators of the extensions. See below for further details.

2.3 Custom Extension Code Repositories

2.3.1 ndx-custom-record: Custom extension catalog record

The ndx-custom-record is created from the staged-extensions repository as part of the NDX catalog and MUST have the same name as the main namespace of the NDX. This strategy helps ensure that there are no name conflicts between public NDX namespace names. The ndx-custom-record repository MUST contain the following:

In addition the ndx-custom-record may contain the following OPTIONAL documents related to the review of the extensions:

2.3.2 ndx-custom: Custom NWB Extensions

The ndx-custom repository contains the sources of the user extension that has usually been created from the ndx-template. This repository lives in the user’s own Git space and is NOT part of the nwb-extensions organization. This provides users the ability to maintain ownership of the extension.

2.4 Additional Tools and Methods

2.4.1 Documenting extensions

The nwb-docutils provide tools for generating Sphinx RST documentation directly from the YAML sources. The “generate_format_docs.py” command-line tools MUST be used to auto-generate documentation from the YAML sources.

As part of the ndx-template Git repository, NWB provides a standard setup for generating Sphinx documentation for an extension based on the nwb-docutils. Extensions MAY customize the documentation but MUST follow the standard for building and generating the extensions.

2.4.2 Creating extension specifications

Extensions SHOULD be created using the PyNWB specification API to improve maintainability of the extensions and to insulate the creators of the extensions from low-level details of the NWB specification language (see the PyNWB docs and tutorial for details). The sources for creating the extensions SHOULD be shared along with the extension YAML specification files to facilitate maintenance and compliance with the NWB specification language. Extensions MAY also be created by just sharing YAML files (e.g., written by hand) that are compliant with the NWB specification language, however, this is NOT RECOMMENDED. The ndx-template includes dedicated locations for the YAML specification and sources, and includes a template Python file for implementing specifications using PyNWB.

3. Extension process (create, deploy, update)

The figure below illustrates the process for (1) creating a new extension, (2) creating a record to register an extension with the extension catalog, and (3) updating an extension/record. The figure also illustrates the automated CI processes that are managed in the nwb-extensions catalog. The catalog process is modeled after the conda-forge model, enabling us to largely automate the catalog process using free, public services and avoid the need to host our own services.

NDX Catalog Workflow

3.1 Continuous Integration (CI) for Extensions

This can be separated into CI for three main repositories:

  1. the staged-extensions repository as part of the NDX Catalog,
  2. the ndx-custom-record repo as part of the NDX Catalog, and
  3. the user’s ndx-custom repository.

Parts 1 and 2 are part of the NDX Catalog and are maintained by the NWB development team. Part 3 will need to be set up and maintained by the extension developers. The ndx-template can provide templates for defining CI for ndx-custom repositories, but ultimately this will be up to the extension developers to do.

3.1.1 staged-extensions CI

For the staged-extensions repository, we have common tests to validate basic compliance and working order of the extensions before creating the ndx-custom-record repo. Here we can test, e.g., the extensions metadata, i.e., are all required fields for the catalog specified and valid.

3.1.2 ndx-custom-record CI

Similar to the staged-extensions, we can provide CI for ndx-custom-record repositories to automatically test compliance and working order of the extensions before merging pull requests. Here we can test, e.g., the extensions metadata, i.e., are all required fields for the catalog specified and valid.

3.1.3 ndx-custom CI

Here we provide example configuration files and instructions (e.g., as part of the ndx-template repository), but ultimately this is up to the user to set up and maintain. The ndx-template repo itself will likely not include CI itself. However, it may include templates for setting up CI for ndx-custom user extension repositories. See the ndx-template repo for details. We generally recommend to use CI to: