Status Report: Data Standards and Technical Outreach

June 2022 - October 2022

Ethan Davis, Ward Fisher, Hailey Johnson, Dennis Heimbigner, and Ryan May

Areas for Committee Feedback

We are requesting your feedback on the following topics:

No requests currently.

Activities Since the Last Status Report

NCZarr/Zarr Specification Efforts

As part of implementing Zarr support in both the netCDF-C and -Java libraries, the NCZarr convention/extension has been developed to provide a clean and complete mapping between the netCDF and Zarr data models. During this work, the netCDF developers have been participating in discussions around clarification and evolution of the Zarr (version 2 and 3) specifications.

Progress has been made on the following:

• The Zarr community has added an Implementation Council to the Zarr governance structure. Dennis will represent the netCDF-C library and Hailey will represent the netCDF-Java library on the Zarr Implementation Council.

CF Conventions for netCDF activities

Unidata has a long history of involvement in the development of the Climate and Forecast (CF) Conventions for netCDF. These efforts continue with ongoing participation in development conversations on the CF GitHub repositories, participation in and help in organizing the annual CF Workshops, and participation in the governance of CF.

Progress has been made on the following:

• The 2022 CF Workshop was held 13-15 Sept 2020 in Santander, Spain.
• Ethan Davis continues serving as chair of the CF Governance Panel.

WMO Task Team for CF-netCDF

The WMO Expert Team on Data Standards (ET-Data) was formed in late 2020 and tasked with maintaining and developing the various WMO data standards. The newly formed Task Team for CF-netCDF (TT-CFNetCDF) is one of several ET-Data Task Teams and is tasked with developing WMO profiles detailing how WMO will store data in CF-netCDF. So far the TT-CFNetCDF has developed WMO Profiles for radar data (based on CF-Radial) and oceanographic glider data and is working on a WMO Profile for aircraft data. The radar and glider profiles have been approved for experimental distribution on the WMO Information System (WIS) 2.0. The WIS 2.0 provides similar functionality to the GTS as well as more interactive access to data and is intended to eventually replace the GTS.

Progress has been made on the following:

• Ethan Davis continues as a member of the TT-CFNetCDF (and ET-Data).
• Taking part in discussions of how WMO CF-netCDF profiles and the WIS 2.0 transition will impact and benefit the University community.

Dependencies, challenges, problems, and risks include:

• Possible direct connection between LDM/IDD and WIS 2.0 would likely require development work.
• WMO moves very slowly, WIS 2.0 transition likely years away.

Updating and Reorganizing NetCDF User's Guide (NUG)

The NetCDF User’s Guide (NUG) was initially developed when the netCDF-C library was the only netCDF implementation. While many sections of the NUG apply to all netCDF implementations (e.g., the data models and the file format descriptions), these more abstract parts of the current NUG are often still intertwined with netCDF-C implementation details. Similarly, support, development, and advancement of the NUG have also been intertwined with the netCDF-C library.

The goals of this work are to

1. Separate the aspect of netCDF that are useful to any user/developer, independent of which library or tool they use (i.e., data model, file formats, CDL definition, conventions, and best practices) from those that are library or language specific and
2. Clarify where and how the netCDF community can ask questions about the NUG as well as discuss and contribute to the development and advancement of the NUG.

Progress has been made on the following:

• Work on this effort has been slow but we plan to have a draft version of a library independent NUG document (GH repo)  available in the next six months.

Registering netCDF Media Type (application/netcdf) with IANA

The idea of registering a netCDF media type has been discussed a number of times over the years but never gained the momentum needed to undertake the effort. A request from the group developing the OGC Linked Data in NetCDF standard (and their offer of assistance) initiated the current effort to officially register the netCDF media type with IANA.

Progress has been made on the following:

• The netCDF media type ("application/netcdf") has been added to IANA's provisional registry list with Unidata listed as the standards-related body supporting the effort.
• Documents in support of this effort are being developed in the new Unidata/netcdf GitHub repo (see PR #45).

Ongoing Activities

We plan to continue the following activities:

• Represent Unidata in Earth System Information Partners

• Unidata has been a Type II ESIP Partner Organization since 1999
• Ethan Davis is currently the Unidata voting representative to ESIP.

• Represent UCAR and Unidata in OGC and various OGC working groups

• Ethan Davis is the UCAR voting representative to the OGC Technical Committee, Jeff de la Beaujardiere (NCAR/CISL) is alternate voting representative.
• Participate in OGC MetOcean Domain Working Group (DWG) meetings.
• Ethan Davis is co-chair of the OGC netCDF Standards Working Group (SWG)
• Track and participate in the OGC Environmental Data Retrieval (EDR) SWG meetings.
• Track and participate in OGC Community Standard process for CoverageJSON.

New Activities

Over the next three months, we plan to organize or take part in the following:

• Deploy a draft version of the new, library independent, NetCDF User’s Guide (NUG).

Over the next twelve months, we plan to organize or take part in the following:

• Submit request for full registration of the netCDF media type with IANA

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Managing Geoscience Data
   Unidata's various data standards efforts contribute to important tools for data producers, especially those that design and develop new data products, and for those that develop software tools for data management, analysis, and visualization.

Prepared October 2022