Status Report: netCDF

October 2017 - March 2018

Ward Fisher, Dennis Heimbigner

Areas for Committee Feedback

We are requesting your feedback on the following topics:

1. To what extent is Amazon S3 used within your organization? Would you benefit from native netCDF support for S3 storage?
2. Are there other cloud-based block storage formats/locations (zarr, Azure, etc) that are actively in use? That we should consider storing.
3. Are there any emergent avenues (stack overflow, etc) for user support which the netCDF team should investigate?
4. How can we encourage more user testing of the release candidates we provide?

Activities Since the Last Status Report

We are using GitHub tools for C, Fortran and C++ interfaces to provide transparent feature development, handle performance issues, fix bugs, deploy new releases and to collaborate with other developers.  Additionally, we are using docker technology to run netCDF-C, Fortran and C++ regression and continuous integration tests.  We currently have 108 open issues for netCDF-C, 23 open issues for netCDF-Fortran, and 16 open issues for netCDF-C++.  The netCDF Java interface is maintained by the Unidata CDM/TDS group and we collaborate with external developers to maintain the netCDF Python interface.

 In the netCDF group, progress has been made in the following areas since the last status report:

• Addition of user-defined compression filters.
• Work towards enhanced parallel I/O.
• Further extension of the netCDF build-and-test platforms using Docker technology.
• Further enhancements to the netCDF documentation.
• Extended continuous integration platforms have been adopted.
• An architecture roadmap is available describing how the netcdf-c library will support thread-safe operation in *nix* and Windows environments. The draft proposal is available as netcdf-c github issue #382.
• We have seen an uptick in the number of contributions to the netCDF code base(s) from our community.  While these contributions require careful review and consideration, it is encouraging to see this model of development (enabled by our move to GitHub) being more fully embraced by our community.  
• Increased support for native CDF5 on 64-bit platforms, potentially obviating the need for the parallel netcdf library for reading/writing CDF5 files in serial fashion.

Dependencies, challenges, problems and risks include:

• Small group (and shrinking) of developers for supporting large project.
• Dependency on HDF5, controlled by external group.
• Slow progress in user adoption of netCDF-4 features.

Ongoing Activities

We plan to continue the following activities:

• Provide support to a large worldwide community of netCDF developers and users.
• Continue development, maintenance, and testing of source code for multiple language libraries and generic netCDF utility programs.
• Improve organization of Doxygen-generated documentation for netCDF-C and Fortran libraries.
• Extend collaboration as opportunities arise, for increasing the efficiency of parallel netcdf-3 and netcdf-4.

New Activities

Over the next three months, we plan to organize or take part in the following:

• Seek out, and prepare material for upcoming, conferences and other outreach opportunities.
• Work on reducing the defects reported by static analysis.
• Release the next versions of netCDF-C, netCDF-Fortran, netCDF-C++.
• Modernize the netCDF documentation to provide easy access to documentation for older versions of netCDF.
• Provide thread-safety for the netCDF C library.

Over the next twelve months, we plan to organize or take part in the following:

• Release an official Windows port of the netCDF-Fortran and netCDF-C++ interfaces.
• Participate in development of new CF 2.0 conventions for climate and forecast simulation output and observational data in netCDF-4 form.
• Continue to encourage and support use of netCDF-4's enhanced data model by third-party developers.
• Create and release online educational material in the form of Youtube video tutorials for using netCDF.

Beyond a one-year timeframe, we plan to organize or take part in the following:

• Implement support for Amazon S3 in the netCDF C library.
• Improve scalability to handle huge datasets and collections.
• Improve the efficiency of parallel netcdf3 and parallel netcdf4.

Relevant Metrics

There are currently about 183,700 lines of code (up from 142,810 lines of code) in the netCDF C library source. The Coverity estimate for defect density (the number of defects per thousand lines of code) in the netCDF C library source has been increased slightly from 0.36 six months ago to 0.7 today. According to Coverity static analysis of over 250 million lines of open source projects that use their analysis tools, the average defect density with 100,000 to 500,000 lines of code is 0.50.  The jump in defect density is a result of the addition of the DAP4 code. As this is new code, the initial defects are still being worked out.

Google hits reported when searching for a term such as netCDF-4 don't seem very useful over the long term, as the algorithms for quickly estimating the number of web pages containing a specified term or phrase are proprietary and seem to change frequently. However, this metric may be useful at any particular time for comparing popularity among a set of related terms.

Currently, Google hits, for comparison, are:

• 946,000 for netCDF-3
• 937,000 for netCDF-4
• 507,000 for HDF5
• 94,600 for GRIB2

Google Scholar hits, which supposedly count appearances in peer-reviewed scholarly publications, are:

• 307 for netCDF-3
• 593 for netCDF-4
• 10,200 for HDF5
• 848 for GRIB2

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Enable widespread, efficient access to geoscience data
   by developing netCDF and related cyberinfrastructure solutions to facilitate local and remote access to scientific data.
   
2. Develop and provide open-source tools for effective use of geoscience data          by supporting use of netCDF and related technologies for analyzing, integrating, and visualizing multidimensional geoscience data; enabling effective use of very large data sets; and accessing, managing, and sharing collections of heterogeneous data from diverse sources.
   
3. Provide cyberinfrastructure leadership in data discovery, access, and use
   by developing useful data models, frameworks, and protocols for geoscience data; advancing geoscience data and metadata standards and conventions; and providing information and guidance on emerging cyberinfrastructure trends and technologies.
   
4. Build, support, and advocate for the diverse geoscience community
   by providing expertise in implementing effective data management, conducting training workshops, responding to support questions, maintaining comprehensive documentation, maintaining example programs and files, and keeping online FAQs, best practices, and web site up to date; fostering interactions between community members; and advocating community perspectives at scientific meetings, conferences, and other venues.

Prepared  March 2018