Status Report: netCDF

April - September 2015

Ward Fisher, Dennis Heimbigner, Russ Rew

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Enable widespread, efficient access to geoscience data
   by developing netCDF and related cyberinfrastructure solutions to facilitate local and remote access to scientific data.
   
2. Develop and provide open-source tools for effective use of geoscience data          by supporting use of netCDF and related technologies for analyzing, integrating, and visualizing multidimensional geoscience data; enabling effective use of very large data sets; and accessing, managing, and sharing collections of heterogeneous data from diverse sources.
   
3. Provide cyberinfrastructure leadership in data discovery, access, and use
   by developing useful data models, frameworks, and protocols for geoscience data; advancing geoscience data and metadata standards and conventions; and providing information and guidance on emerging cyberinfrastructure trends and technologies.
   
4. Build, support, and advocate for the diverse geoscience community
   by providing expertise in implementing effective data management, conducting training workshops, responding to support questions, maintaining comprehensive documentation, maintaining example programs and files, and keeping online FAQs, best practices, and web site up to date; fostering interactions between community members; and advocating community perspectives at scientific meetings, conferences, and other venues.

Activities Since the Last Status Report

We are using JIRA, GitHub tools for C, Fortran and C++ interfaces to provide transparent feature development, handle performance issues, fix bugs, deploy new releases and to collaborate with other developers.  Additionally, we are using docker technology to run netCDF-C, Fortran and C++ regression and continuous integration tests.  We currently have 98 open issues for netCDF-C, 18 open issues for netCDF-Fortran, and 3 open issues for netCDF-C++.  The netCDF Java interface is maintained by the Unidata CDM/TDS group (which also uses Jira and GitHub), and we collaborate with external developers to maintain the netCDF Python interface.

• In the netCDF group, progress has been made in the following areas since the last status report:

• We presented on netCDF at the Unidata Python training workshop in July.
• We were invited speakers at the Rocky Mountain Advanced Computing Consortium (RMACC) HPC Conference in August, speaking on netCDF.
• We have incorporated support for the 64-bit-everything netCDF format from the parallel netCDF project at Argonne and Northwestern into a branch of the netCDF project, pending inclusion in the next release.
• Further enhancements to the netCDF documentation.
• Extended continuous integration platforms.
• Collaborated with HDF5 and Engility Corp on a proposal for a DoD-funded project to incorporate new HPC functionality (ExaHDF5) into netCDF.  The  proposal was recently accepted and will be funded.

• Dependencies, challenges, problems and risks include:

• Small group (and shrinking) of developers for supporting large project.
• Dependency on HDF5, controlled by external group.
• Slow progress in user adoption of netCDF-4 features.

Ongoing Activities

We plan to continue the following activities:

• Provide support to a large worldwide community of netCDF developers and users.
• Continue development, maintenance, and testing of source code for multiple language libraries and generic netCDF utility programs.
• Improve organization of Doxygen-generated documentation for netCDF-C and Fortran libraries.

New Activities

Over the next three months, we plan to organize or take part in the following:

• Prepare material for the upcoming AGU and AMS conferences.
• Release the next versions of netCDF-C, netCDF-Fortran, netCDF-C++.
• Modernize the netCDF documentation to provide easy access to documentation for older versions of netCDF.

Over the next twelve months, we plan to organize or take part in the following:

• Begin integration of the upcoming ExaHDF5 features into the netCDF-C, Fortran and C++ interfaces.
• Release an official Windows port of the netCDF-Fortran and netCDF-C++ interfaces.
• Deploy a release with compression competitive with GRIB2.
• Participate in development of new CF 2.0 conventions for climate and forecast simulation output and observational data in netCDF-4 form.
• Continue to encourage and support use of netCDF-4's enhanced data model by third-party developers.
• Create and release online educational material in the form of Youtube video tutorials for using netCDF.

Beyond a one-year timeframe, we plan to organize or take part in the following:

• Implement DAP-4 client support in netCDF C library.
• Implement support for Amazon S3 in the netCDF C library.
• Provide thread-safety for the netCDF C library.
• Improve scalability to handle huge datasets and collections.

Areas for Committee Feedback

We are requesting your feedback on the following topics:

1. Are there any HDF5 features that you wish netCDF supported?
2. Should netCDF be ported to and/or maintained for any other programming computing/development environments?
3. How can we encourage more user testing of the release candidates we provide?

Relevant Metrics

There are currently about 140,500 lines of code in the netCDF C library source.

The Coverity estimate for defect density (the number of defects per thousand lines of code) in the netCDF C library source has been reduced slightly from 0.35 six months ago to 0.34 today. According to Coverity static analysis of over 250 million lines of open source projects that use their analysis tools, the average defect density with 100,000 to 500,000 lines of code is 0.50.

Google hits reported when searching for a term such as netCDF-4 don't seem very useful over the long term, as the algorithms for quickly estimating the number of web pages containing a specified term or phrase are proprietary and seem to change frequently. However, this metric may be useful at any particular time for comparing popularity among a set of related terms.

Currently, Google hits, for comparison, are:

• 535,000 for netCDF-3
• 517,000 for netCDF-4
• 384,000 for HDF5
• 82,700 for GRIB2

Google Scholar hits, which supposedly count appearances in peer-reviewed scholarly publications, are:

• 233 for netCDF-3
• 354 for netCDF-4
• 5,920 for HDF5
• 490 for GRIB2

Prepared  September 2015