Status Report: THREDDS
September 2015 - April 2016
Sean Arms, Ethan Davis, Dennis Heimbigner, Ryan May, Christian Ward-Garrison
Activities Since the Last Status Report
The THREDDS Project
The THREDDS Project encompases four projects: netCDF-Java, the THREDDS Data Server (TDS), Rosetta, and Siphon (the Unidata Python client to interact with a TDS). For specific information on Siphon, please see the Python Status Report.
Released netCDF-Java / TDS version 4.6.5 (Stable)
Progress has been made on the following:
- The 4.6.x line of development is now in maintenance mode so that the team can focus on v5.0
Focus netCDF-Java / TDS (Soon-to-be Beta) v5
The THREDDS team is preparing to release a beta version of the THREDDS Data Server version 5.0
Progress has been made on the following:
- New Coverage data type allows for subsetting across array boundaries (often called the “seam” problem)
- Uses the new edal-java based ncWMS 2.0 server, as well as javascript client Godiva3
- CatalogScan feature allows for incremental updating of TDS catalogs without the need to restart Tomcat
- Unit and Integration tests are passing
Dependencies, challenges, problems, and risks include:
- The longer the 4.6.x line of development is maintained, the longer it will take to move forward with the 5.x line of development
- John Caron is now employed by Google, and as such will have minimal to no involvement with future TDS development. Prior to his employment with Google, John was going to continue to extend TDS on a contract basis.
Rosetta
The THREDDS team has extended the transformation capabilities of Rosetta.
Progress has been made on the following:
- Rosetta now supports the Trajectory DSG, which means CSV datasets from aircraft, drifting buoys, radiosondes, etc. can now be transformed into netCDF CF-1.6 compliant files.
- Rosetta now support automatic translation of the “EOL Sounding Composite” ASCII-based format into netCDF CF-1.6 compliant files. This capability was added to support the Data Management Resource Center’s work with Millersville University, in which all radiosonde datasets from the PECAN project are being transformed into CF compliant netCDF files.
Dependencies, challenges, problems, and risks include:
- The discontinuation of the ACADIS project presents funding challenges for work on Rosetta. External funding is being pursued and collaboration opportunities are appreciated.
Ongoing Activities
We plan to continue the following activities:
- Documentation updates - reworking the tutorial material to use Docker
- Maintain thredds.ucar.edu and keep up with the addition of new datasets to the IDD
- Continue development of the TDS python client siphon, as well as potentially extend its functionality to interface with the AWIPS-II EDEX server
- Continue to implement a Rosetta interface for each discrete sampling geometry (DSG) from the CF-1.6 specification (http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#discrete-sampling-geometries) Only the profile DSG is left to implement.
The following active proposals directly involve THREDDS work:
- New EarthCube award: "Advancing netCDF-CF for the Geosciences". This two-year, Unidata lead project will work to extend netCDF-CF conventions in ways that will broaden the range of earth science domains whose data can be represented.
- Beginning the second year of NASA ROSES ACCESS award: "High Performance Multidisciplinary Open Standard Data Services to Serve Terrestrial Environmental Modeling" with USGS CIDA.
- Three EarthCube awards are finishing up on a no-cost extension:
1) EarthCube Building Blocks award: "Integrating Discrete and Continuous Data" with Univ of Texas, Austin and others.
2) EarthCube Building Blocks award: "Specifying and Implementing ODSIP, A Data-Service Invocation Protocol" with OPeNDAP, Inc.
3) EarthCube Building Blocks award: "Deploying Web Services Across Multiple Science Domains" with IRIS, UNAVCO, and others. Period of performances: Oct 2013 - Sept 2015. - Still awaiting decision on NASA ROSES ACCESS proposal:
"Leveraging available Technologies for Improved interoperability and visualization of Remote Sensing and in-situ Oceanographic Data at the PO.DAAC" with JPL/PO.DAAC. [Rosetta] - Submitted new DIBBS Early Implementation proposal with Texas Tech: “CIF21 DIBBs: EI: Enhance Unidata Data Facility With a Scalable Lightning Data Infrastructure and Geospatial Capabilities". [netCDF-Java/TDS/Siphon]
New Activities
Over the next three months, we plan to organize or take part in the following:
- Releasing a beta of TDS 5.0
- Making public a TDS 5.0 Test Server
- Finalize visualization preview of converted data in Rosetta
Over the next twelve months, we plan to organize or take part in the following:
- Create a TDS plugin layer for external services
- Upgrade the ncWMS, ncISO, and other plugin services to use the new TDS 5.x plugin layer
- Incorporate ncSoS into TDS
- Transitioning thredds.ucar.edu to TDS 5.x
- Getting TDS v5.0 to a stable release
- Getting netCDF-Java v5.x to a stable release
Beyond a one-year timeframe, we plan to organize or take part in the following:
- Enable Rosetta to publish to a TDS
- Move to a fully online based training tutorial, reserving in-person, annual training for advanced topics
Relevant Metrics
569 unique IPs started up thredds between October 2015 and March 2016, 84 of which are publicly accessible servers. This information is only known for servers running v4.5.3 and above. The differences in these numbers could be due to:
- Reporting TDS running behind a firewall that does not allow incoming traffic on 80, 8080, 443, or 8443 (the ports tested)
- It might be possible that a TDS running through a proxy server may not been “seen” in this analysis as publically reachable
- People testing the TDS on their local machine, but not actually running a server
Note that the vast majority of the publicly accessible servers are running v4.6.3 or above (v4.6.4 was the most current release during this period, and was released on 11 February 2016 ). This indicates that users and organizations running the TDS tend to follow along closely with the current releases of the TDS.
Strategic Focus Areas
We support the following goals described in Unidata Strategic Plan:
- Enable widespread, efficient access to geoscience data
The work of the THREDDS group is comprised of two main areas: the THREDDS Data Server (TDS) and the Common Data Model (CDM) / netCDF-Java library. The TDS provides catalog and data access services for scientific data using OPeNDAP, OGC WCS and WMS, HTTP, and other remote data access protocols. The CDM provides data access through the netCDF-Java API to a variety of data formats (e.g., netCDF, HDF, GRIB). Layered above the basic data access, the CDM uses the metadata contained in datasets to provide a higher-level interface to geoscience specific features of datasets, in particular, providing geolocation and data subsetting in coordinate space. The CDM also provides the foundations for all the services made available through the TDS.
The data available from the IDD is a driving force on both the TDS and netCDF-Java development. The ability to read all the IDD data through the netCDF-Java library allows the TDS to serve that data and provide services on/for that data.
- Develop and provide open-source tools for effective use of geoscience data
Unidata's Integrated Data Viewer (IDV) depends on the netCDF-java library for access to local data, and on the THREDDS Data Server (TDS) for remote access to IDD data. At the same time, the CDM depends on the IDV to validate and test CDM software. Many other tools build on the CDM / netCDF-Java library (eg ERDDAP, Panoply, VERDI, etc) and on the TDS (ESGF, LAS, ncWMS, MyOcean, etc).
- Provide cyberinfrastructure leadership in data discovery, access, and use
The Common Data Model (CDM) / netCDF-Java library is one of the few general-purpose implementations of the CF (Climate and Forecast) metadata standards. Current active efforts in CF that we are involved with include use of the extended netCDF-4 data model (CF 2.0) and for point data (Discrete Sampling Geometry CF-DSG).
The TDS has pioneered the integration of Open Geospatial Consortium (OGC) protocols into the earth science communities. Strong international collaborations have resulted in WCS and WMS services as part of the TDS.
The CDM and TDS are widely used implementations of the OPeNDAP DAP2 data access protocol. Unidata has worked with the OPeNDAP group to design, develop, and implement a new version of the DAP specification, DAP4, which is now available in the TDS server and the netCDF-Java client software stack.
- Build, support, and advocate for the diverse geoscience community
The THREDDS project is involved in several international standardization efforts (CF, OGC, etc.) which cross-cut a multitude of disciplines, both inside and outside of the geoscience community. The netCDF-Java client library, as well as the TDS often serve as incubators for new pushes in these efforts.
Prepared April 2016