Status Report: Internet Data Distribution

April 2017 - October 2017

Mike Schmidt, Jeff Weber, Steve Emmerson, Tom Yoksas

Activities Since the Last Status Report

Internet Data Distribution (IDD)

IDD data volumes continue to increase.  The following output is from a Linux-based motherlode clone that the UPC operates on behalf of the community, lead.unidata.ucar.edu:

``bqb

Data Volume Summary for lead.unidata.ucar.edu

Maximum hourly volume  72480.665 M bytes/hour
Average hourly volume  49696.601 M bytes/hour

Average products per hour     390913 prods/hour

Feed                           Average             Maximum     Products
                    (M byte/hour)            (M byte/hour)   number/hour
FSL2                  14651.912    [ 29.483%]    17446.562    22399.975
CONDUIT                7887.856    [ 15.872%]    21436.385    87792.900
NGRID                  7596.554    [ 15.286%]    13110.295    40235.375
EXP                    6085.977    [ 12.246%]     8779.037     3440.750
NEXRAD2                5387.548    [ 10.841%]     7704.114    62359.450
NOTHER                 3699.461    [  7.444%]     7465.859     7456.375
NEXRAD3                1803.479    [  3.629%]     2401.904    90171.375
FNMOC                  1214.143    [  2.443%]     4750.510     3301.400
HDS                     749.161    [  1.507%]     1156.088    27504.625
GEM                     203.950    [  0.410%]     1157.301     1229.975
NIMAGE                  156.532    [  0.315%]      349.965      180.950
FNEXRAD                 123.776    [  0.249%]      153.917      103.850
IDS|DDPLUS               68.174    [  0.137%]       81.383    44113.425
UNIWISC                  64.426    [  0.130%]      120.964       46.875
LIGHTNING                 3.654    [  0.007%]        9.726      575.700
``bqe

``bqe

Ongoing Activities

We plan to continue the following activities:

• Unidata continues to receive the NOAA/GSD experimental High Resolution Rapid Refresh (HRRR) grids (both 2D and 3D fields) in an LDM/IDD  feed from NOAA/GSD and feed these products to a small number of university sites on hrrr.unidata.ucar.edu.

• The NCEP operational HRRR is being served with other model output at: 

                http://thredds-jumbo.unidata.ucar.edu/thredds/catalog/idd/forecastModels.html         

(.xml for machine access)

• Other data sets we continue to explore with NOAA/GSD/ESRL are:

• FIM
• HIWPP

• NCEP (operational) HRRR fields and forecasts times have been added to the CONDUIT IDD datastream.

NOAAPort Data Ingest

• Ingest of the upgraded NOAAPort Satellite Broadcast Network (SBN) products and their relay to end-users via the IDD has been “operational” at the UPC since the August 2014.

• The NOAAPort-derived data streams (HDS, IDS|DDPLUS, NGRID, NIMAGE, NEXRAD3 and NOTHER) are being redundantly injected into  the IDD at three geographically separate locations: Unidata, UW/SSEC, and LSU/SRCC.  The NOTHER data stream contains GOES-16 tiles that need to be stitched together to make full image scenes usable to end-user applications.

We continue to look for a fourth ingest site to increase robustness of the IDD distribution of NOAAPort derived data.

• Unidata's NOAAPort ingest package is bundled with current versions of the LDM.  The current LDM release is v6.13.6.

• Raytheon continues to submit modifications to the LDM for enhanced AWIPS functionality.

Relevant Metrics

• Approximately 590 machines at 250 sites are running LDM-6 and reporting real time statistics to the UPC. Unidata staff routinely assist in the installation and tuning of LDM-6 at user sites as a community service.

• A number of organizations/projects continue use the LDM to move substantial amounts of data that do not report statistics  to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private companies, etc.).

• UCAR IDD toplevel relays, idd.unidata.ucar.edu and iddb.unidata.ucar.edu

The IDD relay cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more  than 1250 downstream connections.

Over the period from April 1  through September 30, 2017 the average volume of LDM/IDD data flowing from the UCAR/NCAR network averaged around 3 Gbps (~31 TB/day), and peak rates reached 6 Gbps (which would be ~63 TB/day if the rate was sustained).

Cluster real server backends and accumulator nodes routinely have instantaneous output volumes that can exceed 1.5  Gpbs.  Bonding of pairs of Ethernet interfaces was needed to be able to support these output data rates.  The next generation of cluster machines will need to have 10 Gbps Ethernet capability.

The 6-8 GB/hr  increase in IDD data volume shown in the volume list for lead.unidata.ucar.edu reflects the test relay of GOES-16 data.

Questions for Committee Members

• Suggestions regarding content of data streams like CONDUIT, FNEXRAD, UNIWISC and NLDN LIGHTNING? ... We (UPC, the Unidata community and UAlbany for the NLDN component of LIGHTNING) have control of the content of these data streams, so they are open for suggestions.
• We have been preparing the top level IDD relay clusters we maintain to support relay of the full volume of GOES-16 imagery and products being received on the GOES-R downlink that we installed at the NCAR Mesa Lab.  The data volume for this future feed can be seen in the EXP feed in the volume listing for lead.unidata.ucar.edu.  The question for the governing committees is if relay of the GOES-16 data would be desired by the community, and  if end users can handle the data volume.

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Enable widespread, efficient access to geoscience data
   A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments.
   
2. Develop and provide open-source tools for effective use of geoscience data
   The IDD is powered by the Unidata LDM-6 which is made freely available to all. The Unidata NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages are bundled in AWIPS.

3. Provide cyberinfrastructure leadership in data discovery, access, and use
   The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers.
   
4. Build, support, and advocate for the diverse geoscience community
   Providing access to data in real-time is a fundamental Unidata activity.

The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa. The Universidad de Costa Rica is actively pursuing IDD relay in the Latin America based IDD-Caribe.

Prepared  October 2, 2017