Status Report: Internet Data Distribution

September 2015 - March 2016

Mike Schmidt, Jeff Weber, Steve Emmerson, Tom Yoksas

Activities Since the Last Status Report

Internet Data Distribution (IDD)

0.25 degree GFS data was added to the CONDUIT data stream starting with the 12Z run on July, 28 . Monitoring has shown that peak CONDUIT data volumes increased from approx. 8 GB/hr to approx. 21 GB/hr for all forecast hours for the 0.25 degree GFS.

``bqb

Data Volume Summary for emo.unidata.ucar.edu

Maximum hourly volume  38312.123 M bytes/hour
Average hourly volume  19769.266 M bytes/hour

Average products per hour     327728 prods/hour

Feed                           Average             Maximum     Products
                    (M byte/hour)            (M byte/hour)   number/hour
CONDUIT                7878.758    [ 39.854%]    21708.893    89182.364
NGRID                  5206.945    [ 26.339%]     8596.371    37461.750
NEXRAD2                2836.811    [ 14.350%]     4648.726    41083.250
NEXRAD3                1578.781    [  7.986%]     1886.712    89978.795
FNMOC                  1274.134    [  6.445%]     3163.891     3624.205
HDS                     282.288    [  1.428%]      579.506    19087.614
NOTHER                  274.371    [  1.388%]      783.128     1360.091
NIMAGE                  155.168    [  0.785%]      251.517      181.523
GEM                      76.094    [  0.385%]      474.508      792.295
IDS|DDPLUS               65.822    [  0.333%]       81.338    44297.227
EXP                      47.827    [  0.242%]       91.171      328.068
FNEXRAD                  45.000    [  0.228%]       83.920       43.614
UNIWISC                  44.757    [  0.226%]       90.713       21.386
LIGHTNING                 2.395    [  0.012%]        6.377      284.977
GPS                       0.113    [  0.001%]        0.997        1.045

``bqe

Recently, top level IDD relays and the sites that they are feeding CONDUIT data to have been experiencing unusually high latencies that correspond with the transmission of the 0.25 degree GFS data.  Current testing suggests that a large fraction of the latencies being experienced originate at or near NCEP.  Investigations are ongoing.

Ongoing Activities

We plan to continue the following activities:

• Unidata continues to receive High Resolution Rapid Refresh (HRRR) grids (both 2D and 3D fields) in an LDM/IDD  feed from NOAA/GSD and feed these products to a small number (3) of university sites on hrrr.unidata.ucar.edu. Since HRRR and ESTOFS data were added to the NOAAPort Satellite Broadcast Network (SBN) in late September, 2014, continuing to relay the HRRR ingested from NOAA/GSD is considered to be of lesser importance and will be discontinued if the sites currently receiving the NOAA/GSD data are amenable.

• The HRRR is being experimentally served at:          http://thredds-jumbo.unidata.ucar.edu/thredds/modelsHrrr.html 

(.xml for machine access)

• Other data sets we are actively exploring with NOAA/GSD/ESRL are:

• FIM
• HIWPP

• HRRR and ESTOFS products were added to NOAAPort in late September, 2014. The following TINs announced these additions:

http://www.nws.noaa.gov/os/notification/tin14-28hrrr-cca.htm

http://www.nws.noaa.gov/os/notification/tin13-43estofs_noaaport_aaa.htm

Briefly, these additions are comprised of:

• HRRR: 81 products, hourly F00-15 each hour. CONUS 2.5km grid184. ~44 GB/day
• ESTOFS: 3 products, hourly F00-F180, 00, 06, 12, 18z runs. CONUS 2.5km grid, Puerto Rico 1.25 km grid. ~2 GB/day

• HRRR fields and forecasts times that are not included in the NOAAPort expansion will be evaluated as additions to the CONDUIT IDD datastream.

• The UPC continues to relay FNMOC and CMC data model output directly to the community. FNMOC provides the COAMPS and NAVGEM model output and the CMC provides the GEM model output. Unidata has provided access to these data for the          past 8 years, but on a "point-to-point" basis. GEM model output was converted from GRIB1 to GRIB2 in January, 2015. The CMC is now feeding output of their new hi-resolution (15 km) GEM model to Unidata who, in turn, relays the data to IDD participants.

NOAAPort Data Ingest

• Ingest of the upgraded NOAAPort Satellite Broadcast Network (SBN) products and their relay to end-users via the IDD has been “operational” at the UPC since the August 2014.

• The NOAAPort-derived datastreams (HDS, IDS|DDPLUS, NGRID, NIMAGE, NEXRAD3 and NOTHER) are being redundantly injected into  the IDD at three geographically separate locations: Unidata, UW/SSEC, and LSU/SRCC.  Attempts to add a Northrop Grumman office in Northern Virginia was not successful due to their having very limited network bandwidth.  We are still seeking a fourth ingest site to increase robustness of the IDD.

• Unidata's NOAAPort ingest package is bundled with current versions of the LDM.  A new LDM release, v6.13, will be released real soon now.

• Raytheon continues to submit modifications to the LDM for enhanced AWIPS-II functionality.

Relevant Metrics

• Approximately 565 machines at 253 sites are running LDM-6 and reporting real time statistics to the UPC. Unidata staff routinely assist in the installation and tuning of LDM-6 at user sites as a community service.

• A number of organizations/projects continue use the LDM to move substantial amounts of data that do not report statistics  to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private companies, etc.).

• UCAR IDD toplevel relay node, idd.unidata.ucar.edu

The IDD relay cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more  than 1250 downstream connections.

Data input to the cluster nodes averages around 20 GB/hr (~0.5 TB/day);  average data output from the entire cluster exceeds 2.9 Gbps (~32 TB/day); peak rates routinely exceed 6.4 Gbps (which would be ~70 TB/day if the rate was sustained).

Cluster real server backends and accumulator nodes routinely have instantaneous output volumes that exceed a Gpbs.  Bonding of pairs of Ethernet interfaces was needed to be able to support these output data rates.  The next generation of cluster machines will need to have 10 Gbps Ethernet capability.

The increase in IDD data volume over the past six months is largely attributable to the addition of 0.25 degree GFS data to CONDUIT, the overall increase in the volume of data being transmitted in NOAAPort (which now routinely exceeds 10 GB/hr), and the increase in dual polarization NEXRAD data.  During the end of August/beginning of September GOES-R test period, the NOTHER datastream (which contains simulated GOES-R products among other things) pushed the total volume of data being sent over NOAAPort to peaks in excess of 20 GB/hr.

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Enable widespread, efficient access to geoscience data
   A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments.
   
2. Develop and provide open-source tools for effective use of geoscience data
   The IDD is powered by the Unidata LDM-6 which is made freely available to all. The Unidata NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages are being bundled by Raytheon in AWIPS-II.

3. Provide cyberinfrastructure leadership in data discovery, access, and use
   The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers.
   
4. Build, support, and advocate for the diverse geoscience community
   Providing access to data in real-time is a fundamental Unidata activity.

The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa. The Universidad de Costa Rica is experimenting with relaying data received in the IDD to Colombia.

Prepared  March, 2016