Status Report: Internet Data Distribution

October 2021 - May 2022

Steve Emmerson, Mustapha IIes, Mike Schmidt, Jeff Weber, Tom Yoksas

Questions for Committee Members

• Do you have suggestions regarding content of data streams like CONDUIT, FNEXRAD, NIMAGE, UNIWISC, NLDN Lightning, SATELLITE, etc.?
  
  We (UPC, the Unidata community and UAlbany for the NLDN component of LIGHTNING) have control of the content of these data streams, so their contents  are open for suggestions.

NB: We now have access to Himiwari imagery and Level 2 products on AWS S3, so we could add some of these products to the IDD (probably in the SATELLITE feed), IF the committee thinks that this would be useful AND manageable by sites.  Also, we are working with the UCP JCSDA group to get access to METEOSAT imagery from Eumetsat.  If this access can be established, select METEOSAT imagery could be added to the IDD feed (probably in the SATELLITE feed).

Activities Since the Last Status Report

Internet Data Distribution (IDD)

IDD data volumes have not increased since the last meeting.

The following output is from a Linux-based data server  that the UPC operates on behalf of the community, lead.unidata.ucar.edu:

``bqb

20220523

Data Volume Summary for lead.unidata.ucar.edu

Maximum hourly volume 123306.025 M bytes/hour

Average hourly volume  77538.177 M bytes/hour

Average products per hour     507236 prods/hour

Feed                           Average             Maximum     Products

                     (M byte/hour)            (M byte/hour)   number/hour

CONDUIT               16120.791    [ 20.791%]    51477.359    93879.316

SATELLITE             15073.617    [ 19.440%]    19881.729     6688.842

NIMAGE                 9851.451    [ 12.705%]    13670.083     7662.316

NEXRAD2                9641.304    [ 12.434%]    13249.611   110831.553

NGRID                  7414.763    [  9.563%]    11523.417    67577.316

HDS                    5179.291    [  6.680%]    12504.108    30438.895

FNEXRAD                4750.455    [  6.127%]     5226.165     9820.053

EXP                    4259.352    [  5.493%]     6533.706    27448.526

NEXRAD3                3542.625    [  4.569%]     4919.910    94583.658

UNIWISC                 958.698    [  1.236%]     1138.804      919.947

FSL2                    322.458    [  0.416%]      824.791     1560.816

NOTHER                  280.998    [  0.362%]      884.447       46.421

IDS|DDPLUS               86.472    [  0.112%]       99.782    55131.605

SPARE                    50.786    [  0.065%]       60.202      237.000

LIGHTNING                 5.116    [  0.007%]       13.433      409.947

``bqe
Data Distribution:

IDD CONDUIT feed:

After the GFS model was upgraded from v15.3 to 16.0 in the spring of 2021,  we  upgraded our ingest machine to be able to handle the increased volume.  The same action was not taken by NCEP on the virtual machines that they use to create the CONDUIT datastream until the early spring this year.  Since the NCEP relay machines were updated, IDD latencies have dropped from unacceptable levels back to levels typical of the period before the GFS upgrade.

IDD FNEXRAD, NIMAGE and UNIWISC feeds:

We continue to create the content for the FNEXRAD (NEXRAD Level III national composites), NIMAGE (GOES-East and -West Level 2 images and products, fully reconstituted images from NOAAPort tiles and with broadcast headers and footers stripped off to leave “bare” netCDF4 files), and UNIWISC (select GOES-East and -West images converted to McIDAS AREA format for use in legacy systems like GEMPAK).  

Recently, the N0Q (so called “high res” base reflectivity) U.S. national composite was replaced with a N0B (so called “super res” base reflectivity) U.S. national composite.  This change was a result of the replacement of lower resolution NEXRAD Level III products, including N0Q, with higher resolution products in NOAAPort.  We submitted a request for an LDM feed of all NEXRAD Level III products from the RPCCDS to NCEP, and the request has been approved.  We were informed that it would take 2 to 3 months for the LDM feed to be setup on the NCEP side.  After the feed has been established, it is our intention to re-add products that have been removed from NOAAPort to the IDD.  Exactly which products will be re-added is to be determined.

IDD NIMAGE feed:

The NIMAGE feed, which was originally populated solely with GINI imagery distributed in NOAAPort, was enhanced by the addition of three products being created by CSU/CIRA:  GeoColor, DebraDust and CloudSnow.  GOES-East CONUS and -West PACUS coverages are available for the GeoColor and CloudSnow products while the DebraDust product is available in a GOES-East CONUS coverage.  All three of these are RGB products - displays are created using different wavelength channels to drive the Red, Green and Blue portions of a composite display.  The GeoColor product is quite useful (they are especially useful in identifying sources of smoke from wildfires), so users are strongly encouraged to take a look!

Support for displaying these RGB images has been added to Unidata AWIPS,  Unidata and SSEC McIDAS-X, the IDV and McIDAS-V.

Experimental HRRR feed to eventually be replaced by RRRS:

Unidata used to receive experimental High Resolution Rapid Refresh (HRRR) grids (both 2D and 3D fields) in an LDM/IDD  feed from NOAA/GSL and feed these products to a small number of university sites on hrrr.unidata.ucar.edu (which is also known as lead.unidata.ucar.edu).   Once the HRRR data went operational, NOAA/GSL stopped creating experimental HRRR output.  The experimental HRRR is, however, being replaced by the RRFS (Rapid Refresh Forecast System) in NOAA/GSL.  We have requested a feed of these data, but we have been told that the RRFS is still a few months away from being available.

Existing Data Distribution:

The primary top level IDD relay cluster, idd.unidata.ucar.edu, has been operating well since its move to the NCAR Wyoming SuperComputer (NWSC) facility in Cheyenne, WY.

The data volume seen in the SATELLITE (which is known as DIFAX in LDM distributions prior to v6.13.6) listing above represents all products received in the GOES ReBroadcast (GRB) downlinks that we installed in UCAR (currently  GOES-17 at the NCAR Mesa Lab  and GOES-16 at UCAR Foothills Lab 2).  The data volume seen in the NIMAGE entry represents GOES-East/West ABI Level 2 imagery that has been reconstituted by stitching together tiles that are distributed in NOAAPort and all other Level 2 products.  In both cases, binary headers and footers that are added to products before distribution in NOAAPort have been stripped off leaving “raw” netCDF4 files.  The UNIWISC feed represents the volume of 3 select channels (0.64um VIS, 6.2um WV and 10.3um IR) for all coverages (CONUS, FullDisk, Mesoscale-1 and Mesoscale-2)  of GOES-East/West image products that are in PNG compressed McIDAS AREA format that is suitable for use in GEMPAK, the IDV and McIDAS-V, McIDAS-X, and AWIPS.

Challenges, problems, and risks:

More sites, including UCAR, are installing intrusion detection/prevention systems (e.g., Palo Alto), which can adversely affect LDM throughput if not configured correctly.

Ongoing Activities

We plan to continue the following activities:

• Unidata has long distributed GPS radio occultation solutions from COSMIC.  The LDM feed from COSMIC to Unidata was interrupted by UCAR security perimeter changes (as was/is the SUOMINET products).  

• Many, but not all, of the products in NCEP operational HRRR are being distributed in the NOAAPort SBN and relayed in the IDD NGRID feed.  Fire weather products (HRRR Smoke) that are being made available by NOAA/GSL in an EXP feed were added to the set of HRRR products that are available from hrrr.unidata.ucar.edu.  All of these products along with with other model output are available via the TDS and  Unidata AWIPS EDEX: 

• Other data sets we continue to explore with NOAA/GSD/ESRL are:

• FIM
• HIWPP
• RRFS

• NCEP (operational) HRRR fields and forecasts times were added to the IDD CONDUIT datastream.

NOAAPort Data Ingest

• Ingest of the DVBS-2 NOAAPort Satellite Broadcast Network (SBN) products and their relay to end-users via the IDD has been “operational” at the UPC since August 2014.
  
  Unidata continues to assist LSU/Climate (formerly LSU/SRCC) with  the maintenance of  their NOAAPort ingest capability.   Activities have included providing a spare LNB to bring their NOVRA S300N receiver to Boulder for testing, configuration, power supply replacement, routine monitoring of their data and distribution, and consultation on satellite dish maintenance and pointing.

Considerable effort has been expended in streamlining our NOAAPort ingest systems and assisting sites (UW/SSEC, NOAA/GSL, NOAA/SPC, Fox13 TV) in troubleshooting problems being experienced in their systems.  More on the most recent of these activities can be found in the LDM status report.

• The NOAAPort-derived data streams (HDS, IDS|DDPLUS, NGRID, NIMAGE, NEXRAD3 and NOTHER) are redundantly injected into  the IDD at five geographically separate locations: UCAR/Unidata, UW/SSEC, LSU/Climate, Allisonhouse.com and Fox13 TV in Tampa, FL.  Even though GOES-15 will periodically be taken out of storage for use in western Pacific hurricane monitoring, the GINI image products that are created from its scans are only available during the short periods when the satellite is taken out of storage.
  
• Unidata's NOAAPort ingest package is bundled with current versions of the LDM.  The current LDM release is v6.13.16.   A new LDM distribution is being tested on Unidata ingest machines, and should be ready for release in early summer.

Relevant Metrics

• Approximately 556 machines at 192 sites are running LDM-6 and reporting real-time statistics to the UPC.
  
  We routinely observe that the number of sites reporting real-time statistics fluctuates.  We are not certain why this may be the case, but our best guess is that some sites do not keep their LDMs running all of the time; campus firewall adjustments block the sending of the statistics; and/or sites decide to stop sending statistics.  The latter possibility seems to be happening more frequently.
  
  We know that there are a number of sites that are participating in the IDD, but are not reporting real-time statistics back to us.  Reporting of real-time statistics is not and never has been mandatory.
  
  Unidata staff routinely assist in the installation and tuning of LDM-6 at user sites as a community service.  We have learned about sites not sending real-time statistics during these kinds of support activities, and a number of times the impediment to sending in stats is firewall configurations at the user sites.

• A number of organizations/projects continue to use the LDM to move substantial amounts of data that do not report statistics  to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private companies, etc.).

• UCAR IDD toplevel relays, idd.unidata.ucar.edu and iddb.unidata.ucar.edu
  
  The IDD relay clusters, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more  than 1205 downstream connections.  The primary IDD relay cluster, idd.unidata.ucar.edu, was moved to the NCAR/Wyoming Super Computing facility in Cheyenne, WY in late August 2019.
  

Over the period from April 12 through May 21, 2022 (IDD volume snapshots are taken during periods that do not have monitoring dropouts in NetVizura plots)  the average volume of LDM/IDD data flowing from the UCAR/NCAR network averaged around 7.2 Gbps (~78.8 TB/day), and peak rates reached 14.6 Gbps (which would be ~158 TB/day if the rate was sustained (which it is definitely not)).

The following table of volume snapshots shows that the volume of data flowing to downstreams out of UCAR has been reasonably consistent:

NB: The units for Src and Total Ave and Max are Gbps (gigabits per second), and the units for Dst are Mbps (megabits per second).

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Managing Geoscience Data
   The IDD project demonstrates how sites can employ the LDM to move and process data in their own environments.  

2. Providing Useful Tools
   The freely available LDM software and the IDD project that is built on top of the LDM have served as a demonstration for distribution of real-time data for a variety of organizations including the U.S. National Weather service.
   
   The cluster approach for LDM/IDD data relay that Unidata  pioneered has been been adopted by several Unidata university sites, and is currently being implemented at U.S. government sites.
   
   Unidata’s NOAAPort ingest package, which is bundled with LDM-6, is being used by a variety of university, U.S. government, and private sector entities.
   
   Both the LDM and NOAAPort ingest packages are bundled with AWIPS.
   
3. Supporting People
   The IDD is the primary method that core Unidata sites use to get the meteorological data that they need.  Providing access to data in near real-time is a fundamental Unidata activity.  The IDD-Brasil, the South American peer of the North American IDD, and IDD-Caribe, the Central American peer of the North American IDD, are helping to extend real-time data delivery throughout the Americas
   

Prepared  May, 2022