Status Report: LDM

October 2020 - April 2021

Steve Emmerson, Tom Yoksas, Mike Schmidt, Mustapha Iles, Yuanlong Tan (University of Virginia)

Activities Since the Last Status Report

LDM

The LDM is the primary software package by which research and education institutions obtain near real-time meteorological and related data.

Progress has been made on the following:

• NOAAPortIngester:

• GEMPAK tables:

• Added many new WMO parameters
• Added many NWS MRMS parameters

• Added option to set the receive buffer size to accommodate a request from the NWS. The operating system must allow the new size. This had the added benefit of reducing the number of missed NOAAPort packets at the Unidata Program Center.

• pqact(1):

• Squashed bug that caused abnormal termination under extremely rare conditions
• Improved documentation and log messaging

• ldmadmin(1):

• Made handling of netstat(1) output more general
• Improved help for the “pqactcheck” command

• scour(1):

• Replaced scour(1) script with multi-threaded C program  for vastly improved performance

• LDM registry:

• Added parameter for scour(1) configuration-file
• Added SCOUR_EXCLUDE_PATH for pathname of file that lists directories to be excluded by new scour(1) program

• pqcreate(1):

• Increased the mean number of bytes in a data-product from 51 k to 140 k for computation of the number of slots when the registry parameter is “default”

• Logging:

• Added FATAL logging level for serious problems

• Miscellaneous:

• Removed lint identified by clang(1) and Coverity Scan

• Support:

• Answered many questions from Universities, NOAA, US Military, and corporations
• Troubleshot several sites that were having problems that were, overwhelmingly, network-related

Dependencies, challenges, problems, and risks include:

The LDM is sometimes held responsible for decisions made by the NWS when they don’t follow their own policy on how to categorize and name data products (not a new challenge).

More sites are installing intrusion detection/prevention systems (e.g., Palo Alto), which can adversely affect LDM throughput if not configured correctly.

Mathew Lazarra (University of Wisconsin–Madison) is spearheading an effort to formally integrate the LDM into the Antarctic meteorological data-distribution network.

Multicast LDM (aka LDM-7)

The multicast LDM project is separately funded by CISE in NSF. The goal is to reduce the outgoing bandwidth requirement of the LDM -- yet retain the current level of reliability -- by converting it into a hybrid system that combines use of the new, semi-reliable multicast protocol developed jointly with the University of Virginia with the time-tested unicast capability of the current LDM.

This project ended April 1st, 2019, but a no-cost extension was approved because funding was delayed.

Progress has been made on the following:

• Improved handling of the OpenSSL library. (Because any site can send to an IP multicast group, security is a concern and was a point in the proposal.)
• Corrected bug in the logic for tracking of the last multicast product and made its handling more robust
• Created three levels for the message authentication code (MAC) for multicast packets depending on the value of environment variable FMTP_MAC_LEVEL:

• Unset or 0: No MAC
• 1:          SHA256 Hash-based MAC
• 2:          Ed25519 Digital Signing Algorithm

• Created a “BlackHat” module for spoofing valid packets to verify the message authentication mechanism
• Conducted numerous test runs to explore the parameter space (i.e., performance as a function of 3 security-levels, 5 packet sizes, and 5 VLAN bandwidths). Each run takes an hour.
• Improved log messaging
• Improved documentation

Dependencies, challenges, problems, and risks include:

• The amount of manual intervention required to maintain the multipoint VLAN is considerable

Ongoing Activities

We plan to continue the following activities:

• Support and maintain  the LDM

New Activities

Over the next twelve months, we plan to organize or take part in the following:

• Work with NSF’s Polar Programs regarding integration of the LDM into the Antarctic meteorological data-distribution network

Relevant Metrics

• Data on LDM downloads
• The LDM system at the Unidata Program Center powers the Unidata IDD (Internet Data Distribution) system. Metrics on that program can be found in the IDD status report.

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Managing Geoscience Data
   By enabling researchers, teachers, and students to process a wide variety of meteorological and related data in near real time.
   
2. Providing Useful Tools
   By enabling researchers, teachers, and students to obtain a wide variety of meteorological and related data in near real time and at no cost via the Internet.
   
   By using the LDM to move data into the cloud and developing multicast technologies.
   
3. Supporting People
   By answering support questions, writing documentation, and conducting workshops.
   

Prepared  April 2021