Status Report: LDM

April 2020 - September 2020

Steve Emmerson, Tom Yoksas, Mike Schmidt, Yuanlong Tan (University of Virginia)

Activities Since the Last Status Report

LDM

The LDM is the primary software package by which research and education institutions obtain near real-time meteorological and related data.

Progress has been made on the following:

• Product-Queue:

• Corrected bug in function’s return code

• pqact(1):

• Corrected truncation of output file by the FILE action when both "-overwrite" and "-metadata" are specified
• Demoted "Deleting ... entry" to DEBUG if it's not due to an error in order to save log file space

• Logging:

• Documented internally how log_refresh() will cause the log file to be closed/opened on the next log message and not before

• ldmd(1):

• Top-Level Server:

• Eliminated superfluous error messages at process termination about the upstream LDM database not existing if the "-n" (do nothing) option is specified.

• Upstream LDM:

• Replaced use of select() with poll()for better error-control
• Increased service timeout from 60 s to 120 s

• noaaportIngester(1):

• Corrected discussion in the documentation of UDP packet reassembly and the setting of system parameter net.ipv4.ipfrag_max_dist to zero
• Added Vertical Velocity (Geometric)  to the NCEP GEMPAK table
• Changed logging-level of messages for first fragment of GOES imagery from NOTE to INFO

• Misc:

• Improved logic in ghostname() for determining the fully-qualified name of the local host, which is used by rtstats(1)
• Removed lint found by Coverity Scan

• Support:

• Answered many questions from Universities, NOAA, US Military, and corporations
• Troubleshot several sites that were having problems that were, overwhelmingly, network-related

Dependencies, challenges, problems, and risks include:

The LDM is sometimes held responsible for decisions made by the NWS on how to categorize and name data products (not a new challenge).

More sites are installing intrusion detection/prevention systems (e.g., Palo Alto), which can adversely affect LDM throughput if not configured correctly.

Mathew Lazarra (University of Wisconsin–Madison) is spearheading an effort to formally integrate the LDM into the Antarctic meteorological data-distribution network.

Multicast LDM (aka LDM-7)

The multicast LDM project is separately funded by CISE in NSF. The goal is to reduce the outgoing bandwidth requirement of the LDM -- yet retain the current level of reliability -- by converting it into a hybrid system that combines use of the new, semi-reliable multicast protocol developed jointly with the University of Virginia with the time-tested unicast capability of the current LDM.

This project ended April 1st, 2019, but a no-cost extension was approved because funding was delayed.

Progress has been made on the following:

• Fixed bugs, particularly in the mapping from multicast product-index to LDM product-signature and in the backstop mechanism for products missed by the multicast module
• More sites added to the test deployment
• The switch handling LDM7 traffic from UCAR was replaced. This switch was mistakenly limiting the outgoing rate of multicast packets to 47 Mbps for an unknown reason.

Dependencies, challenges, problems, and risks include:

• The amount of manual intervention required to maintain the multipoint VLAN.

Ongoing Activities

We plan to continue the following activities:

• Support and maintain  the LDM

New Activities

Over the next twelve months, we plan to organize or take part in the following:

• Add message-authentication to the data-packets multicast by LDM7 (Unidata time to be paid by a separate grant from NSF)
• Work with NSF’s Polar Programs regarding integration of the LDM into the Antarctic meteorological data-distribution network

Relevant Metrics

• Data on LDM downloads
• The LDM system at the Unidata Program Center powers the Unidata IDD (Internet Data Distribution) system. Metrics on that program can be found in the IDD status report.

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Managing Geoscience Data
   By enabling researchers, teachers, and students to process a wide variety of meteorological and related data in near real time.
   
2. Providing Useful Tools
   By enabling researchers, teachers, and students to obtain a wide variety of meteorological and related data in near real time and at no cost via the Internet.
   
   By using the LDM to move data into the cloud and developing multicast technologies.
   
3. Supporting People
   By answering support questions, writing documentation, and conducting workshops.
   

Prepared  August 2020