Status Report: LDM

October 2015 - March 2016

Steve Emmerson, Tom Yoksas, Mike Schmidt

Activities Since the Last Status Report

LDM

The LDM is the primary software package by which research and education institutions obtain near real time meteorological and related data.

Progress has been made on the following:

• LDM logging was rewritten to eliminate dependence on the system logging daemon because:

• It was the major impediment to correctly using the LDM;
• It was the major impediment to putting the LDM in a Docker container;
• Only 7 "local" syslog(8) facilities are guaranteed and NOAAPort ingestion could use more; and
• Some system logging daemons would drop LDM log messages depending on how heavily they were loaded.

• Missing parameter entries were added to the GEMPAK tables for NOAAPort ingestion.
• The possibility of pqact(1)not processing the last successfully-received data-product due to a restart of the LDM was eliminated.
• Access to the product-queue was made thread-safe.
• Raytheon’s changes to NOAAPort ingestion to properly handle GRIB-2 messages that mix compressed and uncompressed frames were incorporated.
• Version 6.13.0 was released.

Dependencies, challenges, problems, and risks include:

• Dealing with missing GEMPAK table entries is a pain.
• Very low bus factor (if Emmerson gets hit by a bus, the LDM’s in trouble).

Multicast LDM (alias LDM-7)

The multicast LDM project is separately funded by CISE in NSF. The goal is to reduce the outgoing bandwidth requirement of the LDM -- yet retain the current level of reliability -- by converting it into a hybrid system that combines use of the new, semi-reliable multicast protocol developed at the University of Virginia with the time-tested unicast capability of the current LDM.

Progress has been made on the following:

• An oral presentation on LDM-7 was made at last year’s Fall AGU meeting.
• Testing of LDM-7 over static VPN circuits on Internet2 has begun. Participants include University of Virginia, Rutgers University, Indiana University, and University of Wisconsin.

Dependencies, challenges, problems, and risks include:

• Time.
• Very low bus factor.

Ongoing Activities

We plan to continue the following activities:

• Support and maintenance of the LDM
• Deployment and testing of LDM-7

New Activities

Over the next three months, we plan to organize or take part in the following:

• Use of the LDM to bring data into the Cloud as part of the NOAA Big Data Project.

Over the next twelve months, we plan to organize or take part in the following:

• Use of the LDM to bring data into the Cloud as part of the NOAA Big data Project.

Relevant Metrics

The LDM powers the IDD, whose statistics can be found here.

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Enable widespread, efficient access to geoscience data
   By enabling researchers, teachers, and students to obtain a wide variety of meteorological and related data in near real time and at no cost via the Internet.
   
2. Provide cyberinfrastructure leadership in data discovery, access, and use
   By using the LDM to move data into the cloud.
   

Prepared  March 2016