Status Report: McIDAS

October 2021- May2022

Tom Yoksas

Areas for Committee Feedback

We are requesting your feedback on the following topics:

1. Are there any features that users would like to be added to Unidata McIDAS-X and/or ldm-mcidas?

Activities Since the Last Status Report

Aside from routine updates/bugfixes to existing code and tables, the main area of activity recently has been the incorporation of Unidata-developed code into the next UW/SSEC McIDAS-X/XCD release, v2021.1.   The main development area that we have been involved in is enhancing ADDE serving to better handle the various forms of ABI satellite imagery, satellite  Level 2 products, and NEXRAD Level 3 products.

Current Activities

• Support use of McIDAS-X/XCD in-house and in the community

• Ensure that the Unidata instances of McIDAS ADDE continue to  function efficiently (ADDE serves on the order of 1.7 TB/day from three servers that Unidata operates on behalf of the community)

• McIDAS-X is used to convert GOES-East/West ABI imagery that is in netCDF4 format to McIDAS AREA format that is usable by all supported display and analysis packages (except Python/MetPy) for the Unidata-Wisconsin (UNIWISC aka MCIDAS) IDD feed.  

The SSEC v2021.1 release features the following:

• Added support for McIDAS-X on macOS 11 systems.
• Updated GOES-R Series ABI and GLM servers with additional bug fixes and enhancements, including preparations for future GOES-18 and GOES-19 satellites, adding support of database search capabilities for improved performance in the SDS ABI archive, and access to ABI L2 Cloud and Moisture Imagery (CMIP) data.
• Updated the VIIRS and MSG servers and calibration modules to allow IMGPROBE to return VIIR REF and NREF values and MSGS/MSGT RAD values with more precision.
• Updated the ADDE servers to honor both the client’s and the server’s ADDETIMEOUT environment variable. ADDETIMEOUT now overrides the 600 second timeout of the default ADDE server.
• The GVAR servers were updated for EWS-G1 satellite data. GOES-13 became EWS-G1 on 8 September 2020. McIDAS-X checks for that date to appropriately set the SS number correctly for each satellite.
• The GRIB servers and GRD* commands were updated to list and display GRIB data with decimal pressure levels between 0 and 1 hPa, levels in potential vorticity units (PVU), and soil temperature levels in units of meters below ground (MBG).
• The mcinet.sh script was updated to allow the McIDAS-X system service to be controlled by systemd or xinetd.
• Improvements were made to the IMGPROBE command. The new NORM option enhances the displayed image with a histogram normalization by taking the min/max of the specified unit in the defined region and having the displayed BRIT values stretched from 0 to 255. IMGPROBE was updated so the BOX region has a limit of 30,000,000 points, and the BOX CONT (contouring) option has a limit of 10,000,000.
• The GAMMA= keyword was added to the IMGOPER command to apply gamma values to output AREA files, and RGB.MCB was added to -XRD, along with many RGB recipes for current real-time satellites. The RGB recipes use IMGOPER with GAMMA= to calculate the red, green, and blue channels, and then RGBDISP to display the products.
• Changes were made to the McIDAS-XRD Python Package to better handle single quotes in McIDAS-X commands.  Each user must redo the mcxpyinstall installation step to utilize this new feature.

Ongoing Activities

We plan to continue the following activities:

• SSEC McIDAS Advisory Committee (MAC)
  
  The UPC (Yoksas, Ho) continues to participate as the Unidata representative to the McIDAS Advisory Committee (MAC) that is operated by SSEC.
  
  The MAC was assembled by UW/SSEC to advise SSEC on McIDAS-X users’ needs/concerns/desires for development in the next generation McIDAS, McIDAS-V. The MAC was modeled after the Unidata IDV Steering Committee.

• Interest in McIDAS by non-core users
  
  The UPC occasionally  receives requests for McIDAS-X and help using McIDAS-X from international university users, U.S. government agencies and other non-traditional Unidata users (e.g., private businesses, etc.). Government agencies and non-traditional Unidata users are referred to UW/SSEC for access to McIDAS; international educational community user requests are granted on a case-by-case basis after they provide a clear statement of their acceptance of the terms of use provided by SSEC.

• Continued support of existing and new community members

New Activities

• Add support for new types of data when they become available, otherwise McIDAS-X support is in maintenance mode.

Relevant Metrics

• Data delivered by the Unidata McIDAS ADDE servers exceeds 1 TB/day.  The great majority of the data being served is imagery from GOES-16 followed by imagery from GOES-17.
  

• McIDAS-X/-XCD Inquiry Metrics

ldm-mcidas Decoders Activities

Development

ldm-mcidas releases are made when needed to support changes in software development and operating system environments. ldm-mcidas v2012 was released at the end of September, 2012.  Recently, the ldm-mcidas code was moved to GitHub.

Geostationary Satellite Data Ingest and Data Serving

Unidata continues to ingest GOES-East and GOES-West imager data at the UCAR Foothills Lab and NCAR Mesa lab campuses in Boulder.

• Direct, programmatic access to real-time GOES-East (GOES-16) and GOES-West (currently GOES-17 but soon to be GOES-18) data via McIDAS ADDE services on three publicly accessible servers (lead.unidata.ucar.edu, atm.ucar.edu and adde.ssec.wisc.edu) has been averaging on the order of 1.7TB/day for the past two years.

Planned Activities

Ongoing Activities

Continued ingest, distribution via the IDD and ADDE serving of GOES-East  and GOES-West imagery from the GRB downlinks we installed in UCAR

Continued ingest and ADDE serving of GOES-15 and GOES-14 imagery when available.  GOES-15 and GOES-14 were put into standby mode on March 2, 2020.  GOES-14 remains in its standby location (104W) and will be turned on for periodic testing as needed. GOES-15 supplemental operations began on Sunday, August 9, 2020 at 0000 UTC and continues to provide surveillance during Pacific hurricane seasons.

These efforts require maintenance of the satellite ingest and data serving equipment.

New Activities

Establish a testbed for generating Level 2 products from GOES-East/West imagery and select model output.  The intention is to be able to test vetted algorithms submitted by community members for a long enough period for the algorithms to be fully tested.

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Managing Geoscience Data
   Remote, programmatic access to data provided by the Abstract Data Distribution Environment (ADDE) environment of McIDAS has been a model for the development of remote access methodologies since 1994.  Concepts articulated in ADDE inspired the development of THREDDS (to address the lack of rich metadata available in ADDE) and RAMADDA.  ADDE remains one of the most used data services in the Unidata suite.  ADDE servers operated by Unidata are currently serving in excess of 1.6 TB/day.
   
2. Providing Useful Tools
   McIDAS remains the application of choice for the satellite meteorology community.  The Abstract Data Distribution Environment (ADDE) component of McIDAS was the first application offered by Unidata to provide remote, programmatic access to a wide variety of data that is important to the atmospheric science community.
   
   The fifth generation of McIDAS, McIDAS--V, unlike its predecessors, is a fully open source application that is in wide scale and growing use in the worldwide satellite meteorological community
   
   McIDAS ADDE continues to evolve and provide access to a rapidly increasing volume of imagery and non-image data.

3. Supporting People
   McIDAS is still in active use by those interested in satellite meteorology worldwide.
   

Prepared May, 2022