Status Report: McIDAS
June 2022 - October 2022
Tom Yoksas
Areas for Committee Feedback
We are requesting your feedback on the following topics:
- Are there any features that users would like to be added to Unidata McIDAS-X and/or ldm-mcidas?
Activities Since the Last Status Report
Aside from routine updates/bugfixes to existing code and tables, the main area of activity recently has been the release of Unidata McIDAS v2021. The main development area that we have been involved in is enhancing the HimawariCast ADDE serving to handle the Himawari HRIT images that are being distributed in EUMETCAST, images that are different from the ones being distributed in HimawariCast by the JMA.
Current Activities
- Support use of McIDAS-X/XCD in-house and in the community
- Ensure that the Unidata instances of McIDAS ADDE continue to function efficiently (ADDE serves on the order of 1.7 TB/day from three servers that Unidata operates on behalf of the community)
- McIDAS-X is used to convert GOES-East/West ABI imagery that is in netCDF4 format to McIDAS AREA format that is usable by all supported display and analysis packages (except Python/MetPy) for the Unidata-Wisconsin (UNIWISC aka MCIDAS) IDD feed.
The Unidata v2021 release includes these SSEC v2021.1 release features:
- Added support for McIDAS-X on macOS 11 systems.
- Updated GOES-R Series ABI and GLM servers with additional bug fixes and enhancements, including preparations for future GOES-18 and GOES-19 satellites, adding support of database search capabilities for improved performance in the SDS ABI archive, and access to ABI L2 Cloud and Moisture Imagery (CMIP) data.
- Updated the VIIRS and MSG servers and calibration modules to allow IMGPROBE to return VIIR REF and NREF values and MSGS/MSGT RAD values with more precision.
- Updated the ADDE servers to honor both the client’s and the server’s ADDETIMEOUT environment variable. ADDETIMEOUT now overrides the 600 second timeout of the default ADDE server.
- The GVAR servers were updated for EWS-G1 satellite data. GOES-13 became EWS-G1 on 8 September 2020. McIDAS-X checks for that date to appropriately set the SS number correctly for each satellite.
- The GRIB servers and GRD* commands were updated to list and display GRIB data with decimal pressure levels between 0 and 1 hPa, levels in potential vorticity units (PVU), and soil temperature levels in units of meters below ground (MBG).
- The mcinet.sh script was updated to allow the McIDAS-X system service to be controlled by systemd or xinetd.
- Improvements were made to the IMGPROBE command. The new NORM option enhances the displayed image with a histogram normalization by taking the min/max of the specified unit in the defined region and having the displayed BRIT values stretched from 0 to 255. IMGPROBE was updated so the BOX region has a limit of 30,000,000 points, and the BOX CONT (contouring) option has a limit of 10,000,000.
- The GAMMA= keyword was added to the IMGOPER command to apply gamma values to output AREA files, and RGB.MCB was added to -XRD, along with many RGB recipes for current real-time satellites. The RGB recipes use IMGOPER with GAMMA= to calculate the red, green, and blue channels, and then RGBDISP to display the products.
- Changes were made to the McIDAS-XRD Python Package to better handle single quotes in McIDAS-X commands. Each user must redo the mcxpyinstall installation step to utilize this new feature.
Ongoing Activities
We plan to continue the following activities:
- SSEC McIDAS Advisory Committee (MAC)
The UPC (Yoksas, Ho) continues to participate as the Unidata representative to the McIDAS Advisory Committee (MAC) that is operated by SSEC, but meetings of this group have been rare for quite some time.
The MAC was assembled by UW/SSEC to advise SSEC on McIDAS-X users’ needs/concerns/desires for development in the next generation McIDAS, McIDAS-V. The MAC was modeled after the Unidata IDV Steering Committee.
- Interest in McIDAS by non-core users
The UPC occasionally receives requests for McIDAS-X and help using McIDAS-X from international university users, U.S. government agencies and other non-traditional Unidata users (e.g., private businesses, etc.). Government agencies and non-traditional Unidata users are referred to UW/SSEC for access to McIDAS; international educational community user requests are granted on a case-by-case basis after they provide a clear statement of their acceptance of the terms of use provided by SSEC.
- Continued support of existing and new community members
New Activities
- Add support for new types of data when they become available, otherwise McIDAS-X support is in maintenance mode.
Relevant Metrics
- Data delivered by the Unidata McIDAS ADDE servers exceeds 1.7 TB/day. The great majority of the data being served is imagery from GOES-16/17/18.
ldm-mcidas Decoders Activities
Development
ldm-mcidas releases are made when needed to support changes in software development and operating system environments. ldm-mcidas v2012 was released at the end of September, 2012. Recently, the ldm-mcidas code was moved to GitHub.
Geostationary Satellite Data Ingest and Data Serving
Unidata continues to ingest GOES-East and GOES-West imager data at the UCAR Foothills Lab and NCAR Mesa lab campuses in Boulder.
- Direct, programmatic access to real-time GOES-East (GOES-16) and GOES-West (currently GOES-17 but soon to be GOES-18) data via McIDAS ADDE services on three publicly accessible servers (lead.unidata.ucar.edu, atm.ucar.edu and adde.ssec.wisc.edu) has been averaging on the order of 1.7 TB/day for the past two years.
Planned Activities
Ongoing Activities
Continued ingest, distribution via the IDD and ADDE serving of GOES-East and GOES-West imagery from the GRB downlinks we installed in UCAR
Continued ingest and ADDE serving of GOES-15 and GOES-14 imagery when available. GOES-15 and GOES-14 were put into standby mode on March 2, 2020. GOES-14 remains in its standby location (104W) and is turned on for periodic testing as needed. GOES-15 supplemental operations began on Sunday, August 9, 2020 and continues to provide surveillance during Pacific hurricane seasons. It is most likely that GOES-15 operations will end when GOES-18 assumes the role of GOES-West from GOES-17. It is our understanding that GOES-17 will be drifted towards the GOES-14 standby orbit and function as an in-orbit standby for GOES-16 and GOES-18.
These efforts require maintenance of the satellite ingest and data serving equipment.
New Activities
Establish a testbed for generating Level 2 products from GOES-East/West imagery and select model output. The intention is to be able to test vetted algorithms submitted by community members for a long enough period for the algorithms to be fully tested.
Strategic Focus Areas
We support the following goals described in Unidata Strategic Plan:
- Managing Geoscience Data
Remote, programmatic access to data provided by the Abstract Data Distribution Environment (ADDE) environment of McIDAS has been a model for the development of remote access methodologies since 1994. Concepts articulated in ADDE inspired the development of THREDDS (to address the lack of rich metadata available in ADDE) and RAMADDA. ADDE remains one of the most used data services in the Unidata suite. ADDE servers operated by Unidata are currently serving in excess of 1.6 TB/day.
- Providing Useful Tools
McIDAS remains the application of choice for the satellite meteorology community. The Abstract Data Distribution Environment (ADDE) component of McIDAS was the first application offered by Unidata to provide remote, programmatic access to a wide variety of data that is important to the atmospheric science community.
The fifth generation of McIDAS, McIDAS--V, unlike its predecessors, is a fully open source application that is in wide scale and growing use in the worldwide satellite meteorological community
McIDAS ADDE continues to evolve and provide access to a rapidly increasing volume of imagery and non-image data.
- Supporting People
McIDAS is still in active use by those interested in satellite meteorology worldwide.
Prepared October, 2022