Status Report: AWIPS II and GEMPAK

April - September 2015

Michael James

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Enable widespread, efficient access to geoscience data
   Both AWIPS II and GEMPAK are freely available, and both incorporate LDM/IDD technology for efficiently accessing geoscience data.
   
2. Develop and provide open-source tools for effective use of geoscience data
   Both AWIPS II and GEMPAK are open-source, and while GEMPAK is now in maintenance mode (not actively developed), AWIPS II is continuously being developed to support new data types and facilitate easier use.
   
3. Provide cyberinfrastructure leadership in data discovery, access, and use
   Unidata is the only known entity to provide a freely-available and non-operational version of the AWIPS II software package.
   
4. Build, support, and advocate for the diverse geoscience community
   Using LDM/IDD technology to provide access to real-time meteorological data; providing visualization tools for data analysis.

Activities Since the Last Status Report

AWIPS II

Unidata AWIPS II 14.4.1 was released at the beginning of September 2015, and includes significant changes to the system compared with the operational build, the focus being on minimizing the resource footprint and expanding the supported data types.

Some of the improvements and features in 14.4.1 include:

• Server-side compression to reduce data transfer sizes by an order of magnitude.
• Client-side caching of maps and data resources to reduce querying and loading time.
• Postgres, Pypies, Qpid and EDEX now start automatically on machine reboot.
• Python Data Access Framework allows users to write Python scripts to query remote EDEX maps and data.
• D2D single-pane default perspective, toggleable to 5-pane "classic" view.
• NCP resources can now be loaded directly from the D2D perspective, making available GEMPAK-like grid display without having to switch to the NCP perspective.
• Multi-Radar/Multi-Sensor (MRMS) full support for decoding and displaying gzipped grib2 files from NSSL.
• Support for decoding and displaying USPLN lightning data.
• NEXRAD3 menu includes full set of NEXRAD and TDWR sites, organized by site.
• NEXRAD3 mosaic capability renders a national composite of certain level 3 products queried from individual station files (functional but slow).
• Save/Load Bundle feature added for D2D, allowing users the option of saving bundles (as xml files) locally to disk or synced in ~/caveData to the EDEX server.
• McIDAS AREA files from the UNIWISC IDD feed decoded and displayed by the satellite plugins.
• NO CLIPPING OF GRIDS TO LOCALIZATION AREA
• Java 1.6 -> 1.7
• Fixed problem with how LDM writes CMC-REG grib1 messages which caused EDEX ingestGrib JVM to crash in 14.2.
• Full south polar stereographic UNIWISC McIDAS area file support.
• Partial GVAR native projection support.
• Various D2D UI improvement: max number of frames increased to 400, "Map Scale" menu now includes satellite-specific map projections (Arctic, Antarctica, Alaska-Hawaii-PR Regional, East/West CONUS, Global, etc.), Hydro/NCEP and Upper Air menus are now visible.
• Ingest JVMs now run with maximum 4096M of memory.
• Qpid components updated from 0.30 -> 0.32
• Various fixes to EDEX-side hydro applications.

Ongoing Activities

• MRMS support requires the LDM use a modified version of the gempak “dcgunzip” decoder to decompress the grib2.gz files to disk, and then notify Qpid via a script “qpidNotify” (rather than edexBridge) that the product is available for ingest.  Strangely, the operational build of AWIPS II has incomplete grib tables for MRMS processing, so the Unidata release is ahead of NWS on this matter.  Still, the resource strain is too much for a standalone installation (with MRMS HDF5 files reaching 10 GB per hour, processing of NGRID and CONDUIT grids are slowed to the point of being unuseable), so a distributed EDEX environment (in the cloud, most likely) is needed to find a resource equilibrium.
• I have demonstrated the EDEX Python Data Access Framework (ufpy) to query and display gridded data (matplotlib) and upper air data (metpy), and work continues to provide more documentation and examples of how the framework can be used.  The goal is to include ufpy as a means of requesting and displaying data inside in Unidata Siphon project, as a parallel to thredds/netCDF access.
• Further support of McIDAS GVAR native projection in EDEX, both on the processing side (currently seeing persistence errors when decoding multiple files) and on the visualization side (progressive resolution tiling is problematic with high-resolution UNIWISC files).
• (Exploratory) How best can netCDF-Java handle EDEX data?  Is it possible to use existing (but currently hidden) EDEX technology to read from TDS?

GEMPAK

GEMPAK v7.2.0 was released on August 4, 2015, incorporating NCEP table and map updates, as well as the following fixes and improvements:

• GFS 0.25 degree support
• Correction to allow decoding of 24hr min temp
• TDWR product fixes for nmap2 and gp/radar programs
• GEMPAK download access has been changed from registration-required to freely-available, and I have begun to maintain release packages (zip and tar.gz) on github for distribution rather than hosting on our own web server.

Ongoing Activities

• Maintain releases in parallel with NCEP operational table updates and fixes
• Incorporate a GUI-less GEMPAK build with the AWIPS II CAVE client which can access data from remote EDEX server.

New Activities

Over the next three months, we plan to organize or take part in the following:

• Massively expand AWIPS II user documentation.

Over the next twelve months, we plan to organize or take part in the following:

• Python DAF incorporated into existing Unidata Python technologies
• Deploy a front-end web server (cherrypy is already installed with EDEX) for all EDEX machines to give users and administrators an interactive inventory of what data products are available.

Relevant Metrics

Web server statistics?

Data download statistics?

Something else useful?

Prepared  September 2015