Status Report: AWIPS and GEMPAK

September 2018 - March 2019

Michael James

Activities Since the Last Status Report

AWIPS

Unidata’s Jetstream EDEX server continues to serve real-time AWIPS data to CAVE clients and through the python-awips data access framework (API). By offloading the processing of one very large data feed (NEXRAD3) to a separate EDEX Ingest Node, the current implementation of edex-cloud is now cabable of processing and serving more data than ever before. The distributed architectural concepts of AWIPS allow us to scale EDEX in the cloud to account for the size of incoming data feeds.

Software Releases

18.1.1-5 Feb 2019

AWIPS 18.1.1-5 is a major update to both CAVE and EDEX, including a security update to the latest httpd/mod_wsgi packages.

• EDEX localization levels have changed, and Unidata AWIPS no longer supports REGION, DESK, or WORKSTATION levels, instead using only Base, SITE, and USER.
• EDEX administrators should be aware that USER-level localization names have been merged with the WORKSTATION name, that is, what used to be USER awips will now be USER awips@<workstation name>.

• New GOES Geostationary perspective multi-view bundles loaded from the Satellite menu.

• New menu items for GOES 17 (GOES West) sectors WCONUS, WFD, WMESO-1, WMESO-2, AKREGI, HIREGI.
• WWA (Watch, Warn, Advisory) plugin display updates with phensig color table and full text on sampling. These new bundles are located in the Surface menu, available for flood, wind, marine, as well as All Warnings and Convective Watches.

• Fix for the Linux GFE Perspective to sync required files to run gfeConfig correctly.

• Removal of REGION, DESK, and WORKSTATION localization level. Any configuration or setting which used the DESK or WORKSTATION levels will now use SITE instead. This makes for only 3 localization levels to manage in Unidata AWIPS from 18.1.1-5 onward: 1) BASE (default), 2) SITE (e.g. OAX), and 3) USER.
• More efficient thread control for the ingestGrib JVM in /awips2/edex/etc/ingestGrib.sh rather than using definitions in /awips2/edex/conf/resources/.
• Updated the /etc/init.d/edex_ldm wrapper to allow for the use of ldmadmin clean as root.
• Fixed a bug with Python environments when running edex qpid as root.
• Updated pqact.conf to include GOES 17 products, as well as a fix to ingest both METOP-A and B ascat files.

18.1.1-4 (development release) Dec 2018

18.1.1-3 Nov 2018

• Added a new GEMPAK display plugin to the D2D perspective
• Cleaned up paths available to CAVE in the Localization Perspective
• Cleanup grid name aliasing to avoid multiple names for a single grid (NAM40 vs NAMWX)
• Fix to reset the manual ingest endpoint to /awips2/data_store/ingest/
• Cleanup/remove GFE service backup utility
• Updates to "edex start" service manager
• Minor updates and fixes merged from NWS development baseline (Hydro apps trace precipitation display, GFE Fosberg fire weather index, GFE Haines index, Grib1 decoding updates)

18.1.1-2 Sep 2018

• “edex status” will mute EDEXregistry if Data Delivery/OGC plugins are not installed. Data Delivery/OGC plugins are not being used currently, but work is ongoing to incorporate them into EDEX (more on this in the future).
• New awips2-python-scipy RPM for CentOS 7.
• FNMOC grid names corrected in awips2-edex-dataplugins.
• Updated awips2-ldm and awips2-edex to remove the edex setup command from post-RPM scripts, and from EDEX startup, to avoid overwriting user edits to the LDM registry.
• "edex setup" will be a required command after install and before starting EDEX services.

18.1.1-1 Sep 2018

• Windows CAVE

https://www.unidata.ucar.edu/downloads/awips2/awips-cave.msi

• Requires Python 3.7, Jep (3.8.2), and NumPy (>1.7).
• $PYTHONHOME must be defined for full functionality using gridslice and jep.
• The MSI installer will attempt to copy the gridslice shared library to $PYTHONHOME/Dlls. If the $PYTHONHOME environmental variable is not defined, gridslice will not be installed.
• You can always rerun the installer after defining $PYTHONHOME and then check that the file gridslice.pyd is installed in $PYTHONHOME/Dlls.
• Windows CAVE will run without gridslice, but certain bundles which use derived parameters, such as isentropic analyses, will not load.

• New D2D CAVE macOS client
  https://www.unidata.ucar.edu/downloads/awips2/awips-cave-18.1.1-1.dmg
• New CentOS 6 and 7 client
  wget https://www.unidata.ucar.edu/software/awips2/awips_install.sh
  chmod 755 awips_install.sh
  ./awips_install.sh --cave
• New CloudAWIPS release 18.1.1-1
  https://github.com/Unidata/CloudAWIPS/blob/master/README.md
• GOES-17 ABI sectors TCONUS and TFD
• Fix CAVE crash when starting GFE for non-activated site.
• Updates to allow dynamic contributions to the Volume Browser.
• Search By Name has been added to the Localization Perspective.

• Fix serialization errors which prevented IPv6 addresses from connecting.
• Derived Parameter functions synced to client CAVE sessions are now Python 2 and 3-compliant.
• Support for variable header length in PNG-compressed McIDAS AREA files (UNIWISC)
• Database reconfiguration (moved from /awips2/data to /awips2/database/data, now with 'awipsadmin' ownership).
• New roles/permissions framework to replace nwsauth from versions 17.1.1 and before.
• pqact.conf added Jason-3 altimeter data feed
• pqact.conf cleanup of removed products and duplicate pattern actions
• Updates to the Qpid Broker and Localization REST configuration including Protected File status.

• Refactored gridslice.c source code for a Python 3 environment.
• New viz method IGLMesh for easier extensibility
• Ability to handle derived parameter cubes at a single point
• ADE updates for Eclipse 4.6.1
• Geotools updated to 16.4
• Cython updated to 0.28.3
• httpd-pypies updated to 2.4.27

GEMPAK/NAWIPS

GEMPAK release 7.5.1 February 2019

https://github.com/Unidata/gempak/releases/tag/7.5.1

• Fixed build of program dctama
• Supports GOES-16/17 ABI imagery in NetCDF 4 format

GOES16 and GOES17 ABI products are distributed in NetCDF4 formatted files, and GEMPAK as of release 7.5.1 can now read and display these products in their native formats (it is no longer needed to convert netCDF to McIDAS area format to display in GEMPAK).

For Full Disk and CONUS sectors, GOES16 and GOES17 ABI products are sent across the Satellite Broadcast Network as tiles, each contributing to a "mosaic" of a full sector scan. The Unidata Program Center has provided an alternate data feed in which all tiles are "stitched" together (thanks to the program ldm-alchemy).

To request these stitched images, add the following request line to your ldmd.conf:

REQUEST NIMAGE "GOES" idd.unidata.ucar.edu

and the following to your GEMPAK pqact.conf:

NIMAGE     .*GOES/(.*)/([^/]*)/([^/]*)/([0-9]{8})/GOES(16|17)_([^/]*)_([0-9]{8})_([0-9]{4})(..)_([^/]*).nc4
       FILE    -close  data/gempak/images/sat/\1/\2/\3/\3_\7_\8

This will write to directories in $SAT which are selectable in NMAP2 > New Data Source > Image:

GEMPAK release 7.4.5 December 2018

https://github.com/Unidata/gempak/releases/tag/7.4.5

• GEMPAK Python classes refactored for Python 3-compliance and best practice coding convention.
• AWIPS grid naming updates.

GEMPAK release 7.4.3 September 2018

https://github.com/Unidata/gempak/releases/tag/7.4.3

• Full support for GOES ABI fixed grid projection McIDAS AREA format (UNIWISC).
• New AWIPS lutfils for GOES 16 IR and WV channels (https://www.unidata.ucar.edu/staff/mjames/gempak/goes16/).
• Corrected a bug in 7.4.2 snow squall warnings which caused dcwarn to generate false FFW warnings.
• Library updates for future netcdf4 support (zlib 1.2.6 -> 1.2.8, HDF5 1.8.0 -> 1.8.15, netCDF 4.1.3 -> 4.3.3.1).

Activities Ongoing/In-Progress

EDEX and WRF

Using the Jetstream cloud, I have created an EDEX ingest with self-ingested scheduled UEMS (WRF) runs.  

Unidata AWIPS release 19.1.1 (release TBD)

• Developing a set of plugins for the ingest and display of USGS river gauge RDB files as time-series pointdata.
• Adding world cities, administrative districts, roads, and other international shapefile imports.
• Menu updates for Watch, Warn, Hydro/QPF, Upper Air products.
• Support for the North American Real-time Ocean Forecast System (RTOFS) grids.
• LDM 6.13.7.
• OGC and Data Delivery plugins have been leveraged to read NOMADS product catalog from a CAVE client. Further work is required to enable user-directed ingest/subscription to specific products.

Future Activities

• Test AWS API for public data set retrieval via EDEX.

• Investigate how the Weather Event Simulator 2 Bridge (WES-2 Bridge) can be (and if it should be) merged into the Unidata AWIPS baseline.
• Migrate more NCEP Viz plugins from the NCP to D2D.
• Investigate creating an AppImage execulable for CAVE for wider Linux distro support.
• Add networking to the CAVE Archive Case Study creation tool.

Metrics

AWIPS downloads

GEMPAK downloads

Python-AWIPS downloads

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Enable widespread, efficient access to geoscience data
   Both AWIPS and GEMPAK are freely available, and both incorporate LDM/IDD technology for accessing geoscience data.  The cloud-based EDEX data server continues to see widespread use and growing adoption. More and more datasets continue to be added to the server as Unidata deploys more decode/ingest nodes.

2. Develop and provide open-source tools for effective use of geoscience data
   Both AWIPS and GEMPAK are open-source, and while GEMPAK is now in maintenance mode, AWIPS is continuously being developed.  

3. Provide cyberinfrastructure leadership in data discovery, access, and use
   Unidata is the only known entity to provide a freely-available and non-operational version of the AWIPS software package.  Unidata continues to find and make available new datasets through the AWIPS project.
   
4. Build, support, and advocate for the diverse geoscience community
   Using LDM/IDD technology to provide access to real-time meteorological data; providing visualization tools for data analysis.

Prepared  March 2019