Status Report: AWIPS and GEMPAK

October 2017 - March 2018

Michael James

Questions for Committee Members

(from the GOES Status Report) Which GOES-16/17 bands and coverages should be converted from netCDF-4 to McIDAS AREA format for GEMPAK/NMAP2?
What additional data and map resources could be added to AWIPS?

AWIPS

Unidata continues to maintain an EDEX data server on the Jetstream cloud, serving real-time AWIPS data to CAVE clients and through the python-awips data access framework (API). The distributed architectural concepts of AWIPS allow us to easily scale EDEX in the cloud to account for the size of incoming data feeds. We continue work using Jetstream to develop cloud-deployable AWIPS instances, as imaged virtual machines (VMIs) available to users of Atmosphere and OpenStack, and as docker containers available on Docker Hub and deployable (soon) with the xsede-jetstream toolset.

Jetstream AWIPS EDEX Standalone VMI

This EDEX image can serve either as a full standalone server or as a database/request server.

Jetstream AWIPS EDEX Ingest Node VMI

This image contains all AWIPS Python and EDEX software, but nothing database or http-related. This VMI makes it easy to deploy datatype-specific ingest nodes (Grid, Radar, Satellite, etc.) with simple edits to ldmd.conf and modes-ingest.xml

AWIPS 17.1.1-5 released January 2018

GOES East (16) updates for operational mode, geostationary position adjustment, ingest of stitched CONUS and Full Disk netCDF files, menu updates for Geostationary Lightning Mapper (GLM), and Derived Motion Winds (DMW)
EDEX no longer installs awips2-python-setuptools, which was not required and in some cases would prevent virtual machines from booting due to Python version conflicts.
Removed IFPS server send dialogs from GFE perspective.
Support for NationalBlend, HIRLAM, more complete HRRRX parameters
Distributed computing support for ingest nodes writing data to a remote database/request instance - read more at http://unidata.github.io/awips2/edex/distributed-computing/
Fix for long request times to fill colormap editor right-click menu, which now displays only your own USER-level localization colormap edits
Removed NCEP EDEX data decoders (rpm awips2-edex-ncep) from build and install (this does not affect the common/viz plugins or the NCP)
Menu and bundle updates for WPC QPF/QPE grids
Native handling of PNG-compressed GINI and AREA files with ar.com.hjg.pngj
Pre-built el6 and el7 binaries of edexBridge now used for bundling during build in container
Converted and added Matplotlib colormaps to AWIPS .cmap format, available in the right-click menu Change Colormap dialog:

Removed bundled netCDF RPM packages to avoid conflict with epel repositories
Removal of more unused AWIPS1, Hydro, RadarServer components
Software updates for qpid, httpd-pypies, postgresql, matplotlib, numpy, cython, shapely, python-qpid, dateutil, python-scientific, python-tables.
Removed unused python modules nose, scipy, pupynere, pwm.
The script qpidNotify.py has been added to /awips2/edex/bin and is available in $PATH for non-root users on an AWIPS server.

Python Data Access Framework (python-awips)

An early version of the AWIPS Data Portal has been published at http://js-157-49.jetstream-cloud.org:8080, including EDEX Server Inventories, Forecast Models, NEXRAD level 3, and instructions on how to build a GeoJSON API.

0.9.12 released March 9, 2018

New NEXRAD dictionary in tables.py programmatically-generated from AWIPS WSR-88D radar definition file with class GenerateNexradTable (from awips.tables import nexrad)
Added pandas and h5py to conda environment
All EDEX-side Derived Parameters are now callable from python-awips and returned as 2-d Numpy arrays (as of EDEX 17.1.1-5)

0.9.11 release Feb 11, 2018

Added profiler site metadata to tables.py profiler time series notebook (from awips.tables import profiler)
Created and published a Profiler Wind Barb Time-Series Jupyter notebook using newly-acquired ESRL/PSD profiler hourlies.

GEMPAK/NAWIPS

Docker Container

https://hub.docker.com/r/unidata/gempak/

This image contains the latest GEMPAK RPM (7.4.1), epel-release, and python-pip
Python packages installed via pip include python-awips, numpy, shapely, six
To run:

docker pull unidata/gempak
docker run --user gempak -it --rm=true unidata/gempak:latest
./rungempak.sh from within the container to run GDINFO using remotely-retrieved AWIPS grids.

GEMPAK 7.4.1 (September 2017)

https://github.com/Unidata/gempak/releases/tag/7.4.1

Grib2 table updates for moisture, momentum, mass, radiation, hydrology, cloud/aerosol, more.
Removed repetitive dcmetr log messages.
Pattern actions in ldm/etc/templates/pqact.gempak_decoders_grid for various NamDNG grid and the National Blend of Models.
Alias/mod_res updates, increased max grids in file for gfs, nam, National Blend, SST, URMA, others.
Bug fix for AWIPS grid retrieval in scripts/python/GridDataRetriever.py

Activities Ongoing/In-Progress

AWIPS

Continue work on a deployable edex-ingest Docker container.
Continue work on a deployable edex-ingest-wrf instance
Collaborate with AWIPS developers at GSD on distributed containerization of AWIPS/EDEX.
Continue to create and publish datatype workflow Jupyter notebooks for python-awips
Continue expanding the capabilities of the AWIPS Data Portal as new Jupyter notebooks are created.
Next AWIPS release will be 17.1.1-6
GOES East (16) updates for operational mode, geostationary position adjustment, ingest of stitched CONUS and Full Disk netCDF files, menu updates for Geostationary Lightning Mapper (GLM), and Derived Motion Winds (DMW)
Moved /awips2/edex/bin/edex to /usr/bin/edex to find in sudo user path
Profiler map resource added to CAVE, selectable sites to load a time-series

Added qpid-stat as “edex qpid” command to monitor datatype message queues
Fix for missing colormap to default to Grid/Gridded Data (rainbow) and alert/WARN the user that the colormap in their bundle could not be found. This solves the problem created by simply renaming a .cmap file which used to throw an error and fail to load the product/bundle. Because colormap definitions are controlled on the EDEX side of the server-client exchange, this catch allows us to rename and update colormaps without users having to update or reinstall CAVE.
CreateOpenStack-created VMIs

GEMPAK

Next release will be 7.4.2
GOES fixed grid projection McIDAS navigation added to development GEMPAK
GOES16/17 sectors and products to be made available in AREA format for GEMPAK is still TBD
gdpvsf added back to release after fix for max number of grids to avoid memory relocation error

New/Future Activities

Dockerize gempak on top of LDM.
Test AWS API for public data set retrieval via EDEX.
Expand AWIPS geospatial city table to include major world cities.
Deploy a standalone AWIPS Archive EDEX server in the cloud to contain notable case study events.

Investigate how the Weather Event Simulator 2 Bridge (WES-2 Bridge) can be (and if it should be) merged into the Unidata AWIPS baseline.
Migrate NCEP Viz plugins from the NCP to D2D.
Investigate creating an AppImage execulable for CAVE for wider Linux distro support.
Add networking to the CAVE Archive Case Study creation tool.
Incorporate USER-level bundles and modifications in the D2D data selection menus

Metrics

AWIPS Downloads

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

Enable widespread, efficient access to geoscience data
Both AWIPS and GEMPAK are freely available, and both incorporate LDM/IDD technology for accessing geoscience data. The cloud-based EDEX data server continues to see widespread use and growing adoption. More and more datasets continue to be added to the server as Unidata deploys more decode/ingest nodes.

Develop and provide open-source tools for effective use of geoscience data
Both AWIPS and GEMPAK are open-source, and while GEMPAK is now in maintenance mode, AWIPS is continuously being developed.

Provide cyberinfrastructure leadership in data discovery, access, and use
Unidata is the only known entity to provide a freely-available and non-operational version of the AWIPS software package. Unidata continues to find and make available new datasets through the AWIPS project.
Build, support, and advocate for the diverse geoscience community
Using LDM/IDD technology to provide access to real-time meteorological data; providing visualization tools for data analysis.

Prepared March 2018