Status Report: AWIPS and GEMPAK

April 2017 - October 2017

Michael James

AWIPS Training

In lieu of the annual training workshop held in Boulder, Unidata conducted three regional workshops during the summer of 2017, two of which (Millersville University and the University of Wisconsin in Madison) featured AWIPS sessions covering CAVE and python-awips (via Unidata Python Training Workshop Jupyter Notebooks).  Using virtual python environments created by Conda installed alongside the awips2-python RPMS, we were able to manage two versions in the workshop (Python 2.7 for CAVE/python-awips and Python 3 for MetPy/Siphon) without any conflict for users.

The Unidata AWIPS User Manual, which is the basis of Unidata training workshops, continues to be regularly updated and expanded. Additions to the manual during this period include a walkthrough of the refactored WarnGen tool, supported GOES-16 products, a detailed account of a distributed EDEX network, an overview of the AWIPS Development Environment, and python-awips examples copied from the python-awips “ReadTheDocs” site.

Software Releases

AWIPS 17.1.1-1 (July 2017) Major Release

• Support for GOES-16(R) provisional ABI imagery. New GOES East Full Disk map scale added to D2D http://unidata.github.io/awips2/cave/goes-16-satellite/
• Support for GOES-16(R) GLM (Geostationary Lightning Mapper) as it comes online - glm data are not currently being ingested but will be added to edex-cloud as Unidata begins to test distribution feeds internally.
• WarnGEN and TextWS have been refactored to work for any and all WFO sites, with CWA geometries pre-generated in the build.  I've made available a document and screencast walkthrough for using WarnGen to create (but not send) warning products: http://unidata.github.io/awips2/cave/warngen/
• The NSHARP plugin is now written entirely in Java, no longer dependent on legacy NAWIPS/GEMPAK libraries.
• The Manual Ingest Endpoint has been moved from /awips2/edex/data/hdf5/manual to /awips2/data_store/ingest to allow for manual file ingest on an nfs-mounted raw data store.  
• LDM pattern actions added for TPCWindProb, RTOFS, Sea Ice, CFS-DNG, NAEFS.
• 3-way right-click toggle in CAVE to cycle the resource stack through displaying Products, Maps, and Current Time.

AWIPS 17.1.1-2 (July 2017) Minor Release

• Fixes Open Bundle dialog error in CAVE.
• Preserves current map views by opening saved bundles in their own editors.
• File > Delete Bundle menu item to manage saved bundles.
• Cleanup GOES-R Derived Motion Wind menu for provisional products.
• awips2-gfesuite RPM replaces gfesuite-server and gfesuite-client to avoid package conflict on standalone systems.
• macOS bundled JRE jspawnhelper set to executable to enable CAVE PyDev config.
• gridslice.so added to awips-python.pkg to support NumPy array slicing in CAVE (previously prevented isentropic grid bundles from loading)
• The python dependency package is now called "awips-python.pkg" rather than "awips-python-jep.pkg" and include Jep, gridslice, and other required libraries for macOS.

AWIPS 17.1.1-3 (August 2017) Minor Release

• Updates to GFE client to enable grid editing in macOS, if client connects with default localization site OAX.
• Removed all ISC fax/send dialogs from the GFE perspective.
• Maps and NCEP database RPM fix to include shell login with su psql calls.
• Removed all remaining unused Dat (local data), Radar Server dependencies, LDAD, and OHD plugins which reduced the EDEX RPM build time considerably.

AWIPS in the Cloud

Unidata continues to provide access to real-time AWIPS data in the cloud thanks to the XSEDE Jetstream grant outlined in the Cloud Activities status report.

• edex-cloud is now distributed across 5 Jetstream server, with 4 decoding nodes store data to a single database and request server instance.

• ingest/ingestGrib - default IDS|DDPLUS|FNEXRAD|NGRID|CONDUIT|NIMAGE|UNIWISC
• ingestGoesr - NIMAGE 
• ingestRadar - NEXRAD3
• ingestExternal - NAM3km via NCEP ftp; experimental ingest of AWS public datssets; HRRRX; WRF?

• This configuration separates decoding resources from request resources, and avoids PyPIES and LDM both writing to disk simultaneously.
• Remote PyPIES (HDF5) access is allowed by opening ports 9581 and 9582 in iptables
• Remote PostgreSQL is allowed by opening port 5432 in iptables and specifying unique IP address in pg_hba.conf, and requiring clients to supply passwords (sent with md5 encryption) - outlined at http://unidata.github.io/awips2/edex/distributed-computing/
• This is a different distributed configuration from what was running in the Azure cloud in 2016, which used a shared drive to separate LDM ingest from EDEX decoding (not possible in Jetstream).
• “edex password <newpassword>” command to update hibernate.cfg.xml files (metadata, maps, etc.) - after manually changing password from default on the database/request server (cmd line from psql) - not essential since pg connections from specific IPs are defined in pg_hba.conf and iptables, but good practice.
• yum groupinstall awips2-ingest and yum groupinstall awips2-database available (former does not include postgresql, psql, PyPIES/hdf5, maps-database, topo, etc,; latter does not include LDM, ingest, ingestGrib)
• “edex start ingest” and “edex start database” commands available
• What this means: EDEX lives up to its SOA name, totally modularized… we could spin up a VM every 6 hrs to run a WRF domain, decode output and persist it to edex-cloud before shutting down the VM.
• Easier to manage clean reinstalls of EDEX which don’t remove and reinstall the database and its many large files and slow progress.
• With postgres SSL certificates in the pipeline (17.3.1) we could even allow trusted users to spin up edex decoder instances for decoding their own datasets, which are saved to edex-cloud and available in CAVE and Python.

OS X Support

Unidata has supported CAVE for OS X since 15.1.1. In last year’s status report I outlined three obstacles which were being addressed to make the Mac client more functional, and all three have (mostly) been fulfilled

1. Jep and PyDev configuration has not worked correctly in previous versions (15.1.1, 16.2.2).  Fixing this will allow for derived parameter functions to be run from CAVE (which is running locally-installed Python) - Complete: Jep has been bundled in all 17.1.1 releases and is correctly rendering derived parameters and wind representations
2. GEMPAK and NSHARP libraries should be built without static gcc references, we should not expect users to have installed libgfortran via homebrew or macports in order to plot GEMPAK-like grids - Partially Complete: CAVE now uses pure Java for the NSHARP plugin, no longer dependent on GEMPAK/NAWIPS libraries.  AWIPS GEMPAK-like grid rendering is still dependent on legacy GEMPAK/NAWIPS libraries.
3. CAVE for Mac Sierra (10.12) must now be signed for distribution outside the App Store, otherwise the application will be quarantined by Gatekeeper Path Randomization, which prevents applications delivered outside of the store from running by normal means (taskbar icon or finder)  - Complete: Unidata CAVE for macOS is now signed with the UCAR developer certificate and able to install and run without being quarantined (though you will still be warned about its origin since it is downloaded outside of the App Store).

Python Data Access Framework (python-awips)

The standalone Python Data Access Framework (DAF) for retrieving data from an AWIPS EDEX server has been updated with changes from Vlab 17.1.1 repositories, and a Python 3-compliant version has been made available by PyPI (installable via pip) and integrated into the Python Training Workshop.

However, irregularities have been noticed in what datasets and parameters are available via the Python DAF between EDEX versions 16.2.2 and 17.1.1.  Further changes coming in 17.3.1 may resolve these issues but it’s ongoing work to figure out how AWIPS Java and Python correctly communicate.

Material Design Documentation Framework

I maintain a Material Design documentation package called mkdocs-unidata (a fork of mkdocs-material https://github.com/squidfunk/mkdocs-material), which converts markdown files (including partials and includes) to HTML with a Material Design rendering, hosted seemlessly on GitHub Pages.

• https://mjames-upc.github.io/mkdocs-unidata/

• pip install mkdocs-unidata
• Used for the Unidata AWIPS User Manual https://unidata.github.io/awips2/
• Soon to be used for a redesigned GEMPAK User Manual

GEMPAK/NAWIPS

GEMPAK 7.4.1 (September 2017)

https://github.com/Unidata/gempak/releases/tag/7.4.1

• Grib2 table updates for moisture, momentum, mass, radiation, hydrology, cloud/aerosol, more.
• Removed repetitive dcmetr log messages.
• Pattern actions in ldm/etc/templates/pqact.gempak_decoders_grid for various NamDNG grid and the National Blend of Models.
• Alias/mod_res updates, increased max grids in file for gfs, nam, National Blend, SST, URMA, others.
• Bug fix for AWIPS grid retrieval in scripts/python/GridDataRetriever.py

GEMPAK 7.4.0 (August 2017)

https://github.com/Unidata/gempak/releases/tag/7.4.0

• Increased nex2img maximum GINI size

• Maps and bounds file operational updates from NAWIPS 7.4.0
• Grib table and pqact updates for 2.5km GFSMOS
• Fixed G-AIRMET issues when more than 20 points are drawn below the centroid of a smear or snapshot polygon
• Bug fix in ensemble diagnostic function ENS_CVAL
• Added new GEMPAK grid diagnostic function: neighborhood functions, developed by SPC
• Updated HYSPLIT restore files for parameter name changes
• Fixed nmap2 and gdplot2 bug where program was unable to display global wrapped CED grids for subareas including poles
• Fixed prob2cat bug that occurs while composing convective outlook, and bug where lines are drawn outside international boundaries
• Fixed dccsig when handling isolated or lines of thunderstorms
• Linux binaries built with Docker containers on Travis CI for CentOS 6, CentOS 7, Fedora Core 26, Ubuntu 16.04.

GEMPAK 7.3.2 (June 2017)

https://github.com/Unidata/gempak/releases/tag/7.3.2

• Further gdradr support for dual-pol products
• dcgrib decoding fix for NOAAport NAM212.
• g2vcrdwmo6.tbl, g2varswmo0.tbl update for dcgrib.
• gdplot fix for missing titles

Activities Ongoing/In-Progress

• Continue to expand and add to the Unidata Users Manual.
• Continue to investigate distributed cloud-based AWIPS data ingest and storage.
• Continue to expand the edex-cloud archive of real-time data.

• AWIPS 17.1.1-4 (next release candidate) Continuous Integration

• EDEX and Qpid RPM builds run on remote push by Travis CI, using Centos 6 Docker container to install AWIPS Development Environment (ADE) packages (Java, Eclipse, Python, Ant), clone the multiple awips2 repositories from github.com/Unidata, and build a subset of RPMs.
• It’s not feasible to automate these builds for CAVE given the manual PyDev configuration required within Eclipse on the desktop - need to determine if we can boostrap packages a PyDEV config inside Eclipse for RPM distribution.

• AWIPS 17.1.1-4 (next release candidate) RPM Package Consolidation

• Last year’s AWIPS release 16.2.2 consolidated all awips2-cave-<pluginname> RPMs into two packages: awips2-cave and awips2-cave-wrapper.
• Current release candidate 17.1.1-4 consolidates all awips2-edex-<pluginname> RPMs into the existing awips2-edex-dataplugins package. Rather than installing (awips2-edex-binlightning, awips2-edex-bufr, awips2-edex-grid, etc.), it’s a single RPM: awips2-edex-dataplugins.

New Activities

In the next 3 months

• Dockerize gempak on top of LDM.
• Dockerize and deploy gempak container from Travis CI.
• Test AWS API for public data set retrieval via EDEX.
• Update AWIPS postgres/database to to use SSL certificates for remote connection (17.3.1 baseline)
• Create unique, deployable ingest modes for Radar, Goesr, Grid, other data sources, essentially mini-instances of EDEX for each datatype focus.
• Deploy a standalone AWIPS Archive EDEX server in the cloud to contain noteable case study events.

Further past 3 months

• Investigate how the Weather Event Simulator 2 Bridge (WES-2 Bridge) can be (and if it should be) merged into the Unidata AWIPS baseline.
• Migrate some NCEP-created Viz plugins from the National Centers Perspective to the D2D Perspedctive.
• Investigate creating an AppImage execulable for CAVE for wider Linux distro support.
• Add networking to the CAVE Archive Case Study creation tool.

Metrics

AWIPS Downloads

• 4378 total
• See core institutions
• See all EDUs
• See per month
• See affiliation types
• See countries
• See US vs non-US
• See Core vs non-Core
• See downloads by special interest group
• See list of download sites

AWIPS Cloud Server Usage Metrics

• 41 universities known to use edex-cloud in roughly the last 6-month period (March - October 2017).

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

1. Enable widespread, efficient access to geoscience data
   Both AWIPS and GEMPAK are freely available, and both incorporate LDM/IDD technology for accessing geoscience data.  The cloud-based EDEX data server continues to see widespread use and growing adoption. More and more datasets continue to be added to the server as Unidata deploys more decode/ingest nodes.

2. Develop and provide open-source tools for effective use of geoscience data
   Both AWIPS and GEMPAK are open-source, and while GEMPAK is now in maintenance mode, AWIPS is continuously being developed.  

3. Provide cyberinfrastructure leadership in data discovery, access, and use
   Unidata is the only known entity to provide a freely-available and non-operational version of the AWIPS software package.  Unidata continues to find and make available new datasets through the AWIPS project.
   
4. Build, support, and advocate for the diverse geoscience community
   Using LDM/IDD technology to provide access to real-time meteorological data; providing visualization tools for data analysis.

Prepared  October 2017