Status Report: IDV with RAMADDA

April 2017 - October 2017

Yuan Ho, Julien Chastang

Activities Since the Last Status Report

IDV System Changes

__Latest netCDF-Java Version__

The version of the netCDF-Java library currently distributed with the latest stable IDV version (5.4) is the 4.6.10 (April 19, 2017). The prior version of netCDF-Java to be distributed with the IDV was 4.6.7. There have been many improvements and bug fixes in that range. The complete release notes for these versions can be found here.

__ISL Changes__

Improved ISL offscreen mode
Fixed ISL time driver start and end time bug
Various improvements related to saving and loading of bundles with respect to time and region matching
New ISL feature to publish to the RAMADDA

__Java Version__

The IDV is distributed with Java 8 u51. Because of a Windows 10 related bug on some older Intel hardware, this will be the latest version of Java we will be deploying with the IDV for some time.

__Java 3D Version__

The IDV is distributed with Java 3D version 1.6.0 Final. Java 3D 1.7 prerelease is under development and continues to be supported in some capacity by the open source community.

__HTTP to HTTPS__

As US government servers transition from HTTP to HTTPS, we update URL references in the IDV (rbi files, plugin files, etc.) to reflect these changes.

__Derived Quantities__

Relative humidity calculation improvements for handling supersaturation.
New Jython grid diagnostic formulas for calculating the wind shear between discrete layers.

__Point Observations__

Can now display relative humidity, wind chill, and heat index for metar point observations

__Plugins__

The IDV Jetstream plugin allows easy access to Jetstream installations of the TDS, RAMADDA and ADDE from the IDV. The plugin URL is http://science-gateway.unidata.ucar.edu/jetstream.jar

IDV Display Changes

__GOES-16 GLM (Global Lightning Mapper) data in the IDV__

A new GLM plugin has been developed to add three new data source types in the IDV (5.4 and nightly): GLM GROUPS Data Files, GLM EVENTS Data Files, and GLM FLASHES Data Files. These three new data source types will enable IDV users to load and display GLM data from

Local disk and remote TDS systems.

__Super High Resolution Display__

Added a new FAQ item to help IDV users to deal with the small font and display issues in a super high Resolution screen.

__2D/3D Trajectories__

Improvements and enhancements in the underlying VisAD library has enabled a new grid 2D/3D trajectory feature in the IDV. The 2D/3D grid trajectories can be colored by speed or other parameters. The initial area of the trajectory is the entire domain of the dataset. However, regions can be selected with points, rectangles and closed polygons.

There are a variety of grid trajectory types including ribbon, cylinder, deform ribbon, point.

__Latest Version of VisAD__

The SSEC team at UW, Madison has made a number of improvements to support 3D trajectories.

__Skew T__

Fixed a bug concerning CAPE and CIN calculations in the Skew-T display.

__Transect Line__

Fixed the problem of the transect line modification in the -180 to 0 longitude domain.

__IDV Web Cams__

The IDV webcam list continues to be updated regularly

IDV WRF-Hydro Collaboration

Jeff Weber and Yuan are collaborating with David Gochis, and Dave Yates from NCAR-RAL assisting them ensuring WRF-Hydro data is CF compliant regarding Point Feature type. In addition, we are helping them with visualization of this dataset in the IDV.

IDV Release

IDV 5.4 was released in May of 2017.

IDV EarthCube Proposal Awarded

In collaboration, with University of Miami Professor Brian Mapes, Unidata submitted an EarthCube proposal: “Drilling down from a statistics scatterplot to pre-populated case notebooks”. This proposal was awarded $1.3 million over three years.

Unidata hires Celia Hedrick, a Software Engineer II to work on this project.

IDV Publication Highlights

A Google Scholar Search reveals a number of publications that cite use of the IDV (doi:10.5065/D6RN35XM).

RAMADDA and ADDE in the Cloud

Building upon previously described Docker containerization efforts, we have deployed RAMADDA on the XSEDE Jetstream cloud. This installation of RAMADDA has a “view” of the IDD on Jetstream. In addition, there is a collection of IDV bundles that retrieve data from the TDS installation on Jetstream.

ADDE is a data server developed by SSEC that primarily serves satellite and radar imagery and point observations to the IDV. An ADDE server is available at adde-jetstream.unidata.ucar.edu over (the usual) port 112.

IDV and RAMADDA Training, Conference Attendance and Presence

__Gateways 2017__

Unidata will attend the Gateways 2017 conference in Ann Arbor. Our efforts on deploying RAMADDA, ADDE and its use for the IDV will be described in a conference poster.

__2017 American Geophysical Union (AGU) Annual Meeting__

Yuan will attend 2017 AGU and present: The Comparison of Point Data Models for the Output of WRF Hydro Model in the IDV

__2018 American Meteorological Conference (AMS) Annual Meeting__

Yuan will attend 2018 AMS and present: The new trajectory displays in the UNIDATA's IDV

Ongoing Activities

We plan to continue the following activities:

__IDV Instructional Videos__

No instructional videos on the IDV are planned at this time. We would appreciate input and suggestions on specific video topics.

__IDV Seam Issues__

We will continue to collaborate with the Unidata netCDF-Java team to devise a solution for

long-standing longitude seam issues in the IDV.

__Investigation of Java 3D Alternative__

Because of concerns about the long-term viability the open-source Java 3D project, the IDV team has begun discussions with our University of Wisconsin, SSEC collaborators to replace Java 3D with a more viable alternative within the VisAD API. We have started investigating whether the Ardor 3D can meet that objective. Looking into alternatives to Java 3D was also a goal described in the Unidata 2018 Five-year plan.

New Activities

Over the few months, we plan to organize or take part in the following:

__GOES-16 GLM Collaboration__

At the 2017 EarthCube netCDF-CF Workshop, Unidata began discussions with SSEC, Randy Horne (Excalibur labs), Professor Eric Bruning (Texas Tech) on a path forward for displaying GOES-16 GLM (Global Lightning Mapper) data in the IDV.

Areas for Committee Feedback

We have no questions at this time.

Relevant Metrics

__E-Support__

The IDV team continues to provide the geoscience community with high-quality support through e-support software and idv-users maillist. The volume of e-support remains high and constitutes a large fraction of our daily activities. In the last half year the IDV team has closed ~90 e-support tickets. Each individual ticket may and often does involve many back-and-forth messages. There is an especially large number of support requests coming from international users.

__Usage Metrics__

Raw IDV usage metrics, are available here http://www.unidata.ucar.edu/software/idv/logging/left.html.

Top ten universities running IDV are: Millersville, Oklahoma, University of Utah, St Cloud state, Plymouth, NC State, West Kentucky, Lyndon State, University of Illinois, and San Francisco State.

__Github Pull Requests__

In the area of greater collaborative development, since the migration of the IDV project to github, we have closed a total of 111 “pull requests” or code contributions from internal and external collaborators.

__Youtube IDV Instructional Videos__

In the area of online IDV training, the Youtube IDV instructional videos have been viewed 13,200 times compared with 11,500 from six months ago.

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

Enable widespread, efficient access to geoscience data
The IDV is a state of the art geoscience visualization application. It gives users the ability to view and analyze a rich set of geoscience data, including real time data, in a seamless and integrated fashion. This analysis is captured in IDV bundles. RAMADDA is a content management system and service specifically tailored towards the sharing and distribution of IDV bundles facilitating distribution of scientific data and analysis.
Develop and provide open-source tools for effective use of geoscience data
The IDV has been an open-source project for several years. The IDV is available on the github version control platform for greater open-source collaboration. The IDV provides users the unparalleled ability to analyze, integrate, and visualize heterogeneous geoscience data in two, three, and four dimensions. The IDV coupled with RAMADDA enables geoscience specialists the capability to share and collaborate their IDV analysis via social scientific networks.
Provide cyberinfrastructure leadership in data discovery, access, and use
RAMADDA allows geoscience specialists the ability to search and publish their IDV bundles on-line. Unidata's RAMADDA installation enables the IDV team to communicate more effectively to our users concerning their IDV issues. Specifically, during support ticket conversations, the IDV team requests that users upload pertinent data to RAMADDA for analysis. One of RAMADDA’s best features is the ability to upload a CDM file and obtain the OpenDAP link from the new entry. The DAP link can be shared and opened in the IDV. RAMADDA also has “server-side view” capability where a specific part of the file system can be made available through the RAMADDA interface. This feature can be helpful to view LDM data feeds, for example. The IDV team also takes advantage of RAMADDA to share instructional IDV screencasts with users.
Build, support, and advocate for the diverse geoscience community
Unidata offers yearly multi-day training and occasionally regional workshops for IDV and RAMADDA. The IDV coupled with RAMADDA enables our earth science community partners to distribute geoscience data and metadata through web-based technologies thereby fostering scientific collaborations. Moreover, the IDV's ability to share bundles through RAMADDA creates a scientific social and collaborative network for the geoscience community.

Prepared October 2017