Published using Google Docs
IDV with RAMADDA
Updated automatically every 5 minutes

Status Report: IDV with RAMADDA

September-April 2016

Yuan Ho, Julien Chastang

Activities Since the Last Status Report

IDV System Changes

 __Latest netCDF-Java Version__

The version of the netCDF-Java library currently distributed with the IDV is the 4.6.2 (June, 2015). Its main feature is a rewrite of GRIB handling. This version of netCDF-Java includes changes: AWIPSsat projection, WRF lat/lon coordinates with time dependence, updated grib1 table for NLDAS, handling of scalar runtime coordinate for GRIB collections. See the netCDF-Java Library for a more details on these changes.

 __ISL Changes__

IDV Display Changes

 __Latest Version of VisAD__

In the last six months, there have been several feature enhancements and bug fixes in VisAD:

 __Other Changes__

IDV CESM Demonstration

In March  2016, a well received IDV demonstration was presented by Yuan to the CESM  (Community Earth System Model) group. They are requested by their funding agency to use IDV for their model output visualization.

IDV WRF-Hydro Collaboration

The IDV team is collaborating with David Gochis from NCAR-RAL assisting him in ensuring WRF-Hydro data is CF compliant according Point Feature type. In addition, we are helping David with visualization of this dataset the IDV.

IDV WRF-Tutorial Workshop

In the January WRF tutorial workshop, Yuan gave a 45 minutes presentation of the IDV main features and its applications in the WRF output datasets. We also help out the students in the classroom.

IDV Turbulence Demonstration

With Ryan’s Python script to process the fortran binary array, the IDV is able to demonstrate the fascinating turbulence  vertex structure.

IDV XSEDE Jetstream Collaboration

The Extreme Science and Engineering Discovery Environment (XSEDE) five-year, $121-million award is  a National Science Foundation supported project to provide supercomputing and cloud computing resources to the U.S. academic and research community. The IDV and XSEDE Jetstream cloud computing teams collaborated to demonstrate the cloud streaming IDV at Supercomputing 2015 in Austin, Texas. This collaboration also produced a video (see below) on this effort.

IDV Release

The last IDV release was 5.2 in August of 2015.

IDV Proposal Work

 __EarthCube__

In collaboration,  with University of Miami Professor Brian Mapes, Unidata submitted an EarthCube proposal:  “Drilling down from a statistics scatterplot to pre-populated case note books”

 __NASA ROSES__

In July of 2015, with our University of Wisconsin, Space Science and Engineering Center partners, we submitted a proposal to NASA entitled, “Interactive Algorithm Development and Product Validation through Innovative Data Access and Visualization Methods”. We are still awaiting a response from that proposal.

IDV Publication Highlights

A Google Scholar Search reveals a number of publications that cite use of the IDV.

RAMADDA

Docker is a new cloud-centric technology that borrows from the notion of containers from the shipping industry to facilitate installation and deployment of server side applications. We have implemented a Docker container for easy distribution and installation of RAMADDA in a cloud environment. We coupled this effort with a Dockerized LDM with the goal of serving data in a cloud environment. This work was presented at 2016 AMS annual meeting in New Orleans in a presentation entitled Unicloud: Docker Use at Unidata. We staged three demonstration servers on the Azure cloud.  In this example, RAMADDA is serving IDD data supplied via the LDM.

IDV and RAMADDA Training, Conference Attendance and Presence

 __2016 American Meteorological Conference (AMS) Annual Meeting__

 __2015 American Geophysical Union (AGU) Annual Meeting__

 __IDV Instructional Videos__

Produced a new IDV training video entitled Running the IDV on the XSEDE Jetstream Cloud Demo.


Ongoing Activities

We plan to continue the following activities:

 __IDV in the Cloud __

With the goal of better serving our core community and in fulfillment of objectives articulated in “Unidata 2018: Transforming Geoscience through Innovative Data Services”,  the IDV team will continue to investigate how its technologies can best take advantage of cloud computing. To this end, we have been employing Docker container technology to streamline building, deploying, and running Unidata technology offerings in cloud- based resources. Specifically, we have created Docker images for the IDV, RAMADDA,  we are working on the LDM which, coupled with RAMADDA, will allow for the serving of real-time data in a cloud environment for IDV users. We have been experimenting with these Docker containers in the Microsoft Azure and Amazon.

 __IDV Instructional Videos __

We plan to continue producing more instructional videos on the IDV. We would appreciate input and suggestions on specific video topics.

 __IDV Seam Issues__

We will continue to collaborate with the Unidata netCDF-Java team to devise a solution for long-standing longitude seam issues in the IDV.

New Activities

Over the few months, we plan to organize or take part in the following:

 __IDV Training Workshop, October, 2016__

We will offer our annual training workshop in the Fall of 2016.

 __Investigation of Java 3D Alternative__

Because of concerns about the long-term viability the open-source Java 3D project, the IDV team has begun discussions with our University of Wisconsin, SSEC collaborators to replace Java 3D with a more viable alternative within the VisAD API. We have started investigating whether the Ardor 3D can meet that objective. Looking into alternatives to Java 3D was also a goal described in the Unidata 2018 Five-year plan.

Areas for Committee Feedback

We have no questions at this time.

Relevant Metrics

 __E-Support__

The IDV team continues to provide the geoscience community with high-quality support through e-support software and idv-users maillist. The volume of e-support remains high and constitutes a large fraction of our daily activities. In the last half year the IDV team has closed ~100 e-support tickets.  Each individual ticket may and often does involve many back-and-forth messages. There is an especially large number of support requests coming from international users.

 __Usage Metrics__

Raw IDV usage metrics, are available here http://www.unidata.ucar.edu/software/idv/logging/left.html.

Top ten universities running IDV are: Millersville, Oklahoma, University of Utah, St Cloud state, Plymouth, NC State, West Kentucky, Lyndon State, University of Illinois, and San Francisco State.

 __Github Pull Requests__

In the area of greater collaborative development, since the migration of the IDV project to github, we have closed a total of 97 “pull requests” or code contributions from internal and external collaborators.

 __Youtube IDV Instructional Videos__

In the area of online IDV training, the Youtube IDV instructional videos have been viewed over 8,000 times compared with 6,115 from six months ago.

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

  1. Enable widespread, efficient access to geoscience data
    The IDV is a state of the art geoscience visualization application. It gives users the ability to view and analyze a rich set of geoscience data, including real time data, in a seamless and integrated fashion. This analysis is captured in IDV bundles. RAMADDA is a content management system and service specifically tailored towards the sharing and distribution of IDV bundles facilitating distribution of scientific data and analysis.
  2. Develop and provide open-source tools for effective use of geoscience data
    The IDV has been an open-source project for several years. The IDV is available on the github version control platform for greater open-source collaboration. The IDV provides users the unparalleled ability to analyze, integrate, and visualize heterogeneous geoscience data in two, three, and four dimensions. The IDV coupled with RAMADDA enables geoscience specialists the capability to share and collaborate their IDV analysis via social scientific networks.
  3. Provide cyberinfrastructure leadership in data discovery, access, and use
    RAMADDA allows geoscience specialists the ability to search and publish their IDV bundles on-line. Unidata's RAMADDA installation enables the IDV team to communicate more effectively to our users concerning their IDV issues. Specifically, during support ticket conversations, the IDV team requests that users upload pertinent data to RAMADDA for analysis. One of RAMADDA’s best features is the ability to upload a CDM file and obtain the OpenDAP link from the new entry. The DAP link can be shared and opened in the IDV. The IDV team also takes advantage of RAMADDA to share instructional IDV screencasts with users.
  4. Build, support, and advocate for the diverse geoscience community
    Unidata offers yearly multi-day training and occasionally regional workshops for IDV and RAMADDA. The IDV coupled with RAMADDA enables our earth science community partners to distribute geoscience data and metadata through web-based technologies thereby fostering scientific collaborations. Moreover, the IDV's ability to share bundles through RAMADDA creates a scientific social and collaborative network for the geoscience community.


Prepared April 2016