April - October 2017
Julien Chastang, Ward Fisher, Michael James, Ryan May, Jen Oxelson, Mohan Ramamurthy, Christian Ward-Garrison, Jeff Weber, Tom Yoksas
Building upon our previous containerization efforts, we are developing a Unidata Science Gateway on NSF-funded XSEDE Jetstream Cloud: http://science-gateway.unidata.ucar.edu/. A collection of Unidata related technologies can be found here for our community to make use of directly or with client applications such as the IDV. The following resources are available on this gateway:
Gateway users, coupled with XSEDE HPC resources, can achieve complete end-to-end scientific computing workflows. We presented this work at the ESIP Summer Meeting 2017.
Unidata maintains an EDEX data server on Jetstream to ingest and serve real-time AWIPS data for rendering by the CAVE client and the python-awips data access framework: edex-cloud.unidata.ucar.edu. This EDEX server has successfully supported several AWIPS workshops, and is used by CAVE clients in the Unidata community.
Unidata is running a Nexus Server on Jetstream for the distribution of netCDF-Java artifacts (e.g., netcdfAll.jar, toolsUI.jar, ncIdv.jar): https://artifacts.unidata.ucar.edu. netCDF-Java documentation is also hosted at that location.
To further investigate how the Unidata community can benefit from Unidata technologies in the cloud, Unidata obtained a large XSEDE “Research” grant on the Jetstream cloud-computing platform worth $425,000 in cloud computing resources. The Extreme Science and Engineering Discovery Environment (XSEDE) five-year, $121-million award is a National Science Foundation supported project. In the last six months, we completely transitioned our research and development from our initial "Start Up" allocation to our "Research" allocation.
We have been employing Docker container technology to streamline building, deploying, and running Unidata technology offerings in cloud-based environments. Specifically, we continue to refine and improve Docker images for the IDV, LDM, ADDE, RAMADDA, THREDDS, and Python with Unidata Technologies. We have been experimenting with these Docker containers in the NSF XSEDE Jetstream cloud.
Containerization efforts are currently in maintenance mode with most of the initial development completed. The TDS docker container is the most active of our containerization efforts. We continue to receive open-source contributions. It is also our most popular container on DockerHub.
It is unlikely that most of our community will use these containers directly. Rather they will be leveraged by experts on behalf of the community, or they will be abstracted from users by being integrated into a user-friendly workflow. For example, on Jetstream we have a JupyterHub server currently in development: https://jupyter-jetstream.unidata.ucar.edu. This server was deployed with the aid of cloud computing technologies including Docker. These details, however, are hidden from the user.
In addition, there are overlapping (perhaps, competing or complementary) technologies such as Ansible that are emerging alongside Docker that need to be investigated.
Unidata obtained supplemental funding from NSF to host the 2017 Modeling Research in the Cloud Workshop, May 31 - June 2 2017. The purpose of the conference was to facilitate an in-depth discussion of the myriad aspects and formulate approaches for integrating cloud computing capabilities into the weather and climate prediction landscape and discuss the significance of such integration for advancing discoveries.
After the conference, we had a discussion with Kevin Tyle (University at Albany - SUNY and SAC member), Carlos Maltzahn (University of California, Santa Cruz, Big Weather Web Lead), and John Exby (formerly of NCAR, RAL) on Jetstream cloud use. We initiated a “Startup” grant on Jetstream for experimentation purposes. We also educated the group on the use of the TDS Docker container with a demonstration at Unidata. Subsequently, Kevin Tyle, and Julien Chastang collaborated on the use of the TDS Docker container.
For the past three years, Unidata generated products for the IDD, FNEXRAD and UNIWISC data streams have been created by a VM hosted in the Amazon cloud. This production generation has been proceeding very smoothly with almost no intervention from Unidata staff.
The Open Science Data Cloud, a resource of the Open Commons Consortium (OCC), provides the scientific community with resources for storing, sharing, and analyzing terabyte and petabyte-scale scientific datasets. The OSDC is a data science ecosystem in which researchers can house and share their own scientific data, access complimentary public datasets, build and share customized virtual machines with whatever tools necessary to analyze their data, and perform the analysis to answer their research questions. Unidata is a beta user of resources in the Open Science Data Cloud ecosystem and we have been provided cloud-computing resources on the Griffin cloud platform. Our allocations are renewed on a quarterly basis and Unidata is partnering with OCC on the NOAA Big Data Project. Given the limited staff resources and many ongoing cloud activities on AWS, Azure, and XSEDE environments, Unidata’s activities on the OSDC have been in a temporary hiatus. We are hoping to ramp up our OSDC efforts in the upcoming months.
“Data-Proximate Analysis and Visualization in the Cloud using Cloudstream, an Open-Source Application Streaming Technology Stack”, 2017 AGU Fall Meeting | December 11–15, 2017 – New Orleans, LA USA
We aim to collaborate with Jeremy Fischer at IU, XSEDE and Rich Signell to experiment with the "Zero to JupyterHub" project. The goal is to take advantage of cloud scalability for on-demand use in a classroom setting, for example, with technologies such as OpenStack and Kubernetes.
We would like to promote and advertise the science gateway (http://science-gateway.unidata.ucar.edu/) to our community.
In the long-term, we would like to explore the possibility of migrating some core Unidata services onto the cloud.
Docker image downloads are available from Unidata’s Dockerhub repository.
We support the following goals described in Unidata Strategic Plan:
Prepared October 2017