Published using Google Docs
Python
Updated automatically every 5 minutes

Status Report: Python

September 2015 - March 2016

Ryan May, Sean Arms, Julien Chastang, Ward Fisher

Activities Since the Last Status Report

Siphon

Siphon represents a rebranding of PyUDL, as we try to elevate our Python support in TDS to a higher status. We anticipate developing Siphon to ensure that it is easy as possible to download data from a TDS in Python, keeping pace with new features added on the Java side.

Progress has been made on the following:

MetPy

Feedback on MetPy continues to be positive. We have been publicizing the project in a variety of ways:

There have been some external contributions from the community, including a Pull Request adding a function to reduce sounding data to a desired set of data. Ideas for further development are captured in the GitHub issue tracker; current plans are to focus on enhancing the abilities to read point data (to facilitate generating NetCDF data for hosting on THREDDS) and to objectively analyze such data.

Progress has been made on the following:

Dependencies, challenges, problems, and risks include:

External Participation

The Python team attends conferences as well as participates in other projects within the scientific Python ecosystem. This allows us to stay informed and to be able to advocate for our community, as well as keep our community updated on developments.  Ryan May has also continued to be an active participant in the matplotlib community, reviewing some pull requests and contributing several others. We also continue to host Jeff Whittaker’s netCDF4-python project repository; Jeff continues to be the active maintainer of the project.

Progress has been made on the following:

Unidata Python Workshop

We continue to improve and expand this popular workshop, now three days long, with new material, ensuring it stays current with the latest development in the scientific Python area. Specifically, we enhance and add Jupyter Notebooks in the geoscientific domain in addition to making sure we stay on top of infrastructure changes with the conda package manager, and the Jupyter environment.

Python Online Training Effort

Unidata obtained supplemental funds from NSF as part of our five-year award to start an online Python training effort specifically focused on serving the geoscience community. The Unidata Python group has outlined sections for introductory material, and is currently working towards having a draft resource in the early summer.

Ongoing Activities

We plan to continue the following activities:

New Activities

Over the next three months, we plan to organize or take part in the following:

Beyond a one-year timeframe, we plan to organize or take part in the following:

Areas for Committee Feedback

We are requesting your feedback on the following topics:

  1. What are the biggest obstacles that you see to the use of Python with other Unidata technologies, or for use in meteorology in general?
  2. How valuable do find an effort like MetPy to the Python meteorology community? Are there additional barriers we could remove through this project? Are there other efforts over which this should take priority?
  3. Have you seen the two-part blog post using MetPy and Siphon? How was the tone for a “notebook of the week”-type series?
  4. We continue to maintain the Unidata Python Workshop with fresh, relevant, and up-to-date content. However, we would welcome feedback from our committees  on topics we may not be covering in the workshop.

Relevant Metrics

Siphon:

MetPy:

Unidata Python Workshop (from github):

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

  1. Enable widespread, efficient access to geoscience data
    Python can facilitate data-proximate computations and analyses through Jupyter Notebook technology. Jupyter Notebook web servers can be co-located to the data source for analysis and visualization through web browsers. This capability, in turn, reduces the amount of data that must travel across computing networks.
  2. Develop and provide open-source tools for effective use of geoscience data
    Our current and forthcoming efforts in the Python arena will facilitate analysis of geoscience data. This goal will be achieved by continuing to develop Python APIs tailored to Unidata technologies. Starting with the summer 2013 Unidata training workshop, we developed an API to facilitate data access from a THREDDS data server. This effort has been encapsulated with the new siphon project, which is an API for communicating with a THREDDS server. Moreover, Python technology coupled with the HTML5 Jupyter Notebook technology has the potential to address "very large datasets" problems. Jupyter Notebooks can be co-located to the data source and accessed via a web browser thereby allowing geoscience professionals to analyze data where the data reside without having to move large amounts of information across networks. This concept fits nicely with the "Unidata in the cloud" vision and the goals outlined Unidata 2018 Five-year plan. Lastly, as a general purpose programming language, Python has the capability to analyze and visualize diverse data in one environment through numerous, well-maintained open-source APIs. The additional development of MetPy fills the need for domain-specific analysis and visualization tools in Python.
  3. Provide cyberinfrastructure leadership in data discovery, access, and use
    The TDS catalog crawling capabilities found in siphon will facilitate access to data remotely served by the Unidata TDS, as well as other TDS instances around the world.
  4. Build, support, and advocate for the diverse geoscience community
    Based on interest from the geoscience community, Unidata, as part of its annual training workshop, now hosts a three day session to explore Python with Unidata technology. Also, to advance the use of NetCDF in Python, Unidata has promoted Jeff Whitaker’s NetCDF4-python project, including hosting its repository under Unidata’s GitHub account. Unidata is also fostering some community development of meteorology-specific tools under the MetPy grassroots project.

Prepared  March 2016