Published using Google Docs
Python
Updated automatically every 5 minutes

Status Report: Python

November 2022- March 2023

Ryan May, Drew Camron, Julien Chastang, Nicole Corbin, Thomas Martin

Executive Summary

Unidata’s Python efforts continue to encompass: training on the use of Python for the community; development and maintenance of several tools for the community (most notably MetPy but also Siphon and data processing scripts); and participation within the broader scientific Python community. The training aspect continues to be busy with two workshops taught at the AMS Annual meeting. We also continue to plan additional courses along with supporting the 2023 Unidata Users Workshop. We are also furthering development of asynchronous training materials through Project Pythia, where we are working to migrate our existing workshop and gallery materials into a so-called “cookbook” within the broader project. MetPy development continues with the 1.4.0 feature release (largely based around “spherical calculus” support) and a recent 1.4.1 bugfix release. We are planning a 1.5 release at the end of April which will largely consist of work done in fulfillment of our NSF CSSI award. Participation within the broader community continues with anticipated attendance of SciPy and direct participation on the matplotlib, cartopy, and conda-forge projects. It has become increasingly difficult to dedicate time to these efforts given the full portfolio of responsibilities on the team.

Questions for Immediate Committee Feedback

Nothing at this time.

Activities Since the Last Status Report

Python Training Efforts

Continued collaboration of Unidata teams providing material development, instructional design practices, and Science Gateway infrastructure support have enabled multiple synchronous workshops at the turn of the year. After our Python Readiness course at CSU in the fall, we prepared two workshops for the 2023 AMS Annual Meeting: a day-long professional short course surveying unique weather data workflows in Python and a quarter-day intro to data basics in Python for the 2023 AMS Student Conference. We are supporting development for regional workshops and the Unidata Users Workshop this summer.

We continue to support Project Pythia’s Cookbook ecosystem, and the infrastructure and educational content it provides to the community. We directly engage with content development, infrastructure and maintenance, and community engagement. We are supporting the upcoming Pythia Summer Hackathon in Boulder, June 2023.  We are still transitioning our “Unidata Python Training” resource to its own Pythia Cookbook(s) with support from regular MetPy developers- and community-call participants.

Progress has been made on the following:

MetPy

Development continues to be driven by requirements for our dedicated awards (in addition to bug reports and pull requests from community members). MetPy 1.4.0 was released in late December 2022 (despite plans and efforts to release earlier) with a variety of fixes and enhancements including:

The MetPy team continues to work to increase the release cadence for the project, nominally trying to make a release every other month. This is aimed to avoid having releases slide to get “one more thing” in the release, and instead more readily get developments in the hands of the user–whenever they choose to upgrade. Success in doing so has been mixed, with the most recent push to get improved derivative calculations (aka “spherical support”) into the 1.4 release. MetPy 1.4.1 was released in early March 2023 with a few minor code and documentation fixes.

Moving forward, 1.5 is planned for release at the end of April 2023. This is planned to include the long-promised support for plotting fronts and analysis from the WPC. More broadly, we will also be continuing the performance improvement work that is the focus of the CSSI award, as well as include support for accessing data from the radar and satellite S3-based data archives.

To lower the barrier to project participation by community members, Unidata and regular participants in the existing “MetPy Dev Call” (biweekly on Google Meet, see MetPy GitHub discussion #2624) planned for the first “MetPy Community Call”. This community call is a synchronous meeting which shifts the discussion focus to open-floor demonstrations of successful (and unsuccessful) MetPy workflows, and removes the expectations that participants only contribute through “development”. The first call was advertised at AMS 2023 and delivered 19 Jan 2023 to around eight attendees, including five new in the space. This call occurs approximately monthly.

Progress has been made on the following:

Siphon and Data Processing

Siphon continues to exist in a steady state–continued maintenance and use, but minimal feature advancement. Some of this is due to limited development resources being focused on MetPy’s needs; it is also due to limited pressing needs on the data access side. Largely, Siphon meets the needs we have identified for Python data access (that aren’t also already met by zarr, xarray, etc.). With that said, Siphon does remain an important part of the stack used by our training work, and by Unidata’s community of Python users in general. The most pressing developments we anticipate for Siphon are improvements to working with Siphon in interactive sessions, like the Jupyter notebook environment: improved catalog crawling interface, better string representations, and tab completion.The decision has been made to separate non-TDS functionality (e.g. Wyoming Upper Air archive access) out into a new remote-access toolset contained within MetPy, and we hope to begin this transition work soon.

We also continue to maintain the LDM Alchemy repository as a collection of LDM processing scripts in Python. Currently this includes the code powering the AWS NEXRAD archive as well as the program that reconstitutes NOAAPORT GOES-16/17 imagery. As we transition more of our internal data processing to Python, this repository will hold those scripts. We have seen several community questions regarding both the GOES and NEXRAD processing software.

External Participation

The Python team attends conferences as well as participates in other projects within the scientific Python ecosystem. This allows us to stay informed and to be able to advocate for our community, as well as keep our community updated on developments. As participants in a broader Open Source software ecosystem, the Python team regularly encounters issues in other projects relevant to our community’s needs. As such, we routinely engage these projects to address challenges and submit fixes. We also continue to host Jeff Whittaker’s netCDF4-python project repository; Jeff continues to be the active maintainer of the project. The overall involvement helps ensure that important portions of our community’s Python stack remain well-supported. Ryan May continues to serve as a core developer for CartoPy as well as a member of Matplotlib’s Steering Council and conda-forge’s core team. It should be noted, though, that it’s becoming increasingly difficult to dedicate time to these efforts given the full portfolio of responsibilities on the team.

Progress has been made on the following: 

Ongoing Activities

We plan to continue the following activities:

New Activities

Over the next three months, we plan to organize or take part in the following:

Over the next twelve months, we plan to organize or take part in the following:

Relevant Metrics

MetPy

Siphon

Strategic Focus Areas

We support the following goals described in Unidata Strategic Plan:

  1. Providing Useful Tools
    Python has become a key tool in the atmospheric sciences, and the geosciences in general. MetPy leverages the rest of the scientific Python ecosystem to provide a suite of documented and tested domain-specific functionality, supporting greater use of Python by the community. Siphon serves to provide access to the growing collection of remote data sets. Together, MetPy and Siphon give the community a platform for scripted analysis of real-time and archived weather data. These tools are also readily used in the Jupyter Lab/Notebook environment, for ease of use in cloud and HPC computing environments, facilitating data-proximate analysis. We also participate in a variety of projects in the broader scientific Python ecosystem, to help ensure the ecosystem’s viability and that it continues to meet our community’s needs.
  2. Supporting People
    We provide a variety of online training resources to facilitate our community’s education and use of Python. We also regularly conduct training workshops to teach attendees how to use tools and apply them to their problems and challenges in research and education.

Prepared October 2022