Mohan Ramamurthy and Russ Schumacher
As part of the recent Strategic Advisory Committee, members and staff were able to meet with many of the AGS program officers and then had a specific briefing from Anjuli Bamzai about the Harnessing the Data Revolution initiative; Bernard Grant also provided a budget briefing. A good bit of the conversation was around the next proposal with the submission expected in September; the focus will be on moving in the direction of cloud-based workflows that support data-proximate analysis; the NOAA presentation from Jeff De La Beaujardiere mentioned the long term possibility of charging users for data egress after a predetermined threshold of free egress.
Education, training, and workforce development was a major conversation as well; specifically, about how we need to evolve to meet the new paradigm.
Mohan Ramamurthy
Tom Yoksas
Tom reported that the 2.5 degree data is no longer available, but .5 degree data is available; however, to get access you are supposed to write a letter to the standing representative to the WMO (which is Louis Uccellini) . Tom proposed following the Albany model whereby folks wo receive the data would agree not to distribute it; the current thinking is that Unidata Program Center (UPC) would be the point of contact. NOAA’s preference would be to have Unidata as the manager of community access. UPC would like to be very transparent that the intent is to relay this data and identify which things have European Union Resolution 40 restrictions.
Victor Gensini noted that some private sector companies are making visualizations available based on this data for free. Tom Yoksas added that visualization has always been treated as freely available; however, the underlying data is not.
UPC has been granted access to all of the GTS data and relay everything through the IDD and the WMO Res 40 restricted stuff to .edu. You have to identify the WMO products that you want instead of just requesting the GTS feed. UPC would like NOAA to provide the product lists to inform the community.
Separately, Don Murray at GSD offered a feed of wind profiler data from a west coast network with the data in BUFR. In AWIPS, this should work already.
There is also GEMPAK users path for access to current GOES-R data. Michael James has added support for fixed grid projection used in GOES-R. GEMPAK will not be updated to view this format; however, there is a workflow to convert it in a GEMPAK readable format (McIDAS area file). UPC proposes to take the full disc images in NOAAPORT (subset 6 degree) and take the CONUS and GRB and send those out. The question is what is desired by the community.
Victor Gensini suggests channel 2 and 9, 13 at CONUS resolution at 5 minute intervals?
UPC staff acked a clarifying question on which data feed: UniWisc or NImage? Committee has no preference on feed just notify where it is available.
Pete Pokrandt: Interest in GDAS .25 degree; NCEP has it available for about a week; however, no one is archiving it. I would like the zero hour to come through.
NCEI no longer archives NARR data though Victor Gensini does have that as an archive. Potentially, this could be hosted by CISL RDA where they already have the data; however, it is not available on THREDDS.
UniWisc SSEC has always made access available to their imagery; they have now made it easier for non-McIDAS users to get this imagery with a new service called McFetch. It does require a sign up because sites are limited to 1gb per day and the products are limited to 6 month old imagery.
Transitioned to NIU and now fully Python 3, worked with Ryan May and John Leeman on the CyberTraining proposal, sending Sean Arms an email on the NARR TDS. Still involved in the GOES 16 page at the College of DuPage. The GOES 16 page only exists because of the Unidata Equipment Grant. GOES 17 is the big thing on the horizon from my perspective. -
USGS is very interested in cloud based workflows. The JetStream stuff has been great. USGS has a requirement to use AWS; however, I’ve been using the the existing caveat that USGS employees may access a collaborators cloud space. Also big unstructured grids (triangular grids); TDS 5.0 includes support for triangular grids; however, it was designed for small unstructured grids so now we are looking back at the python based stuff (e.g. datashader). It is exciting to look at what is happening in the Python world.
Excited to get the docker TDS server that University of Oklahoma is hosting; trying to get folks up to speed with the CAVE.
Busy working on MS research and encouraging more folks to use MetPY for soundings or other data. Mostly trying to get other folks using it and using as a teaching assistant.
Working with Ryan and John to advance the MetPY award. We have been meeting every two weeks and are looking at opening those calls to anyone that is interested. Have also been making MetPy examples to contribute back. Did co-teach the course at AMS and have now fully implemented Jupyter notebooks for synoptic analysis. Gave a talk at AMS about implementing Jupyter notebooks for a numerical weather prediction class. Hoping to have TDS up for folks that are interested in NW Indiana and Chicago radar data. Victor noted that NASA has released a tool called Multidop to produce dual doppler radar analysis.
Since taking over as the State Climatologist I haven’t had much time to do research. I have been using the Unidata Jetstream cloud to consolidate and share model runs. Working with Kevin Tyle to make these outputs and analysis codes available. As a user this doesn’t look any different than any other user terminal. MetPy team is going to come up to CSU to do a tutorial. We have a lot of python users though I’m not sure how many folks are using MetPY; hoping that this session will help make that transition.
Mesooscale class always runs a cloud model; I did put it in docker and the class runs it on their own machines. All you need is a fortran compiler if you can work with a binary output; however, everyone ones netcdf. The storage server that we bought with a Unidata Equipment Grant in 2012 failed; however, the students applied for University funding and received an award above their request so we are getting a new server. We didn’t lose much that was irreplaceable.
IT staff have been stealing sudo privileges; we have been developing a new notebooks for thermodynamics and climate dynamics which are based on MetPY and then applying it to other datasets. Generally headed towards Python based tools. I have been interacting with the Pangea project folks at Columbia to find a better way to optimize calculations for thermodynamics. We did try using the basic MetPy functions; but superseded there capability.
We have this collaboration with NDC radar that can generate netcdf. The radar produced netcdf files for an hour then it stopped. This semester I will be co-teaching a computational methods for meteorology and using metpy for the first time in that course. Trying to go cold turkey and not using any Gempak next Fall. Kevin Goebbert noted that from his experience the biggest hang up will probably be point data.
The University of Wisconsin is hiring three faculty, after losing 2 oceanographers. In terms of software for classes; 1 professor is still a 100% gempak, another is completely python and now using in a 2nd course in dynamics; keep dabbling in GOES data; One of the mesoscale sectors is typically over southern Wisconsin and so have been plotting full visible. Worth noting that Daryl Hertzman was looking for some data from 1998 and I was able to pull stuff from Exabyte tapes going back to 1996. Working to pull data from tapes and push to Daryl who is hosting them.
Up for tenure this year; submitted my dossier and have heard good things. Definitely getting into the groove with teaching and if I receive tenure I plan to get up to speed on MetPY. I’ve used it a bit for class. Next fall I’m teaching radar and satellite meteorology and look forward to hearing more about what I can use in Jetstream. Really excited to see the movement in MetPy.
Full reports are available here.
Some highlights of the discussion:
Rich Signell, Ryan May, Julien Chastang
The idea is to have a zero install option; we’ve had a number of alpha testers since the last meeting. I was hoping to have a quick round table (Kevin Goebbert, Shaun Riley, Pete Pokrandt)
Josh Young
Josh noted to recent encounters with members of the community that for two distinct reasons did not feel well suited to serving on a Unidata committee. The two scenarios identified included: 1) not identifying or trained in atmospheric sciences; however, teaching atmospheric or Earth system science courses, 2) an atmospheric science professor that does not directly rely on Unidata tools. In discussion, the committee felt that we could always do a better a job of reaching out to community members; however, we have about the right balance between those that could use Unidata tools and those that actually do. Members did feel it would make sense to better integrate the climate community.
Sean Arms
The HRRR server is using 5.0 as well as siphon is 5.0. If you go through the web interface the icons update, the subset pages look better. Also incorporated timeleaf as a templating system. 5.0 will look like a standard java application. Also moved from ncwms1 to ncwms2. There has been some bumpyness moving to ncwms2 which RiSi has largely addressed. The other TDS 5.0 difference is that when you are using the subset service it used to make a difference between grids and points then we went to just having a subset service. This was a silent change that broke a lot of scripts for Daryl so in 5.0 those will be different services on our end. With the subset service you will be able to make a request that wraps the array bounds for example you can span the international data boundary. The biggest change is a feature called catalog scan that allows it to detect catalogue changes and then propagate it out.
Carissa Klemmer
Staff noted that a table for WMO headers is what is needed. Carissa Klemmer responded that unfortunately, that is a lot of information. Carissa noted that they should be able to share the Regular expressions for WMO headers; however, she will have to get permission to make that public. Russ Schumacher asked how we can keep track of what happened with a major outage. News42 is an admin message from.
Michael James
John Allen, Kevin Goebbert, Victor Gensini, Josh Young
Organizers noted the lack of diversity on the speaker list and requested suggestions for potential speakers. Recommendations included:
Josh Young
The committee chose the 2018 DeSouza honoree, who will be notified by Kevin Goebbert.
I would like to do as much in the cloud as I possibly can and in some ways I’m making that change because I want to interact with my campus IT as little as possible. The money that we currently use for campus IT staff would go towards cloud providers. What I need is to have a cloud environment with everything set up, I would also like MetPY to have everything for synoptic analysis. Tenure will hopefully be resolved soon. At UNC there are not people that do Linux so it is on us anyway.
I hope there is more convergence around things like Pangeo and that Unidata stays active in that sort of stuff. I think part of it is effectively storing stuff in the cloud and as Dennis said yesterday Unidata has a big role to play in influencing how things like ZARR move forward. It would be great to broaden participation in the Unidata conversations about which service/tool to embrace. If you look at the Pangeo team it is a really impressive group and our community could really benefit from using that service so we could do a lot by sharing that resource and expanding it. Hopefully, Unidata receives the IOOS award to do some unstructured grid stuff.
Looking at doing a bit of a hybrid cloud just to ease the discomfort for some of the folks resistant to the cloud. We have some local clusters and Microsoft has offered us the ability to burst into the cloud when our servers are under a heavy load and then auto archiving data.
Trying to get folks to consolidate around using MetPy. There are people in my University that have heard of it but not looked into it. Others have not heard of it or others that have excellent code that they are not willing to share. Part of my website is MetPy and part of it is stuff that I’ve developed. Russ Schumacher noted that Tomer’s standardized anomaly calculation is really useful. Being able to do those calculations on the fly is really good. Tomer Burg responded that someone else has already produced those in our department but I’m working on Climatological Percentiles I think that would be really beneficial; however, those products take a lot of space on a server so perhaps we could put it on the cloud.
By January of next year, I hope to not teach Gempak anymore. The last task I need is transferring data files. Every time I go to make a map it gets easier and easier to do that. Hopefully, next January it will be Metpy and siphon based stuff. We also have an operational radar again and are getting a new server which means we can repurpose an old server to make that radar data available. I would also like to be able to serve other data once I have a TDS up and available for the community.
Working in government especially an operational shop; it is really difficult to grow and adopt new tech. I have to coordinate with so many people to make any change. My hope and dream is to make one thing a little better. My goal is to upgrade at least 10% of my system to run a current LDM version.
Reiterate some of things I’ve heard about transitioning from Gempak to Metpy and my goal is to also get students working in cloud workflows. I try to do stuff on the University cluster and get things back to my desktop for analysis.
One professor will never leave Gempak and one has already made the transition. Suggest adding GPend to MetPY (partly joke) and want to get more up to speed on AWIPS now that is more stable. A lot of my faculty are not likely to change what they are doing in their classes; however, with new faculty coming on we might get some folks using more python. Also have a grant into the university to get a new server to run jupyterhub. I would also like to learn more about docker since I haven’t used it. I would like to start writing python code that doesn’t look like fortran. Looking forward to GOES17 and having more opportunities to play with new data.
Hoping to get an IT person that is Met specific and competent. Getting AWIPS running locally has taken a year because of our campus IT and hurdles they create. Personally would like to move from Python 2.7 to whatever is the current 3 version. I would like connection parameter calculations and this is something that I’m working with Pangeo on.
Victor Gensini: The skies are cloudy in Illinois until the 2018 midterms. Tomorrow is the last day for Gilbert Sobeski; who has got me here. Our infrastructure is falling apart (both lab and connectivity) so we will be submitting an equipment award next year. I’m also excited about the opportunities through Jetstream or Pangeo. I think there is a lot we can do with Pangeo by putting climate and weather data next to each other. Gilbert was about 2 years away from retirement and he is still looking.
John Tealand is coming as a summer intern and I’m really excited to see what he does. Personally, the python stack has been very painful for me. The problems with Cartopy and shapely have been really problematic. The dream is to not have all of these bugs in the python stack. Ryan May noted that he can work with Daryl on these issues. Cartopy doesn’t not have a good understanding of how to make a polygon. Some of the issues I have in contouring are actually a matplotlib bug. When those line segments get to the borders the question is how do you handle that.
Success stories from 6 years ago, the NCAR ensemble was the closest thing to that but now it is gone. The SPC HREF website has some of that. Big Weather Web has also been a good project for moving in these directions though we have probably raised more problems then solved them. The ubiquitous weather apps do this behind the scenes. We have made progress but not reached the blue sky yet. For me now I’ll be focusing on the State Climatologist issues and challenges. They have a lot of cool tools for their questions though there might be some opportunities to incorporate Unidata tools. Looking at how to advance some of their tools. When I came on to the Users Committee I sort of knew what Unidata does but I had a really limited perception of the variety of things that happen at Unidata. We are only going to need more help from Unidata as the data volumes increase. The need for how to work with data is only going to become a bigger issue. In the community we have great confidence that when Unidata takes up a project that it will be done in the right way.
Getting the users to do more to benefit the community would be a great blue sky. There is a lot of expertise in the community and if we can use some software carpentry techniques to get this knowledge shared by high level users. Maybe these are local workshops run as Unidata Community Workshops that are taught on the ground by power users. Perhaps University folks take the lead on more education stuff.
When Daryl was talking about struggling with CondaForge; in my community there is a lot of frustration there as well. We maintain a list of where things are current and I tell people to go install things again because of a change. There is an IOOS guy who spends half of his time doing CondaForge. CondaForge and Anaconda are not in total agreement which creates pain for users. There is so much active development on so many of these tools so that creates a lot of churn. In the IOOS community there are a lot of notebooks and so we have to maintain this giant environment that can run all of these notebooks.
Russ Schumacher requested to transfer the role of chair during the Spring meeting due to his new role as state climatologist. Kevin Goebbert agreed to serve as the next Chair and has now been invested with the role.