Russ Schumacher (Chair), Colorado State University
Sen Chao, San Jose State University
Wendylynn Flynn, University of Northern Colorado
Victor Gensini, College of DuPage
Kevin Goebbert, Valparaiso University
Daryl Herzmann, Iowa State University
Gretchen Mullendore, University of North Dakota
Warren Pettee, University of North Carolina (Student Representative)
Pete Pokrandt, University of Wisconsin
Becky Cosgrove, NOAA
Rich Signell, USGS
Mohan Ramamurthy (Director)
Terry Mitchell-Sur
Ethan Davis
Josh Young
Mike Schmidt
Matt Perna
Ryan May
Julien Chastang
Tom Yoksas
Steve Emerson
Doug Dirks
Sheri Ruscetta
Dennis Heimbigner
Christian Ward-Garrison
Michael James
Yuan Ho
Larissa Gordon
Sean Arms
Jennifer Oxelson Ganter
Jeff Weber
Mohan Ramamurthy and Russ Schumacher
Spring SAC meeting started development of the next Unidata Strategic Plan; this will also be the focus of the next SAC meeting.
Mohan Ramamurthy
(Mohan’s presentation)
**Question:** What is the timing for the Strategic Plan and the next proposal.
**Response:** the Strategic Plan should be done by the end of 2016. The proposal will need to go in 2018.
**Question:** So 1 TB in per day and 30TB out. That is good right?
**Answer:** It is good and bad because there is more dependence on Unidata infrastructure because universities are stepping away from serving as a top tier node. This creates demand for our services; however, there are continuity of operations concerns. We have backups at Mesa campus but are looking into having something at Wyoming, as well.
AWIPS show of hands for installs: 5 universities.
**Question:** Anyone here actively using AWIPS?
**Response:** Yes, especially when EDEX is available in the cloud; the challenge has been when EDEX has to be run locally.
Question raised about the date for the TDS 5 release, to which Sean Arms noted that THREDDS 5.0 beta will be released in November with final release to be determined by beta performance.
**Question:** If training attendees are government can you make the case that that is still benefitting the academic community.
**Response:** We try to make that argument but NSF has to agree with that path.
**Question:** What is NSF thinking about EarthCube and have things changed now that it is here?
**Response:** This is going from phase one to phase two moving from governance to implementation. Over 40 grants have been awarded in the last two years and in this new phase the office is working with a group of individuals to develop the architecture and the implementation of that architecture. NSF has committed to fund EarthCube at least for the next three years. Beyond three years, the EarthCube program officers are working within the next NSF cyberinfrastructure initiative.
NOAAPort SBN feed naming conventions are ambiguous, for example the MRMS product is very unclear without documentation.
__Dennis Heimbinger:__ Can this be intelligently renamed? Victor responded possibly; is this at the NCEP level or LDM level?
__Tom Yoksas:__ Product labels come from GEMPAK tables, we don’t create the GRIB tables so if they are wrong we can’t update that table. We monitor the GRIB tables online and update based on changes; however, if things are wrong we don’t know it can’t be corrected.
Becky will take Victor’s issue to the MRMS folks, 161 is OAR who owns MRMS when NCEP took on IDP that became an issue.
__Victor Gensini:__ What bands and capacity for GOES-R?
__Tom Yoksas:__ We are installing a GOES-R facility but not going to push it out through IDD so you will have to come pull from our ADDE server. If launch is November 4th there will be a 60 day shake-out which we will make available immediately (with disclaimers).
__Victor Gensini:__ We have been working on Python stuff with GFS and allow more user interaction with site using python and SimpleKML. Can share code if requested. We are using OpenLayers as a GIS framework; plotly.js is useful for working through the browser interface. No libraries to install or language learning curve. Dygraphs is really useful. SimuAWIPS is an AWIPS browser package that allows a lot of usability, developed in Norman, OK.
__Ryan May:__ Plotly does have an API so you can use it in python notebooks.
__Victor Gensini:__ Each SimuAWIPS gets a login and then make selections; however, it is skinned to look at AWIPS without providing the ability to interrogate the data.
__Rich Signell:__ OpenLayers in the notebook use folium which is a wraparound leafletJS.
Working recently on cloud-based data processing especially Amazon cloud, mostly focused on NEXRAD data sets both for research and operational procedures. Looking at cloud solutions for operational scalability during events. We have a forecasting product that we would like to make fully available in the cloud (modeling, integration, products). 10 years ago we had Hydro-Nexrad and now we would like to redesign the system to bring the data into a custom system creating your own products and algorithms. On WRF Hydro we are creating our own forecasts for Iowa. We are comparing our models with the National Water Center’s product.
DMRC was used for a new grant submission recently. Made the move to Python 3 this summer and discovered that SharPy is stuck in the 2s. Also instituted the use of GitHub in the computer applications class. Python will be used in the climatology class next semester. AWIPS is getting closer to be really running; it is fully installed in lab systems just need a solid state server for EDEX. Have run into difficulties keeping the LDM up to date. Want to get a TDS up and running.
Not upgraded to 3.0 Python, most students have received Python training through classes. One challenge is that they tend to go with Python but not notebooks. Tutorials for Python notebooks would be very helpful. Involved in a lot of smart cities projects and trying to bring these Unidata resources to these efforts.
UNCC is largely web based for classes. Everything but computer applications class is web based. No new data stuff b/c mainly focused on research. Students do emergency weather forecasting. The IDV came up as something with bugs and so the question was are we developing too many new features and not focusing on major releases.
NEXRAD is being used by at least one student at CSU. Has a NASA project with Brian Mapes that runs into conflict with remote data access and login required in August to all NASA projects. Passwords in NASA systems required significant tweaking to process. Big Weather Web project has now grown to a related NOAA project with a CSU grad student working on post processing and machine learning. However, what you need for that type of work is a long run of historic forecasts. HRRR datasets are valuable to this work but that requires someone to archive them long term.
Since at UNC trying to modernize the program. LDM broke shortly after arrival and limited IT support. Modernized our computer labs this summer dual boot for both windows and CentOS. Trying to figure out the best way forward. A lot of my classes have moved to IDV. Trying to get our LDM functional, it works but we don’t have enough disk space. A student on a directed project this summer came up with a decoder for archived sounding data which will be polished up and put on github. Would love to do AWIPS but trying to figure out what is the best route for us.
Sean and Ryan came to Madison in June to present a workshop and it was really well received. These workshops are available on YouTube. Tried for 6 months to get a TDS up and running, Sean did it in a 90 minutes. We initially did this for in-house workshop use but we would like to open it up as a community resource. We and SSEC both submitted grants and the plan was to get our CAVE to interact with their EDEX; however, long term error in getting those to connect and talk. Jordan Gerth can answer why SSEC uses the NOAA AWIPS release instead of the Unidata version.
The award from UPC was to upgrade an EDEX server; they should be running UPC version. There is a AWIPS site with documentation.
NTArchive serves a lot of old GEMPAK archives; central IT said it had to be centralized. Who here has trouble with I/O and XFS? Talked to XFS developer at Redhat, found a setting that makes the LDM scour much quicker. Daryl will email it information to committee. Lost lightning data for a while, the multiplicity value is always 0 now. Anyone having issue with GEMPAK and the NLDN files? When they went to 5 minute files. Learned how to get OOM killer to not stop LDM, found a CRON script to prevent LDM from getting killed will email out.
Proposed getting long-term data set at Rutgers that we wanted to get into netCDF. Rosetta made that very possible. Will install Windows version of AWIPS in our lab.
Ioos.us has a lot of interesting tools. Going to Santa Cruz and getting our office up there on TDS and we are really excited about TDS 5 and the ability to use triangle based grids. Working with the Australians on TeriaGS as a framework for web portals that are geospatial. We needed dynamic WMS and that is coming out in a week.
Teach a class where students are getting raw data and then derive something out of it. There is a lot of push-back that UND should have web based products and not use resources from other universities. We need an assessment whether it is bad having students using web-based resources. Has a colleague that won’t allow students to use any resource from any other school. Maybe we could discuss during the Blue Skies Session. How can we work on this together. UND is using Python more classrooms with C still being taught first. We feel the need to start with data types which is better in C. Still using Python 2. AWIPS not working but we are trying to get our new hardware up to speed. Cloud processing is now a part of the Big Weather Web. There are knowledge based issues with cloud but we are also doing an academic based assessment to determine the cost. We are also running into security and policy issues. Hopefully, we can share all of these reports back to the group. We have both state and university policies that may trip up cloud use. Lastly, NSF has a graduate training award that you can put in a proposal in November, Gretchen had previously been willing to lead the charge but is willing to hand over the rein.
Available here
**Committee Question:** What about the tension between providing support and continuing development.
**UPC Response:** Long-standing tension that has positives and negatives; it provides feedback from the community and helps us understand their use case; it does create challenges in getting things done but support does provide stronger ties to the community. The only real answer is to have more people providing support. GEMBUD often has community responses to questions.
**Committee Question:** How much is installation support vs one-off issues? I.e. will “dockering” products reduce installation support.
**UPC Response:** Perhaps for some products but not necessarily all (e.g. netcdf)
**Comment:** UND cannot install docker on a lot of machines (university computers) because of concern about manipulating operating systems. Policy/Security concerns prevent using some of these resources; having Unidata speak sys admin/IT to university IT is able to resolve some of these issues.
**Comment:** IDV report included section on Java 3D and the link there hadn’t been updated in year. Is Docker a solution there?
**UPC Response:** Docker is not the answer because Java3D is not continuing development.
**Committee Question:** Cloud computing has a lot going on especially at Unidata but it is not really featured on the website.
**UPC Response:** The cloud projects have not been featured as much sense they are still under development.
**Committee Comment:** Perhaps just create a central place for the existing blog posts.
**UPC Response:** Noted as an action.
**Committee Question:** What is meant in the THREDDS upload/download?
**UPC Response:** It allows you to fairly easily upload products; however, this upload feature has not gotten much use so it hasn’t received a lot of attention. Download allows you to select a directory that then puts the results in that file.
TDS downloads should be mined for participation in the TDS group.
**Comment:** Folks are running TDS 4.67 which hasn’t been released.
**Committee Question:** Is there a good accounting of what is available?
**UPC Response:** Data streams under our control are well documented but those from NOAA are not well documented and we don’t know. CONDUIT is well documented.
**Request:** Updated information on GRIB files that aren’t identified.
**Comment:** AWIPS NCP is preferred, graphical forecast area would be valuable, yes folks are aware, a long term AWIPS archive would be very cool including case studies just as data without curriculum.
Ryan May
Ryan focused on things he thinks the committee should be aware of, even though they are not happening at the UPC.
**Matplotlib 2.0:** Color changes, stylistic/presentation features (better figures out of the box) See a list of changes.
**Plotly:** See this site.
**Python 2 vs 3:** Several changes to be aware of:
**JupyterLab:** Deploy notebooks as web apps
**Question:** Does dashboard become a part of Jupyter notebooks?
**Response:** Not necessarily part of but related.
**Comment:** Users Committee finds strong interest in Jupyter dashboard.
**Question:** A lot happening in RADAR, how much are you guys involved?
**Response:** Ryan May has talked to PyART folks a lot but not aware of NCAR environment thing.
**Question:** Python for AWIPS?
**Response:** Python and AWIPS combined would be very valuable. Having one LDM bringing in data that python then feeds to AWIPS and GEMPAK.
Value in regional workshops but webinars also work well for regions like UND.
Becky Cosgrove
(Becky’s presentation)
Julien Chastang
(Julien’s presentation)
**Comment:** We use the free public dockerhub version
__Rich Signell:__ Trying to get IOOS community to use the docker TDS version because when they had to update it prompted all kind of Tomcat updates.
__Steve Decker:__ Do you lose any performance cost to using docker?
RESPONSE: You have a very light penalty. The overhead is negligible.
__Rich Signell:__ Docker is great for testing different versions (4.6 TDS vs 5.0TDS)
__Mohan Ramamurthy:__ UPC believes this is a very good approach going forward but as Julien Chastang noted there is a lot of training and communication that must be done.
__Rich Signell:__ As Gretchen noted there is still security issues or at least concerns/rumors.
__Gretchen Mullendore:__ In the long run Jetstream seems like a good option but right now private sector toolbox is more mature. Prototype in Azure then deploy Jetstream.
Daryl Herzmann is the 2016 DeSouza award winner. You can see his talk on the Seminar Series page.
Josh Young/Mohan Ramamurthy
Jeff Weber
Josh Young/Larissa Gordon
Primary update is that continued development will be contingent upon funding availability. However, existing work may assist the community and is available at the DMRC page and the online Python training page.
Michael James
__Warren Pettee:__ What is EDEX’s role if things move to the cloud? MiJa: I don’t know of anything called AWIPS 3; however, moving to cloud based storage NWS would use EDEX for that function. I’d be interested in exploring EDEXs ingestion of Amazon data.
No suggestions, excited about cloud stuff.
API for javascript, we have netcdf-Java and netcdf Python, but we don’t have a good way to deliver Unidata stuff to a browser. Maybe something like protobuff. There are really cool things happening in the browser and we need to get TDS into the browser. Maybe that would be a good EarthCube building block proposal.
Imagine a day where EarthCube, OOI, IOOS, and Unidata are all working on the same infrastructure.
Someday use MetPy instead of GEMPAK. MetPy finished product looks like GEMPAK but it takes more code to generate (100 lines) with a lot being boilerplate that could be abstracted. MetPy could gain ease-of-use.
Revamp RTStats, should be refreshed; API’s to get stats back from the server. Then mine the metadata from the NOAAPort feed and allow folks to search it. Excited about cloud and Python stuff. Seaborne sitting on matplotlib abstracts a lot of things maybe something equivalent in MetPy
More time, but would like to get into AWIPS documentation and pursue cloud educational grants to play with and get some experience.
Not every institution and every student doesn’t need to make their own plots. I would like to take that further and suggest that not every institution needs its own EDEX server. I would really like having EDEX in the cloud (appropriate scale for smaller institutions) so that we are just running the CAVE client. Available today in Azure and soon in Amazon; we could use your feedback.
__Gretchen Mullendore:__ Thinking ahead the whole cloud funding model looks different. Universities may be able to ‘gift’ credits to Unidata to cover the service cost.
__Mohan Ramamurthy:__ Another option is Universities setting up a private cloud/Amazon S3 software is open and available.
__Russ Schumacher:__ Some universities also have large contracts with Google or Amazon; how do those work? Can you set up servers under those agreements.
__Warren Pettee:__ Anything at UNCC without PII is offloaded onto vendor storage.
Like Pete, I feel like I need to learn a million things. Trying to move from deterministic to probabilistic forecasts, with a lot of things Victor was showing fit into that approach. Interfaces straight from data into a browser would be pretty powerful.
Killing model page, try to help out with the philosophy of cloud based resources, and change the culture from a hard drive in my control to it will be ok out there in the world.
One day Unidata can be a one stop shop; where Unidata can provide anything we are looking for so that I can just point students there. There is a gap between the data and the end user. I’m working with a lot of engineers that don’t know how to find this data.
Emphasize some of the web technologies that are getting more and more capable almost comparable to desktop systems. Client side applications can now access multi-cpu processes. We are looking for some GPU based analysis and viz; such as solving climate models. We have prototypes that will allow folks to volunteer their machines for distributed computing. We can solve large scale hydro models through these types of distributed volunteer efforts. We can run these hydro models almost on an operational mode using this approach. We designed a couple of systems with intelligent architecture that will improve the access of non-technical users by providing language recognition (English) to automate providing products. (NOTE ASK IBDE for presentation). Allows access through Skype and chat. Siri is now open up so we are looking into that as well. Interface is almost invisible.
Spring 2018 class should be taught in Python and not GEMPAK. I want to no longer bring data into Valpo. The prospect of GOES-R is just mind boggling.
Appreciate all of the comments on the website, as I mentioned dream is data analysis and viz in the browser/cloud. AWIPS is great for NWS, education, or maybe research but not the lay person.
**Question:** Where do we stand on WRF initialization?
**Response:** Big Weather Web is working on that by dockerizing WRF. Current effort would still be limited to vtables and GRIB. Second part is getting netcdf access. The Developmental Test Bed at NCAR is also interested and they could work with NCAR to do that but they have to show to NOAA that there is value/demand for that task. ACTION NOTED.
Meeting Adjourned