Weekly Meeting Minutes

https://columbiauniversity.zoom.us/j/95352725

2021-04-14 2000 GMT (1pm PDT, 4pm EDT)

Pangeo Showcase Talk:

“The New Qhub 0.3: An easy way to deploy/maintain JupyterHub with Dask Gateway”, Lais Carvalho, QuanSight

https://pangeo.io/pangeo-showcase.html

Discussion

Check-In Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Martin Durant / Anaconda / @martindurant
• Dharhas Pothina / Quansight / @dharhas
• Jim Bednar / Anaconda / @jbednar
• Tom Augspurger / Microsoft / @TomAugspurger
• Orhan Eroglu / NCAR / @erogluorhan
• Alex Kerney / Gulf of Maine Research Institute / @abkfenris
• Dave Hoese / SSEC / @djhoese
• Patrick Tripp / RPS Group / @patrick-tripp
• Adam Kemberling / Gulf of Maine Research Institute / @adamkemberling
• Patrick Gray / Duke Marine Lab / @patrickcgray
• Chris Johnson / Vaisala / @point9repeating
• Chelle Gentemann / Farallon Institute / @cgentemann

60 Second Updates:

• Rich:  Watched Tom’s awesome talk on the Planetary Computer,   AGU Workshop proposal going in today, ESIP session proposals due end of week,
• Dave: Read open file-like objects with rasterio: https://github.com/mapbox/rasterio/pull/2141
• Tom:
• Martin: https://nbviewer.jupyter.org/gist/rsignell-usgs/550ade1fb9fed294d709b713d7324254 
• Chelle: AGU Fall 2021, special session with Fernando,  Paige and others full day session
• Chelle (again): oops forgot to ask a question

• Matt again: Write Short Blogposts (matthewrocklin.com) :)

• Bit twiddling example: Large SVDs (dask.org)
• Chris: working on visualizing zarr, have a NAM example going, looking for other examples

Agenda:

[item]

2021-04-07 1600 GMT (9am PDT, 12pm EDT)

Pangeo Showcase Talk:
“Parallel Analysis Using Pangeo vs the Command-Line”, Seth McGinnis

Discussion

Check-in Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Tom Nicholas / Columbia / @TomNicholas
• John Clyne / NCAR / @clyne
• Jim Bednar / Anaconda / @jbednar
• Chiara Lepore/ LDEO / @chiaral
• Jarrod Lewis / AER / @lewisjarrod
• Chelle Gentemann / FI / @cgentemann
• Martin Durant / Anaconda / @martindurant
• Orhan Eroglu / NCAR / @erogluorhan
• Patrick Gray / Duke Marine Lab / @patrickcgray
• Peter Killick / UK Met Office / @DPeterK
• Ray Bell / DTN / @raybellwaves
• Alex Kerney / Gulf of Maine Research Institute / @abkfenris
• Wesley Banfield / CEREGE (CNRS)  / @WesleyTheGeolien
• Derek Rollend / JHU Applied Physics Lab / @derekrollend
• Alessandro Amici / B-Open / @alexamici
• Mackenzie Blanusa/UConn, GFDL/ @MackenzieBlanusa

60 Second Updates:

• Ryan
• Rich: First USGS Panel apps made public at https://bokeh-pangeo.usgs.gov . These are three simple forecast model viewing apps, including our own Coupled Ocean Atmosphere Wave and Sediment Transport (COAWST model).  The interactive nature of the viewer has already turned up some issues with our forecast!
• John: There is a grass roots Earth System Data Science (ESDS) initiative now underway at NCAR that aims to, among other things, bring geoscientists and software engineers together to help address analysis workflows in a Pangeo/Python environment. Hope to be able to report on progress later.

• Info on Pangeo Showcase here: https://pangeo.io/pangeo-showcase.html

• Peter: blogpost on metadata querying published:
  https://medium.com/informatics-lab/metadata-queries-eeb06c1c9e1e 
  OGC EDR API: https://github.com/opengeospatial/ogcapi-environmental-data-retrieval 
• Martin: fsspec, s3fs and gcsfs releases. Another reference-maker example now with napari here (with small update to spec). Fastparquet speedup.
• Alessandro: upcoming xarray backend API

Agenda:

• Chelle: AGU Fall meeting session proposals? https://www.agu.org/Fall-Meeting

• Pangeo tutorial?
• Open Cloud Computing Science Session?

2021-03-31 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Jim Bednar / Anaconda / @jbednar
• Tom Nicholas / Columbia / @TomNicholas
• Tom Augspurger / Microsoft / @TomAugspurger
• Max Grover / NCAR / @mgrover1
• Chris Johnson/ Vaisala / @point9repeating
• Richard Scott / OZ Minerals / @RichardScottOZ
• Martin durant / Anaconda / @martindurant

60 Second Updates:

• Rich:

• Reminder that we’re moving to a weekly showcase format.
• Scheduled out for 10ish weeks, but looking for more presenters.

• JimB:

• No big news. Interested in chatting about topics from last week, though!

• Thomas Nicholas

• Xarray developer. Just started up with Ryan.

• Tom A

• Would like to talk to folks doing Geospatial stuff in R
• Can push up an R image to pangeo-docker-images if people are interested (no RStudio)

• Max

• Working on populating NCAR ESDS site https://ncar.github.io/esds/.
• Tried out Xoak for first time last week for zonal averaging...

• Chris

• On renewable energy resource team.
• Lots of internal datasets, part of a project to move them to the cloud.
• Looking at forecast datasets, rolling, met office post.

• Richard

• Working with CSIRO on large-scale analysis.  New Australia-wide Sentinel 2 Bare Earth model coming - see below - how to work with that
• Post from Daryl Roberts on linkedin.  https://www.linkedin.com/posts/daleroberts_our-latest-sentinel-2-barest-earth-activity-6780267107142705152-IZEx
• How to feed the above models at many terabyte scale into neural networks
• DEA sandbox accounts - 16GB/2scores 13GB disk space - Australian Open Data Cube access for some datasets  https://docs.dea.ga.gov.au/setup/sandbox.html
• Stackstac did this on one machine (a big one): H:\>gdalinfo SA_sentinel_2020_1bandmedian2.tif
• Driver: GTiff/GeoTIFF
• Files: SA_sentinel_2020_1bandmedian2.tif
• Size is 120416, 135081
• Coordinate System is:
• BOUNDCRS[
•     SOURCECRS[
•         PROJCRS["GDA94 / Geoscience Australia Lambert",
•             BASEGEOGCRS["GDA94",

• Martin

• More async fsspec work.
• Dask Summit organization.
• Loading several files through fsspec-reference-maker. Some performance issues in xarray with multiple files. On the order of 100,000s of files.
• Thomas: xarray devs are looking into this area currently.

• Lucas

• Lurking / learning.

• Ryan

• Working on pangeo-forge. Push some datasets through it currently.
• Cloud Bakery side is picking up.
• GRIB data should work now.

Agenda:

• Pangeo Showcase

• Additional promotion to get the word out (beyond Twitter).
• Blog post:
  https://medium.com/pangeo/new-pangeo-showcase-seminar-series-d589bb6e77ad 
• Share on

• ESIP: Rich
• ROMS mailing list: Rich
• NCAR: Max
• AOSpy email list: Ryan
• MS AI4Earth: Tom

• Xarray accessing overviews from COG

• Using stackstac & stac-vrt.
• Wanted to use overviews, just to view on a map or something. Doesn’t really mesh with the xarray data model.
• Running on a cluster — stackstac documentation, passes in the overview level with the DataArray is constructed.
• There’s an EOSS proposal submitted yesterday for xarray to understand hierarchical datasets (and image pyramids / overviews would fit in that data model).
• ComputeCanada/magic_castle: Terraform modules to replicate a Compute Canada cluster in the cloud (github.com)

• Jim: followup from last week

2021-03-24 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Max Grover / NCAR / @mgrover1
• Peter Killick / UK Met Office / @DPeterK
• Rich Signell / USGS / @rsignell-usgs
• Julius Busecke / LDEO / @jbusecke
• Wesley Banfield / CEREGE / @WesleyTheGeolien
• Tom Augspurger / Microsoft / @TomAugspurger
• Jim Pivarski / Princeton / @jpivarski (no update from me: just keeping up-to-date on everyone else’s progress)
• Jeff Sadler / USGS / @jsadler2
• Jarrod Lewis / AER / @lewisjarrod
• Martin Durant / Anaconda / @martindurant
• Rowan Gaffney / USDA ARS / @rmg55
• Tasha Snow / Colorado School of Mines / @tsnow03
• Lucas Sterzinger / UC Davis / @lsterzinger
• Orhan Eroglu / NCAR / @erogluorhan
• John Clyne / NCAR / @clyne

60 Second Updates:

• Ryan
• Julius

• Paper Polishing…
• Helping out over at GliderTools
• CDSLab Hack

• 3D Visualization tools, still quite frustrating… ->  Julius had a quick go at doing xarray to ipygany here  but very basic xarray https://cerege-cl.github.io/Notebooks-Gallery/notebooks/TopoBathy-3 D-color+warp-slider.html 
• Are there recommended packages? There are so many different ones.
• Dependency issues for Pangeo Cloud Issue 
• Ryan: wants to do 3d viz in JS, bypassing python
  https://deck.gl/ https://loaders.gl/docs 
          Rich: For JS viz, what parts of the workflow do you do remotely/serverless, and which parts on locally/GPU (thinking
• JimB: In addition to Deck we’ve used CesiumJS at some point.  Happy to revisit nicely wrapping up JS 3D tools so that they can be generated easily from Python; haven’t previously had a customer interested in that enough to work on it.

• Rich

• Edited the Pangeo web site for first time (using vscode on Windows (!))
• There were conflicts with bib file in PR

• Rich -> Could Github actions be used to build on pushes? Wesley?

• Max

• Working on adding zonal averaging tool to pop-tools

• Custom indexer for xarray - soon have way to customize this
• https://github.com/xarray-contrib/xoak
• https://github.com/pydata/xarray/blob/master/design_notes/flexible_indexes_notes.md 

• Starting work-in-progress meetings in NCAR CGD, focused on discussing climate diagnostic workflows

• Wesley        

• Looking into using intake for creating catalog for all simulations stac also got mentioned (https://stacspec.org) by CMIP6 people -> any comments?
• Calculating sediment outflow per watershed basin flowing into marine basins
• Compiling fortran code inside docker containers to deploy to web containers for cloud compute
• https://lumen.holoviz.org. JimB: See new webinar recording:
  https://know.anaconda.com/FlexibleMonitoringApp-LP.html

• Tom

• Helped Appalachian Trail Conservancy during a hack week. Forest Inventory Analysis is my new favorite dataset for tabular data.
• Hosted call on STAC + Dask

• Martin

• Async refactor
• Fsspec-reference-maker for Spec1
• Dask IO workshop: suggest for zarr, tileDB, ...

• Jeff

• Aggregating gridded data to polygons for ML training data prep

• Checking out Xagg library
• Had my own implementation using a dot product on xarray with weight grid. This was very efficient with Dask.

• Jarrod:

• Looking into bringing radio occultation data to AWS via NASA ACCESS
• Maybe post on https://discourse.pangeo.io/c/data/13?

• Pete

• Metadata querying demo - blog post here:
  https://medium.com/informatics-lab/metadata-queries-eeb06c1c9e1e 

• Orhan

• GeoCAT-comp version 2021.03.0 released.

• Vertical interpolation of hybrid to pressure coords

• See GeoCAT blog for all updates and releases
• GeoCAT members helping out NCAR’s Python tutorials for public nowadays

• Today 1pm Mountain Time - matplotlib tutorial

Agenda:

• Pangeo Showcase (Rich)
• Raster Meeting (Tom)

• Meeting notes
• Demo notebook

2021-03-17 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Julius Busecke / LDEO / @jbusecke
• Martin Durant / Anaconda / @martindurant
• James Bourbeau / Coiled / @jrbourbeau
• Tom Augspurger / Microsoft / @TomAugspurger
• Max Grover / NCAR / @mgrover1
• Zac Flamig / AWS / @zflamig
• Chelle Gentemann / @cgentemann
• Richard Scott / @RichardScottOZ
• Deepak Cherian / @dcherian
• Scott Henderson

60 Second Updates:¡

• Rich:        

• Attended Philippe Rudigers webinar on Lumen. Dashboards from YAML.
• Discovered the pytide program (not utide)

• Julius:

• Working on a bunch of science

• Once the paper is done, i'll write a blogpost about the whole process

• Group Hack working with pyqg.
• Call with GliderTools community - They want this package to be the open source resource to work with glider data. Ocean gliders are autonomous things with some ability to steer through the ocean rather than just float. Glider community is a bit fragmented. Raw data is hard to use.
• There is a CLIVAR working group for ocean observation.
• Also: https://gliders.ioos.us/data/

• Martin:

• Update intake and intake-xarray deps, blocking on pip-check in https://github.com/conda-forge/intake-xarray-feedstock/pull/8
• Async refactor in fsspec and s3fs; releases to come (but do intend to pull out nesting); lots of fixes coming in to file systems because of dvc integration. Git-like way of sharing large files.
• Implementation of ReferenceFileSystem for spec 1 done; now, how do you make a pangeo-forge recipe? Reference maker now available for tifffile ( PR )

• James

• Hi Pangeo folks!
• James from Dask / Coiled. Checking in to say hi.
• Home | Dask Distributed Summit

• Tom

• Multiple projects around STAC + Dask. Organizing a call: https://github.com/pangeo-data/cog-best-practices/issues/4   (mostly EO data, not model data yet, but discussions underway).   Stac search returns item or items which might need to be handled (stac-vrt) stac-stack can help

• Max

• Hello - recently started at NCAR in the Climate and Global Dynamics division, working on efforts related to the Earth System Data Science initiative, contributing to a variety of projects including both NCAR specific and broader Pangeo projects.
• Pangeo Gallery — Pangeo Gallery documentation
• Mostly CESM model output data.

• Zac

• Testing fsspec hanging async stuff,  grib2 forecasts are showing up on AWS as NOAA restricts download, but hundreds of hourly grib2 files are a pain.  Rolling zarr is a possibility, tiledb? Tom would like to know if folks work on this.
• Interested in if anyone is doing GOES -> Zarr things.

• Chelle

• I’m doing science again. I didn’t forget everything. A bunch of pangeo people & I have paper accepted AGU advances.  Giving it in talk form at EGU in a couple weeks.  UN Ocean Decade trying to organize activities. Kevin NASA CDO. Pangeo gallery time-out. CCMP/NOAAOI. Joe
• Group of scientists advising government agencies on how to roll out cloud data. Not just earth data, astro / helio too. Want to make sure their strategy is doing what we want.
  

• Deepak:

• Thinking about distributed groupby with arrays. amazed by dask-dataframe.
• Aiming for a cf-xarray notebook at earthcube

• Richard Scott

• Still working on country-scale sentinel GM medians.
• Some data issues, have to work around.
• Downscaling pipeline, using 2D CNNs. Data size exploding for spectral unmixing, overlapping patching at billion scale to remove edge effects - Zarr/Dask?

• Scott Henderson

• Conversations and work around STAC + xarray, stacking / mosaicing.
• Organizing hackweeks at UW. Next is NASA’s SnowEx program. Working with 2i2c and Jupyter Book. Focus on single-node.
• Going to Australia for a couple months!

• Stef Nestor

• Saying hi! Background in math / statistics / geospatial / data security. Working at Elastic.

Agenda:

• Discussion on VRTs. Are they good to use? What about nested VRTs?
• Intake Search

• Any search mechanism, like what we have for STAC? There is a default search method that descends into catalogs and looks for text in the descriptions.
• Brookhaven is looking into this. Pushing search into MongoDB.
• Rich wanting to search on ISO metadata.

2021-03-10 1700 GMT (9am PST, 12pm EST)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• John Clyne / NCAR / @clyne
• Martin Durant / Anaconda / @martindurant
• Jim Bednar / Anaconda / @jbednar
• Alec Engell / NextGen Federal Systems
• Deepak Cherian / NCAR / @dcherian
• Tom Augspurger / Microsoft / @TomAugspurger

60 Second Updates:

• Ryan
• Rich

• Working slowly with Azure DAYMET zarr data.   Not sure why a North American dataset is hosted in West Europe Azure region.  30-50s to download each 240mb chunk.  A dask gateway would be nice to run there.  Would try it out, except qhub doesn’t work on azure yet.

• [from Tom]: All of AI4Earth’s data is in west europe. Reach out to Tom for access to compute and I’ll get you added to a list.

• WPS processing service on top of xarray to solve this use case: calculate a time series from a

• John

• Pangeo Showcase: We have four speakers scheduled for the Pangeo Showcase (15 minute virtual seminar), kicking off April 7 with Seth McGinnis from NCAR. Full schedule and sign up sheet is here. There is a draft announcement destined for the Pangeo blog here.

• Martin:

• Working on fsspec-reference-maker spec V1; recipe next to recreate “Ike”
• Fixes to gcsfs
• Conversation around cloud key-value directories ( issue )
• https://summit.dask.org/present/ 
• Streamz blog post

• JimB:

• Datashader 0.12.1 released to fix raster/regridding plots with xarray 0.17; only available on conda for now due to CI issues. Thanks to Rich et al for help and advice.
• If you have large-scale monitoring tasks to do, e.g. if you’re responsible for monitoring JH cluster/HPC system/Kubernetes resource usage, keeping sets of servers or dashboards running, keeping tabs on a bunch of CI jobs, etc., you may be interested in an upcoming webinar for our new Lumen declarative Panel-based dashboarding package on Wednesday March 17 at 2pm EDT/11am PDT:  https://know.anaconda.com/FlexibleMonitoringApp-LP.html

• Alec

• Find out next Friday for work on Xarray supporting WCS (world coord. sys) - primary use is astrophysics

• Must follow https://github.com/pydata/xarray/pull/4979 

• Trying to see if there is anything there in ORNL for Air Force Weather and Pangeo (HPCs and Hub)
• Pitching a Distributed Data Viz capability/enhancement to NextGen for some IRAD funding.
• DKIST

• Deepak
• Tom

Agenda:

• https://docs.dask.org/en/latest/graph_manipulation.html 

2021-03-03 2100 GMT (1pm PST, 4pm EST)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Martin Durant / Anaconda / @martindurant
• Tom Augspurger / Microsoft /  @TomAugspurger
• Jim Bednar / Anaconda / @jbednar
• Deborah Khider/ USC ISI /@khider
• Orhan Eroglu / NCAR / @erogluorhan
• Jeff Sadler / USGS / @jsadler2
• Scott Henderson / UW eScience / @scottyhq
• Kyle Haver / McKinsey & Company / @khaver
• Matthew Rocklin / Coiled / @mrocklin
• Alec Engell/ NextGen Federal Systems
• Richard Scott/OZ Minerals @RichardScottOZ

60 Second Updates:

• Rich:

• First access of data from Azure, using DAYMET data for national hydrologic model forcing.  
• Struggling with dask hanging with just a few jobs left.  Seems I’m not the only one who has encountered this. Is this the answer?  Exploring Qhub deployment at USGS.

• Martin:

• ironing out old bugs in gcsfs; dvc and jupyter-fs taking on fsspec
• working on reference-maker spec
• https://summit.dask.org/ online
• restarted blog martindurant.github.io/blog

• Tom:

• stac-vrt is open-source now: stac-vrt — stac-vrt 1.0.1 documentation (stac-vrt.readthedocs.io)  (builds VRTs on the fly from STAC)
• Pangeo workshop at dask summit Dask Summit 2021 - News & Announcements - Pangeo

• Jim:

• Bokeh 2.3 released, which as predicted broke everything HoloViz, but compatible packages should now be available as of yesterday.
• Xarray 0.17.0 also broke Datashader; wondering if there is a reliable way to access the “nodata” value for an Xarray file, so if want to use HoloViz and xarray right now, pin xarray before 0.17
• Full-time and intern jobs available in the HoloViz group: https://boards.greenhouse.io/anaconda/jobs/2947293
  https://boards.greenhouse.io/anaconda/jobs/2603131 

• Deborah:

• Ran a hackathon in collaboration with 2i2c aimed at paleoclimatologists (observationalists) with no to little experience with Python and no exposure to xarray.

• Jeff:

• relating gridded data (nldas or daymet) to polygons (watersheds). Have been using a weight grid and then doing a dot product. Wondering if there is a more efficient way to do this.

• Orhan:

• Will release a new version (v2021.03.0) of geocat-comp today

• Will give update probably next week

• Scott:

• Playing with JupyterBook
• Working with intake-stac

• Kyle:

• First timer. Senior engineer mckinsey, downscaling of climate data, hazard analysis

• Matt:

• Dask Summit in May, Pangeans very welcome!   Encouraging workshops, would be great to have a Pangeo workshop https://summit.dask.org, consider inviting students and postdocs

• Alec:

• Another first-timer!  EAPS background, working mostly on DOD projects, on air force weather, SMART machine learning soil moisture, using JH on AWS,

• Richard:

• Working on geology modeling et al.  Still considering commercial possibilities other than roll your own for country scale Sentinel modelling.  Large scale geospatial joining.

Agenda:

• Demos from Alec from Air Force SBIR:

• SMART: using HoloViz tools and Dask to explore various raster quantities in order to predict soil moisture, filling in data not yet available from measurements
• Space weather: identifying solar flares and related phenomena, again using HoloViz and Dask tools

• EOSDIS data format comparison

• Looks useful, but seems like it needs more / better-updated info
• Links added from Rich S:

• ESIP Cloud Computing Cluster monthly meeting notes
• ESIP Cloud data optimization knowledge base 
• Pangeo Discourse issue on creating a Cloud Data Cookbook

2021-02-24 2100 GMT (9am PST, 12pm EST)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Deepak Cherian / NCAR / @dcherian
• Riley Brady / McKinsey & Company / @bradyrx
• Tom Augspurger / Microsoft / @TomAugspurger
• Jim Bednar / Anaconda / @jbednar
• Wesley Banfield / CEREGE CNRS / @WesleyTheGeolien
• Orhan Eroglu / NCAR / @erogluorhan
• Patrick Gray / Duke Marine Lab / @patrickcgray
• Martin Durant / Anaconda / @martindurant
• Zach Bruick / McKinsey & Company / @zbruick
• Ray Bell / DTN / @raybellwaves

60 Second Updates:

• Rich:

• Unavco has a PB of data, want to convert to cloud-optimized format. Should be joining the community.   Will submit a PR when we get a few speakers lined up.

• Deepak:

• PR to reduce memory usage on rolling operations.
• Adding a sliding_window_view to Dask
• Xarray release soon

• Riley

• Just finished PhD at CU Boulder. Using pangeo stuff at McKinsey.

• Tom

• Still working on STAC/VRT stuff with COG, starting work on other formats under STAC

• JimB:

• Same update as last week; again Bokeh 2.3 delayed, again Panel and HoloViews releases delayed in response. ETA: Bokeh 2.3 now released!
• Hiring for the HoloViz team
  (at least one full-time/permanent, plus a summer intern)
• Looking for a maintainer (or at least champion) for spatialpandas, which would be someone interested in pure-Python (Numba) scalable (Dask and GPU-friendly) spatial algorithms.

• Wesley:

• CICD on private server

• Nginx reverse proxy behind jupyterhub jupyter server proxy not working :/

• Custom JS for static exports

• https://curvenote.com/@wesley/cerege/interactive-plots-simple-example
• https://softwareunderground.org/events/2021/3/5/portable-interactive-plotting-python-with-javascript-sprinkles-wesley-banfield-steve-purves

• Martin:

• Gcsfs, fsspec release
• Intake derived docs at https://intake.readthedocs.io/en/latest/transforms.html
• More intake drivers (intake-streamz…)
• “Cramjam” may make fastparquet faster
• Streamz event source form user interaction

• Orhan

• GeoCAT-comp

• Released new version (v2021.02.0)

• EOF analysis, Climatology and anomaly, Relative humidity (from temperature, mixing ratio, and pressure), Dew point temperature (from temperature and relative humidity)

• Newer version scheduled (end of Feb.)

• Vertical interpolation (i.e. remapping/transform)

• From hybrid to pressure for this release.

• Patrick

• Lots of rasterio-based georegistration / warping for drone based ocean color mapping.
• Using Landsat provisional aquatic reflectance product for submesoscale SST/chl-a in the Gulf Stream.

• Zach:

• Joining to say hi

• Ray:

• No update

Agenda:

2021-02-17 2100 GMT (1pm PST, 4pm EST)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Jim Bednar / Anaconda / @jbednar
• Martin Durant / Anaconda / @martindurant
• Tom Augspurger / Microsoft / @TomAugspurger
• Thomas Moore / CSIRO / @Thomas-Moore-Creative
• Nancy Williams / USF CMS / @ognancy4life
• Julius Busecke / LDEO / @jbusecke
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Paige Martin / LDEO/Australian National University / @paigem
• Richard Scott / OZ Minerals / @RichardScottOZ
• Scott Henderson / UW / @scotty-hq
• Ryan Abernathey / Columbia / @rabernat

60 Second Updates:

• Rich:

• Deployed Panel apps for the first time to USGS folks on bokeh.usgs.gov using AWS ECS with a slightly-modified container based on the Bokeh Demo Apps Dockerfile and CDK/Gitlab runners. Would be nice to have a Terraform/Github action version.
• Worked with @ocefpaf (Filipe Fernandes) to extract custom Best Time Series data from a THREDDS FMRC “TwoD” dataset into Xarray. Would be nice if xarray could handle these “TwoD” datasets as they are quite common.  
• Using xEMSF to create a ROMS forcing file. from the HRRR “best time series” 2019 Zarr dataset ('s3://esip-qhub/noaa/HRRR/2019')

• JimB:

• Bokeh 2.3 release delayed; still time to pin your CI to 2.2 if needed before we break things! HoloViz.org tools to follow (Panel, HoloViews, maybe hvPlot)
• My recent Datashader webinar is available on demand at https://twitter.com/anacondainc/status/1360359286044508166.

• Martin:

• gcsfs, intake released
• New compressions coming to pydata (because I want to sub-block). Should be useful for fsspec-referencemaker to seek to an offset in a large compressed blob.
• webinar: object storage

• Tom A

• Fun with STAC + GDAL VRTs.
• Demo later if time.

• Thomas Moore:

•  (Feel good!) As part of Australia’s 2021 AMOS ( Australian Meteorological & Oceanographic Society ) conference NCI (one of Australia’s supercomputing centres) held a data training workshop on Pangeo.  While 2 years ago individual Pangeo users were involved in ad hoc lobbying of NCI, advocating for support of Pangeo workflows, it’s welcome to see NCI now owning their Pangeo offerings, providing training, and talking about Pangeo proactively.
• (Challenges) While we all need to meet the expanding cyber-security threats moves to deploy one-size-fits-all approaches to security is somewhat hampering Pangeo HPC efforts in Australia.  Getting this balance right and continuing to advocate for nuanced security protocols for HPC continues to be a challenge and time-sync.  Question: are there good examples of “best-practice” cyber-security out there for HPC?

• Maybe get in touch with Kevin Paul (NCAR)???

• Nancy Williams:

• New PI, transitioning from MATLAB to python and asking my students to use it as well, hiring either a scientific researcher or postdoc, teaching a lab in python and Julius helped me set up a Binder (THANK YOU, JULIUS!) Looking to get into cloud computing with CMIP data eventually.
• Hiring, possibly a postdoc: Latest News & Announcements/Jobs topics - Pangeo

• Julius

• Fighting cmip6 data - Is there a way to ‘throttle’ xarray.to_zarr()?

• Maybe the new annotations? Worker Resources — Dask.distributed 2021.02.0+7.g383ea032 documentation

• Thinking of dropping the ‘longitude ordering’ as standard step in cmip6_preprocessing. If you need that feature, let me know! Issue
• Actually doing science with all the pangeo tools and its wonderful!!!

• Joe

• FYI: Benoit Bovy starting work on xarray index refactor.
• Pangeo-ML kicking off this week, also at NASA ESDSWG last week and gave some talks on Pangeo
• Excited about working on Xbatcher and Intake-STAC again.
• Pangeo on M1s. Who’s tried it?

• Paige

• Slowly but surely learning more about Pangeo software and cloud computing
• Currently working on project that looks at sea surface temperature variance in a few climate models in the frequency domain. Using rechunker.
• Went to AMOS meeting, promoting pangeo.

• Richard

• Talking to people about large scale Sentinel 2 processing.  FrontierSI is apparently a non-profit consultant type group working at a higher level here https://frontiersi.com.au - involved with Open Data Cube etc.
• CSIRO has Earth Analytics Science and Innovation platform - which is apparently a mostly working ODC on EKS https://research.csiro.au/cceo/underpinning-technologies/earth-analytics/ - so a possibility for commercial  organisations to get managed shared or single tenant services.
• Getting information out of Geoscience Australia for data quality - https://github.com/RichardScottOZ/Geoscience-Data-Quality-for-Machine-Learning - sometimes nice data scientists point you to very useful things :- https://services.ga.gov.au/gis/geophysical-surveys/wms?request=GetFeature&service=WFS&version=1.1.0&outputFormat=csv&typeName=gadds:geophysical_survey_datasets
• End up with xarray or many data quality layers, plan being to make them COGs and turn into a web application

• Scott

• Working with non-cloud NASA data. Fighting with authentication issues. Makes using our tools difficult.

• They want visibility into who is using the data.

• New notebook: cog-best-practices/4-threads-vs-async.ipynb at main · pangeo-data/cog-best-practices (github.com). Using aiocogeo for concurrent reads of a single COG.

• Ryan

• Pushing forward with pangeo-forge. Talked through an end-to-end (“steel-thread”).
• NSF infrastructure proposal related to OSN (open storage network). Wanting to create a “science data fabric”, bunch of OSN pods with compute + storage.

Agenda:

• Tom: (STAC + VRTs if time)
• Cloud data cookbook?

• We have a lot of knowledge about working with data in the cloud. How can we share that?
• Simple examples like “open a geotiff from cloud storage”. Lots of small, obscure tricks.
• COSIMA  cookbook is on github (COSIMA/cosima-cookbook: Ocean and sea-ice model analysis tools and examples. (github.com)?)
• Chris Albon has a nice layout.
• Could also use this to highlight areas that should be improved in the tooling.
• Can this go in pangeo-gallery? Might need to update the theme to make it searchable. Would hopefully include updating the website to streamline things.

• Intake-thredds is out

2021-02-10 1700 GMT (9am PST, 12pm EST)
Name / Institution / @GitHub handle:

•  Rich Signell / USGS / @rsignell-usgs
•  Martin Durant / Anaconda / @martindurant
• John Clyne / NCAR / @clyne
• Orhan Eroglu / NCAR / @erogluorhan
• Tom Augspurger / Microsoft / @
• Ray Bell / DTN / @raybellwaves
• Julius Busecke / LDEO / @jbusecke.
• Jim Pivarski / Princeton / @jpivarski
• Peter Killick / UK Met Office / @DPeterK
• Ryan Abernathy / Columbia / @rabernat
• Deepak Cherian / NCAR / @dcherian
• Zac Flamig / AWS / @zflamig
• Jim Bednar / Anaconda / @jbednar
• Wesley Banfield / CNRS / @WesleyTheGeolien

60 Second Updates:

•  Rich:  looking into how to best deploy Panel app on AWS.  Bokeh project approach
  (ElasticBeanStalk), started Serverless Pangeo discussion, Biden’s commitment to 30x30.
•  Martin:

• Fixing some long-standing things in gcsfs prior to release
• Flurry of intake drivers (metabase, stripe, sklearn; salesforce and Marketo coming; streamz from before)
• BNL people to have a go at performant intake server and make protocol more concrete
• Backlog on documentation, particularly “derived”
• https://know.anaconda.com/CommonPitfalls-LP.html - Anaconda webinar next week

• John: John and Rich met to discuss bi-weekly talks for meetings. Rich will update website with schedule, etc. John will work with potential speakers to get talks scheduled. Tentative first talk on April 7. See discourse discussion.
• Orhan:

• GeoCAT-comp
• EOF analysis: eofs package existing: Working on potential gaps from Xarray and Dask compatibility

• Julius:

• Mostly working on cmip6_preprocessing and a paper . Expanding command-line args into pytest matrices
• Testing works on cloud data expanded to a larger group of datasets (https://github.com/jbusecke/cmip6_preprocessing/pull/89)

• Ray:  

• Looking for friendly(?) lists of public weather datasets e.g. https://catalog.pangeo.io/browse/master/atmosphere/; https://registry.opendata.aws/tag/weather/; https://azure.microsoft.com/en-us/services/open-datasets/catalog/noaa-global-forecast-system/;
• Dask = “allowed” vs dask = “parallelized” in xr.apply_ufunc(http://xarray.pydata.org/en/stable/generated/xarray.apply_ufunc.html). Useful for https://github.com/xarray-contrib/xskillscore/pull/221 

• If the function can handle dask arrays use “allowed”. If not use “parallelized” for a “map_blocks” application (technically it uses dask.array.apply_gufunc under the hood).

• Pangeo ML “preprocessed” weather and geo data: http://mldata.pangeo.io/preprocessed_datasets.html

• Jim Pivarski: (my second meeting) I have started looking at the Argo Floats data because of its variable-length structure, to see if Awkward Array can be useful here. Nicolas Morimer has previously considered ingesting these data into XArray and ran into the problem of variable lengths. I did a first translation of all the NetCDF4 files (123 GB), losslessly into Parquet files (6.4 GB!) and working with it with Awkward Array. In the short term, I’m preparing an Awkward Array tutorial for SciPy 2021 and am thinking about mini-analyses that can be done with it in a 2-4 hour period. (Rasterized temperature maps? Something simple, particularly if it uses NumPy-like indexing to slice it up.)

• See CF conventions for ragged arrays: http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#_indexed_ragged_array_representation
• Also: http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#_contiguous_ragged_array_representation
• JimP: The above is a Pandas-style/SQL-style surrogate key for expressing raggedness. It’s a “bottom-up” approach to raggedness, similar to the one taken by Parquet except that Parquet further packs these into “repetition levels.” This form is particularly good for doing reductions along the variable-length dimension. There are also “top-down” approaches used by Arrow and Awkward Array, which are better for random access.
• Deepak: I’m interested in whether we can get this into xarray. Even just a small subset of “awkward arrays” like argo here which is ragged on one dimension.
• JimP: I am, too. That’s more of a long-term goal, to see if Awkward and Xarray can be used together. Argo temp/pres/psal are ragged in two dimensions (profiles and levels), and some of its variables are more.
• JimB: Also see SpatialPandas, which we made to handle a small portion of what Awkward addresses (efficient processing of variable-length polygons). We’d be happy for SpatialPandas to build on awkward, in principle, but are not sure of details in practice, and don’t have funding to expand or refactor SpatialPandas at the moment.
• Deepak: here are xarray’s expectations: basically shape, ndim, dtype; I guess shape could be the “longest” length along each ragged dimension; ndim,dtype limit the “awkwardness” of the array but that’s totally fine: https://xarray.pydata.org/en/stable/terminology.html#term-duck-array
• JimP: Unless we always pad to the longest, the concept of “shape” doesn’t apply. (See conversation with data-apis.) This is something that will take some thought, but I’m willing to think about it long-term. Speaking of Zarr, I’m also looking at developing a Zarr v3 extension for Awkward, to pass this kind of data around with Zarr.

• JimB: shape = 2x3xnp.inf?

• Deepak: np.inf == np.inf is True, which is not nice for shape comparisons.
• Depends on the goal; if it’s a 2x3 array of variable-length items, seems like the shape _should_ match (though the data won’t necessarily)

• Deepak: I don’t think xarray uses the shape too much, it just needs to be defined. It’s OK if not all of xarrays’ API works...
• I used to use exactly that notation! Now I’m using Datashape, which expresses it as “2 * 3 * var * float64”, etc. As you can see, the shape and dtype get mixed if you do this.
  >>> ak.type(argo.profiles.levels[["pres", "temp", "psal"]])
  23977 * var * var * {"pres": ?float32, "temp": ?float32, "psal": ?float32}
  >>> ak.type(argo.profiles.levels.temp)
  23977 * var * var * ?float32
• JimB: Also good.
• JimP: Thanks for the links! I’m following up on that.

• Tom:  working on deep learning demo on GPU using rapids, also set up ti-tiler
• Wesley :

• Set of panel apps behind flask

• Use query params to hook into centralized DB
• bokeh.embed.server_documentt

• fortran-> python water flow / routing files

• Ryan:

• Talks at UN Ocean Decade, ECMWF workshop
• Forge hacking

• JimB:

• Bokeh 2.3 coming by Monday or so. Breaks things, so we’re working on fix releases for Panel and HoloViews. Hopefully we can keep things going without significant disruption; we’re on it!

• Zac:

• Zarr conversions at scale

• Peter:

• DB/mongo for metadata (Vs arco)

Agenda:

2021-02-03 2100 GMT (1pm PST, 4pm EST)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Jim Bednar / Anaconda, Inc. / @jbednar
• Tom Augspurger / Microsoft / @TomAugspurger
• Martin Durant / Anaconda / @martindurant
• Patrick Gray / Duke Marine lab / @patrickcgray
• Zac Flamig / AWS / @zflamig
• Richard Scott / OZ Minerals / @RichardScottOZ
• Scott Henderson / UW eScience / @scottyhq
• Ray Bell / DTN / @raybellwaves
• Rob Caskey / DTN

60 Second Updates:

• Rich -- posted instructions for running the HoloViz Tutorial using conda-forge packages on the HoloViz Discourse. Great STAC and Cloud Demos sessions at the ESIP Winter meeting last week. Should show up on ESIP 2021 Meeting Youtube playlist soon
• JimB:

• Presented at ESIP last week on .interactive() support for xarray
• New shipping traffic example for Datashader/HoloViews:
  https://examples.pyviz.org/ship_traffic/ship_traffic.html
  Also doubles as a demo of how to select individual data points or polygons in a datashaded plot.

• TomA:

• Working on a land-use classification problem using rasterio/xarray with a GPU + dask

• Martin:

• Intake community meeting tomorrow

• bluesky/databroker#644 (https://github.com/bluesky/databroker/issues/644) ?
• https://github.com/danielballan/catalog-server-from-scratch

• Come to his upcoming webinar (Feb 17):
  https://know.anaconda.com/CommonPitfalls-LP.html

• Patrick:

• 4th-year PhD Student at Duke Marine Lab, applying ML/xarray/etc. to ocean science and remote sensing. Works with drone data (which adds up quick!).

• Zac:

• Joe Flasher’s job has opened up:
  https://www.amazon.jobs/en/jobs/1418423/technical-business-development-manager-open-geospatial-data-lead-aws 

• Richard:

• Exploring using Dask for working with large datasets, e.g. Sentinel2.
• Looking at generating tiles
• JimB: try https://github.com/makepath/mapshader? Still brand new and probably not very polished yet, though.
• https://github.com/developmentseed/titiler-pds (AWS CDK cloud formation endpoint deployment - need to fiddle with default region perhaps to deploy to us-west-2 for AWS COGs --profile didn’t pick that up in deploy.
• https://github.com/RichardScottOZ/Geoscience-Data-Quality-for-Machine-Learning
• https://github.com/RichardScottOZ/australia-gravity-data/blob/main/notebooks/Gravity-Quality-PyGMT.ipynb  - data quality possibly a good xarray problem - if you have say, 50 channels into a deep learning model…. Various pixel quality ratings at each layer - in this case for a continent scale model.  A broad discussion - the geoscientists want this, but no-one has actually documented it in their own domain or datasets systematically.
• Operationalising deep learning/clustering of country scale models - Zarr and Dask

• Scott:

• Presented at ESIP last week
• Little NASA greenland SAR panel app https://github.com/scottyhq/greenland-ice-mapping-project#prototype-panel-app 
• Found some wrinkles using HoloViz tools to access COGs that require authenticated downloads; may need some thought to make it work smoothly

• Ray:

• Working on xskillscore, for validating forecasts 

• Rob:

• Data science arm of commodities, weather

Agenda:

• Rich: Running rechunker workflows on AWS [Rich/Zac]

• Rich: Where to store the huge netcdf files involved
• Tom: seems to work well to grab a remote HDF5 file and work with it locally in a higher-performance format (e.g. zarr) on local disk
• Martin et al.: grabbing it all makes sense if the metadata is scattered throughout and is all needed.
• See suggestions in https://github.com/intake/intake-xarray/issues/88 

• Tom: Demo

• Land Use / Land Cover using Dask with Azure Blob Storage
• Using a land-use classification model developed in house
• Looking for changes in classification
• Builds a GDAL VRT virtual raster dataset file
• PyTorch model is broadcast to each work ahead of time for performance (as it’s gigs of memory)
• Found better performance for reading COGs in wide chunks in the x dimension (presumably due to row-major order, possibly also helped by GDAL merging adjacent row chunks)
• Needs to align observation data from different years into a similar grid
• Then computes the changes
• Plots results using hvPlot, Datashader, and Panel
• Used GPU because it was about 300X slower on a CPU. Could add enough cores, maybe, which could be competitive since some part of the computation is memory bound and most of the available cores go idle anyway.
• https://nbviewer.jupyter.org/gist/scottyhq/c30b2c4114e72f8f30c3d1df8330af26 
• https://github.com/intake/intake-stac/issues/65 

• Jim: Demo

• Live app for exploring ship traffic is linked from
  https://examples.pyviz.org/ship_traffic/ship_traffic.html
• If you download and run the project there, it should fetch the underlying data
• Demonstrates how to select data points from a dataset too large to put in your browser.
• Also demonstrates pre-processing massive CSV files to generate a compact and fast Parquet file (takes 7h to load CSVs in Pandas, cut down to 10 min to load CSVs in optimized Dask, then only 10 seconds to load final Parquet file into Dask).
• Also demonstrates using pn.state to share an app that uses lots of memory -- all visitors to the app share the same dataset in memory, so app starts up fairly quick for a new visitor and the memory cost for each new visitor is low.

2021-01-27 1700 GMT (9am PST, 12pm EST)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• John Clyne / NCAR / @clyne
• Martin Durant / Anaconda / @martindurant
• Julius Busecke / LDEO / @jbusecke
• Deepak Cherian / NCAR / @dcherian
• Tom Augspurger / Microsoft / @TomAugspurger
• Scott Henderson / UW eScience Institute / @
• Peter Killick / UK Met Office / @DPeterK
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Kevin Donkers / UK Met Office / @kaedonkers
• Aaron Hopkinson / UK Met Office / @arh89
• Theo McCAie / UK Met Office / @tam203

60 Second Updates:

• Ryan

• Presenting next week at National Academies Ocean Decade US Launch Meeting about “OceanCloud”
• Merged big rechunker refactor:
  https://github.com/pangeo-data/rechunker/pull/77 
  Please try it out!
• Copernicus “Cloudification” Technical Report published:
  Opening new horizons: How to migrate the Copernicus Global Land Service to a Cloud environment | EU Science Hub 
• Lots of Pangeo Forge work

• John:

• Project Pythia update (https://projectpythia.org/)

• Growing collection of external content. Trying to figure out how to organize, make searchable, discoverable, etc.
• Focusing on identifying and  filling in gaps in scientific python basics.

• Martin:

• Access to gzip/bzip2/tar file chunks from fsspec-reference-maker? Should be possible, requires a scan of the file to determine the blocks.
• Xarray PR for fsspec URLs in open_dataset / open_mfdataset done again

• Julius:

• Cookiecutter-science-project template repo

• Beginner friendly way to set up a science repository
• Produces a repo like this example in minutes
• Documentation to enable users to ‘dive deeper’ (but they dont have to).
• Looking for feedback.
• Should we distinguish “science project” from “software library”

• Here’s an NCAR version: https://github.com/NCAR/pypackage-template/
• Update cookiecutter templates once repo is created… https://cruft.github.io/cruft/

• Welcome — The Turing Way (the-turing-way.netlify.app) is nice (thx!)

• Billion zarr chunks was mostly cmip6_preprocessing’s fault (sorry!)

• A sortby call that may not have been necessary was making a bunch of chunks.

• Deepak

• Not much. Some cf-xarray stuff

• Tom:

• Open PR to dask-labextensions fixing the issue with accessing dashboards from a Hub: Allow errors in redirect check by TomAugspurger · Pull Request #167 · dask/dask-labextension (github.com)
• Started branch on parallel reads from threads with rioxarray: Comparing corteva:master...TomAugspurger:lockless · corteva/rioxarray (github.com)
• Chat with Brendan about xarray-spatial

• Scott

• Attending ESIP
• Disabled the AWS binder (bitcoin). Need to implement auth. Looking for help.

• Pete:

• AWS Service Workbench. Hopefully removes a bunch of the maintenance hassles of running a hub-like thing.
• AWS link: https://aws.amazon.com/government-education/research-and-technical-computing/service-workbench/

• Joe:

• Not a lot of dev, doing lots of science and adminy things.
• NASA funded Pangeo-ML project kicks off in February

• Most of the coordination will happen on the pangeo-ml monthly call
• Includes credits on Amazon. Will hopefully keep the AWS binder running.

• Charles

• Pangeo-forge issues.
• A bit of design work / thinking on Pangeo’s landing page. Happy to have discussions on this.

• Kevin

• Just checking in.

• Aaron

• At informatics Lab. First time joining. Checking in.

• Theo introduction

• Updating on the ARCO-ready dataset processing for archive data.

Agenda:

• Pangeo virtual talks (see discussion on discourse) [john]

• Frequency of talks?

• John: weekly seems like a high bar. But biweekly makes the morning / afternoon timezone issues difficult to track.
• Can we alternate seminars with check-in meetings? Then do two weeks early (seminar, check-in), two weeks late (seminar, check-in)?
• Seminars would be recorded. So that somewhat offsets the timezone issue.
• How many Australians are (or want to be) involved? A bit hard to say, since we’re changing up the format.
• Plan to post each seminar recording to discourse and encourage technical discussions.
• Proposed plan: keep the current times (seminars during the early meetings). Open to shifting the time if a presenter can’t make it. Can hold separate “viewings” of the recorded content.

• Dask conference, suggestions for invites 

• Looking for diversity in speakers, fields, to give a talk.
• Ideally users demonstrating what / how they’re using Dask to do their work.
• Conference is the near the last week of April. No deadline on the invited talks / CFP yet.

• Intake_informaticslab intro

• Repo, binder.
• Met Office opening up some largeNetCDF based datasets such as global and uk weather and Air quality 2020 onwards, 2015-2020 UK weather, Met Office ‘big 4’ atmospheric models.

• Can be difficult to share these datasets in cloud-optimized formats. Sometimes technical barriers.

• `intake_informaticslab` is an attempt to make these datasets analysis ready and cloud optimised.
• It relies on files following a known naming convention. (In theory this could be querying an API to retrieve the data)
• It does this by creating a Zarr store (in memory) that maps between the temporal indices and the NetCDF files.
• It has no need to pre-read any of the files. Rather relies on some 3rd party knowledge that dictates the above mapping.

• Not requiring a pre-read makes it possible to present data stored in tape-archive as analysis ready.

• Uses intake as a Python API. The Driver and the Catalog are tightly coupled.
• Early feedback is positive.
• Also does some caching. Read from tape. Might cache that chunk to blob storage as a Zarr dataset. Gradually build up a real ARCO Zarr dataset on demand.
• Big difference from fsspec-referencemaker

• Uses the original driver (e.g. h5py) to pass through the data (Aaron: the data are passed through - in theory we can read with anything - loader function returns bytes, see here)
• fsspec-referencemaker does a conversion to bytes. Mostly focused on single large HDF files, that are internally chunked.

• One challenge: you’re stuck with the existing chunks (which is dictated by the file structure). Pairs nicely with rechunker.
• Thought: apply the “caching” idea to a rechunked version. So cache that layer rather than the data as read.
• Xarray has been looking for something along these lines for open_mfdataset, where there’s a well-known format / convention for how the files are written.

• The informatics lab is applying this to more and more datasets. Starting to see some patterns that can be abstracted.
• https://github.com/pydata/xarray/issues/2697
• https://github.com/pydata/xarray/issues/2039

• Unclear if we can define a “spec”, but hoping to get to the point where the glue code is pretty small.
• Interesting overlap with pangeo-forge. Thinking about similar ideas of building an ARCO dataset from a set of non-ARCO files stored in some convention.
• Hoping to remove the `-informatics-lab` specific components.
• Perhaps works similarly to this example Ryan developed for GOES on S3?
  https://nbviewer.jupyter.org/github/awslabs/amazon-asdi/blob/main/examples/dask/notebooks/goes16_dask.ipynb 
• Discussion on filepaths being expanded into coordinates / values. Also came up in fsspec-referencemaker.
• “Zarr cloner” idea + intermediate chaching + rechunking will make it easier to break the 1:1 relationship between source files and output Zarr chunks.
• The `Dataset.to_zarr(..., region=...)` keyword might be worth looking at. Facilitates parallel writes.

• Maybe challenging for rolling datasets (or are these expanding every day? Sounds like expanding since the shape is changing).

• Lots of people looking at forecast data. Not an obviously best way of doing that right now.
• NCML: https://www.unidata.ucar.edu/software/netcdf-java/v4.6/ncml/ 

• Xarray & ncml: https://github.com/pydata/xarray/issues/2697
• fsspec-reference  maker does handle URLs, so you can do ncml-like things.

• Also worth looking at for existing spec (XML as well :( https://gdal.org/drivers/raster/vrt_multidimensional.html 

• Pangeo Forge update

• Recipe project board:
  https://github.com/pangeo-forge/staged-recipes/projects/1
• Software project board:
  https://github.com/orgs/pangeo-forge/projects/1
• Looking for community members who know the details of the datasets.

2021-01-20 2100 GMT (1pm PST, 4pm EST)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Kevin Tyle / UAlbany / @ktyle
• Jim Bednar / Anaconda / @jbednar
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Zac Flamig / AWS / @zflamig
• Martin Durant / Anaconda / @martindurant
• Richard Scott / OZ Minerals / @RichardScottOZ
• Chelle Gentemann / FI / @cgentemann
• Thomas Moore / CSIRO Australia / @Thomas-Moore-Creative
• Tom Augspurger / Microsoft / @TomAugspurger
• Ryan Abernathey / LDEO / @rabernat
• Scott Henderson / UW / @scottyhq

60 Second Updates:

• Rich:

• Thinking about referenceFileSystem with reference to Discourse post (link?)

• Shared metadata between different dataset formats

• Reviving some dead Discourse posts?
• Spotlight on Project Pythia!

• Kevin:

• Involved in Project Pythia, goal to put together an educational resource for the Pangeo ecosystem, just finished an online class at UAlbany, introducing students to the Pangeo ecosystem (including holoviz).
• Links:  

• https://projectpythia.org/
• https://github.com/ProjectPythia/projectpythia.github.io
• https://www.youtube.com/watch?v=_UxlrwB_evo&feature=youtu.be&ab_channel=ProjectPythia 

• Jim:

• The HoloViz team is happily working on other non-geo (but currently funded :-) projects; still waiting on the NASA ACCESS grant to get underway so that we can get back to more geo-specific topics.

• Charles: transitioning into new job with NVIDIA RAPIDS team (with Tomlinson), updating pangeo-forge documentation (could use some help with readthedocs)
• Martin:

•  https://github.com/intake/intake/pull/564 
• Intake-streamz  -  project for realtime processing, aggregations into dataframes, now there is an intake reader.   Can watch for new data in a variety of ways.  
• DICOM is a medical format that we could use offset and chunk on (referenceFileSystem again!)

• Richard Scott:   working with mining data, remote sensing, geophysics, using Sentinel data for Peru, 100+TB size data, trying to do country size problems.   Rich: here is @jkellndorfer’s Sentinel-1 time stack in Zarr to look at expansion of illegal gold mining in Peru (video,  notebook)
• Thomas:

• Pangeo on Australian HPC resources maturing thanks to work of many.
• Looking at moves to Cloud providers. Recently gave Pangeo intro to some Australian cloud folks.  Any Pangeo on Azure experts? Yes! @tjcrone, @tomaugspurger?
• Working on expanding our Pangeo support  internal to our org.
• Suggested future agenda item: Commercialization and Pangeo community values, guidelines?    
• Ryan: great for folks to make money off Pangeo, but hopefully in way that promotes the community (e.g. Anaconda, 2I2C).  The pangeo licenses are truly open, and components can be used however folks want!

• Zac:

• New datasets on AWS:

• https://aws.amazon.com/about-aws/whats-new/2021/01/new-aws-public-datasets-available/

• Notebook on CMIP6 data access:

• https://nbviewer.jupyter.org/github/zflamig/dask-era5/blob/main/notebook/cmip6_zarr.ipynb 

• Challenges: appending to zarr is challenging, perhaps ReferenceFileSystem could be the solution!

• Chelle: new NASA twsc project funded (openscapes PI lowndes) open science project coord / E.8 supplemental OSS award / taking to JAXA re: cloud for AMSR3 launch 2023 / hope to leverage Pythia!
• Tom:

• Back on async / batch jobs for pangeo hubs
• Putting together design docs / research / proposal for a raster geospatial data library  (Tom to brainstorm with xarray-spatial Brendan Collins)

• Ryan:

• Working on epic Pangeo-Forge PR w/ new API
• Need organizational help with Summer Meeting, weekly meetings, web pages (see meta forum issues on Discourse)

• Scott: staying engaged in STAC,  ESIP Winter Meeting STAC session and Cloud Demos session. STAC Session, Cloud Demos session

Agenda:

• Pangeo Office Hours reboot [Rich]
• Project Pythia update?
• Rich

2021-01-13 1700 GMT (9am PST, 12pm EST)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Julius Busecke / LDEO / @jbusecke
• Deepak Cherian / NCAR / @dcherian
• Charles Blackmon-Luca / LDEO / @charlesbluca
• John Clyne / NCAR / @clyne
• Tom Augspurger / Microsoft / @TomAugspurger
• Orhan Eroglu / NCAR  / @erogluorhan
• Peter Killick / UK Met Office / @DPeterK
• Jim Bednar / Anaconda / @jbednar

60 Second Updates:

• Julius:

• Persistent issues with dask ‘auto-slicing’ cmip6_preprocessing issue
• Some discussion over at xgcm about how to implement high level div/grad/curl Issue Please chime in!

• Charles

• Working with STAC browser; publishing via npm

•  John:

• New Project Pythia URL:  https://projectpythia.github.io

• A Community Educational Resource for teaching Geoscientists to use the Python Scientific Ecosystem, (Jupyter, and Cloud computing )

• Open meetings to discuss educational content and infrastructure, Thursdays, 1pm ET. Details here: https://projectpythia.github.io/pages/meetings.html
• Starting to add content to the site. Looking for your input!

• Deepak:

• NASA Xarray + remote sensing proposal about to be submitted
• Outreachy / GSoC internships are open! (http://outreachy.org/)
• See pint-xarray: https://github.com/xarray-contrib/pint-xarray

• Tom

• Playing with GPUs for land cover model2

• Orhan:

• GeoCAT tutorial video on AMS: https://youtu.be/-ElqWVIAXpI
• GeoCAT stack (GeoCAT-comp and GeoCA-examples) getting fully compatible with Xarray (and Dask whenever needed)

• JimB:

• Grant proposal submitted for solar / space-weather version of GeoViews.
• Datashader 0.12 released, covering all changes since August.
• The HoloViz group will have a paid internship for summer 2021 in Austin or Berlin; if anyone has candidates please send them my contact info (jbednar {at} anaconda.com)

• Rich

• ResCogNito

Agenda:

• Met Office update (Peter K)

• Working closely with Microsoft on putting together a Pangeo that runs as much as possible on Azure ML

• Hooked it up to jupyterhub: spawner gives different options (images, resources), code is OS, working with Alan Turing institute
• Challenges related to proxying

• Contact Jupyter folks / Erik Sundell via public channels

• AWS service workbench

• Should we have a meeting in August?

• Piggy back on AGU / ESIP?
• Rich will look into option for ESIP Summer 2021

• GSOC / outreachy projects?

• Outreachy:

• Xarray documentation? Inspired by https://numpy.org updates..
• Xgcm HTML repr? (too narrowly scoped?)

• GSoC:

• New Xarray indexes?

• https://github.com/ESM-VFC/xoak
• Unstructured mesh indexing?

• https://developers.google.com/season-of-docs/

• https://discourse.pangeo.io/t/get-involved-in-pangeo-entry-points-for-new-contributors/643 

• Report from Cloud Ops WG

• https://docs.google.com/document/d/1I-2VNNHoAjjeYvlCezQhFLmiu2OevqGDS5nUAP-6Hfw/edit 

• New ML Datasets Webpage

• http://mldata.pangeo.io/ 

2021-01-06 2100 GMT (1pm PST, 4pm EST)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Martin Durant / Anaconda / @martindurant
• Rich Signell / USGS / @rsignell-usgs
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Tom Augspurger / Microsoft / @TomAugspurger
• Paige Martin / LDEO / @paigem
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Julius Busecke / LDEO / @jbusecke
• Scott Henderson / UW

60 Second Updates:

• Martin: intake-derived, intake-streamz, pitfalls webinar, Dask
• Rich: Still struggling with  HRRR, cloud con
•  Charles:

• Checking out intake-stac, seeing what goes into adding Zarr functionality? Issue
• Trying to set up STAC Browser as part of Pangeo Forge’s website, a little difficult because stac-browser isn’t a published package (yet?)

• Joe:

• Xarray user survey:
  https://docs.google.com/forms/d/e/1FAIpQLSfhVUao634zgpWP3BdrMPwzCd3WUqRbZZ4Baq_l2shoMhcIlQ/viewform

• Julius:

• Moved all xgcm examples to xgcm-examples PR  which is now on pangeo gallery

• We would like to collect as many examples for different models as possible. If you have used xgcm and have a nice example for ROMS, MOM5 etc please reach out/submit a PR!

• Speed up cmip6_preprocessing cloud data testing PR

•     Any suggestions input very welcome.

• Tom

• Doing netCDF -> Zarr conversion
• Looking into batch scheduling

• Could jupyterhub spawn a prefect agent?

• MS focused on geospatial raster data; what kinds of algorithms / operations is Pangeo stack missing?

• Paige

• Absorbing / learning
• Using Pangeo cloud for processing lots of oceanographic model data

• Ryan

• https://github.com/pangeo-data/rechunker/pull/77 
• https://cospar2021.gitlab.io/cloudcomputing/page/programme/ 

• Scott

• Blog post out! Skip the download! Stream NASA data directly into Python objects
• Wrapping up NASA proposal

Agenda:

• New meeting format

• https://medium.com/pangeo/pangeo-2-0-2bedf099582d
• https://docs.google.com/forms/d/1XEOpxyx-oMgY7JKGj29mqPuZZm9sldg4kaJRjMBkaFM/edit#responses 
• What would people like to see?

• The latest in approaching some type of data?

• Scott with geospatial rasters
• Jessica with icesat2
• Along-track satellite data
• Rob Hetland’s glider data
• Lagrangian data (floats / balloons etc.)

• Package updates

• Climpred
• GeoCAT
• Metpy
• Xclim
• Xcube
• Xarray-leaflet
• Xgcm

• Web-browser + Zarr stuff

• Joe
• Bio people

• STAC

• Rob emmanuel / Matt Hanson / Mattias Mohr

• Planetary Computer
• 2i2c
• Winter incubator at UW eScience

• Dhruv

• GPU Stuff (Jacob Tomlinson)

• Zarr / STAC / Intake / Xarray stuff

• Need mime type for zarr but zarr is not necessarily a file
• Joe: we are missing an abstraction: what type of store is it
• How to deal with Zarr v3
• Intake stac is meant for crawling STAC datasets, but actual opening lives in intake-xarray
• Should there be intake-zarr? YES! ZarrGroupCatalog (in Intake core)

2020-12-16 1700 GMT (9am PST, 12pm EST)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• John Clyne / NCAR / @clyne
• Wesley Banfield / CEREGE CNRS / @WesleyTheGeolien
• Orhan Eroglu / NCAR / @erogluorhan
• Jim Bednar / Anaconda / @jbednar
• Julius Busecke / LDEO / @jbusecke
• Zac Flamig / AWS / @zflamig
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Tom Augspurger / Microsoft / @TomAugspurger
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Martin Durant / Anaconda / @martindurant

60 Second Updates:

• Rich: Rechunking the HRRR forecast archive on AWS from hourly GRIB2 to Zarr w/100mb chunks and 72 time steps for forcing coastal ocean model hindcasts (use case: 2019 simulation of Delaware Bay).  Trying to figure out how to parallelize “wgrib2 hrrr.t07z.wrfsubhf1530.grib2  -netcdf hrrr.t07z.wrfsubhf1530.nc”.
  Blog post out on accessing netCDF4/HDF5 on the Cloud with new zarr, s3fs and fsspec releases. (fsspec=0.8.5, s3fs=0.5.3)
  Ryan -- could we use this same approach for GRIB2 files?

• Rich to post the HRRR conversion workflow to
  https://github.com/pangeo-forge/staged-recipes/issues/new/choose 

• John Clyne: no update
• Wesley

• Working on work group website: https://cerege-cl.github.io/
• Deploy JupyterHub locally
• Setup GitHub actions to auto deploy dockerfile to Dockerhub
• Work with IT on setting up Kubernetes / Rancher CI/CD workflows
• Preparing multi webpage panel app
• Bug in templates for panel
• rpa@ldeo.columbia.edu  still need that netcdf url!

• Don’t block on me! I’m overloaded and there are many other people in the community who can help

• Testing iooxa: https://iooxa.com/ 

• Julius

• Gave my first Pangeo talk at Machine Learning in  Science & Engineering

• Is there a way to see this? Do we need to register?

• Zarr problems. Finding a bunch of 0’s. Is there a way to ‘verify’ a write to a zarr store?

• Open in Zarr, check array.info
• Post on discourse

• Cmip6_preprocessing: Is there a way to get more oompf on GHA?

• Github workflow “self-hosted runner”

• Xarray.resample now works properly (lazily) with .weighted PR

• JimB

• xarray/pandas .interactive() support released; see https://hvplot.holoviz.org/user_guide/Interactive.html
• Please chime in at https://github.com/holoviz/hvplot/issues/533 if you have opinions about using hvPlot as the universally recommended access point for HoloViz tools (now including Panel, thanks to .interactive().)

• Zac

• https://aws.amazon.com/blogs/big-data/amazon-emr-studio-preview-a-new-notebook-first-ide-experience-with-amazon-emr/

• Dask-Gateway on EMR: Figure out how to bootstrap the dask user for proxyuser impersonation in an EMR cluster · Issue #64 · dask/dask-gateway (github.com) (doesn’t look good)
• Dask-Yarn — Dask Yarn documentation

• https://aws.amazon.com/blogs/aws/amazon-location-add-maps-and-location-awareness-to-your-applications/

• Joe

• Working a lot on xarray

• talk at CZI EOSS [link to slides?]
• Backends refactor is moving forward well; allows third party backends

• Working on Azure for CarbonPlan
• Spinning up Pangeo ML project

• Tom

• Playing with day-met (?) archive (gridded obs), converting to Zarr, learning about all the issues: Daymet - Azure Open Datasets Catalog (microsoft.com)
• UI stuff for JupyterHubs

• Jupyter server proxy? Theia https://github.com/jupyterhub/jupyter-server-proxy

• Charles

• Going to work for NVIDIA Rapids

• Ryan

• Admin admin admin!

• 2i2c Pangeo job posted:
  https://2i2c.org/job/osie-pangeo/ 
• Spinning up contracts with Anaconda, DevSeed, Coiled
• Wasabi contract

• Rechunker

• Martin

• Fsspec reference maker
• Intake
• releases

Agenda:

• NetCDF in the cloud breakthrough

• Cloud-Performant NetCDF4/HDF5 with Zarr, Fsspec and Intake - blog post

• Pangeo Pivot blog post and new meeting structure

• Draft link (do not share):
  https://medium.com/pangeo/pangeo-2-0-2bedf099582d 
• Weekly or biweekly?

• Maybe alternate

• Weekly “keynote”?

• Don’t make it another science seminar

• Schedule:
  https://docs.google.com/spreadsheets/d/1hQiX4YxTmuciUnm2cPp3G2knYYDTydzm2EDhdRhc524/edit?usp=sharing 
• Office hours idea?

2020-12-09 2100 GMT (1pm PST, 4pm EST)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Tom Augspurger / Microsoft / @TomAugspurger
• Julius Busecke / LDEO / @jbusecke
• Jim Bednar / Anaconda / @jbednar
• Martin Durant / Anaconda / @martindurant
• Charles Blackmon-Luca / LDEO / @charlesbluca

60 Second Updates:

• Rich

• MUR notebook crashing with Matplotlib, but not crashing with Holoviz
• Happy with the new async stuff (7x speedup on national water model workflow, opening, loading).
• Worth looking into how data was written to ensure that the coordinates aren’t unnecessarily chunked. What does xarray do?
• Did notice dask workers dying, probably OOM. Is this related to async?

• Tom

• Look at working with COGS, rioxarray. Looking for use cases, problematic workflows.     Scott’s COG best practices repo.

• Julius

• cmip6_preprocessing is now on the docker image PR
• I am now able to run CI that tests the google cmip6 data directly (thx @charlesbluca) PR

• This will help with some issues reported for specific models issue

• Preparing a pangeo talk for MLSE 2020
• Trying to fix some non-laziness with xarray.weighted PR issue

• JimB

• New datashader release sometime this week, with point/line dilation (spreading) supported at the xarray level, not just RGB (as originally requested by Rich 2 years ago!).
• In general, the HoloViews 1.14 release (last week) and bokeh 2.2 (and, better, 2.3 coming up) mean that you should never need datashade anymore except for categorical data; all the features are now available from rasterize, which lets you use hover and colorbars to reveal the numeric data values.
• New holoviews release also makes Datashader play nicely with Plotly and Dash, if anyone is using Dash as a deployment server.

• Charles

• Working on integrating dataset cataloging into the refactor of Pangeo Forge
• Had a talk with someone from Wasabi to check its potential as a new (cheaper!) cloud storage provider
• Opening issue for ESMSource feature for Intake-esm

• Martin

• Fsspec and intake moved to github actions for CI

Agenda:

• Teaching Pangeo (again).   gather.town

2020-12-02 1700 GMT (9am PST, 12pm EST)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Zac Flamig / AWS / @zflamig
• Julius Busecke / LDEO / @jbusecke
• Soeren Thomsen / LOCEAN / @soerenthomsen
• Chelle Gentemann / FI / @cgentemann
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Scott Henderson / eScience / @scottyhq
• Jaard Hauschildt / GEOMAR / @jaard
• John Clyne / NCAR / @clyne

60 Second Updates:

•  Zac

• AWS Reinvent is going on; lots of new features announced
• https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/
• https://aws.amazon.com/blogs/aws/amazon-ecr-public-a-new-public-container-registry/
• https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/
• https://aws.amazon.com/blogs/aws/coming-soon-ec2-c6gn-instances-100-gbps-networking-with-aws-graviton2-processors/ 

•  Julius

• Got my first particles to run in the cloud, need to understand dask internals better. Parcels is still not on the main deployment PR but the mamba solution link works fantastically.
• Discussion with Ag Stephens to improve cmip6_preprocessing and generalize error database
• Refactoring cmip6_preprocessing to run tests on actual google cloud data. Comments welcome PR
• Added a convenience function to xarrayutils to mask out the mixed layer in gridded files example.

• Soeren

• first time here, plans to do carbon auditing (carbon fluxes, storage on country level)
• plan to use geopandas for the shade files (EEZ’s), regional mask, xarray for the gridded fields (2D, 3D)
• want to make sure my approach to this problem is as effective as possible, find experts who did similar stuff already, how do deal with boundaries towards land or other EEZ’s
• https://discourse.pangeo.io/c/science/5
• https://xesmf.readthedocs.io/en/latest/ 

• Chelle

• ‘Science storms the cloud’ submission AGU Advances
• Fsspec fight club

• 0.8.4 has some speedups
• https://github.com/intake/filesystem_spec/issues/485 ←

• Region testing

• Ryan

• Lots of work on Pango Forge: https://github.com/pangeo-forge/pangeo-forge/pull/27
• Released xmitgcm 0.5:
  https://xmitgcm.readthedocs.io/en/latest/development.html
• Released rechunker 0.3.2
• Pushed for Xarray 0.16.2 release
• Pushed for Zarr 2.6.0 release
• Dask / Xarray debugging
  https://github.com/dask/dask/issues/6869 

• Charles

• Working on making STAC catalogs to mirror Pangeo’s Intake catalogs
• Making dashboards to track dataset interactions using Google Cloud Monitoring
• Checking out the CI for Pangeo repos - should we move away from Travis CI? Org site is shutting down later this month

• Scott

• Working on updating JupyterHubs to use dask-gateway 0.9 and latest JHub
• Merged PR on intake xarray

• John Clyne

• Considering EarthCube proposal to provide better support to unstructured grids (due in March). Please reach out to John if you have interests in discussing further, possibly partnering with, or simply talking about your needs.

Agenda:

• AGU rundown

• Ryan’s poster about  pangeo forge
• Panel on CMIP6
• Chelle ARD session

• Supporting NASA / NSIDC AGU Workshop

• Ryan raise issue about “pre-heating” binder

• Partnership with SWOT crossover campaign for model intercomparison

• https://github.com/roxyboy/SWOT-AdAC-ocean-model-intercomparison

2020-11-25 2100 GMT (1pm PST, 4pm EST)

• [cancelled due to Thanksgiving Holiday in USA]

2020-11-18 1700 GMT (9am PST, 12pm EST)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Julius Busecke / LDEO / @jbusecke
• Martin Durant / Anaconda / @martindurant
• Tom Augspurger / Microsoft / @TomAugspurger
• Scott Henderson / UW eScience / @scottyhq
• Takaya Uchida / CNRS / @roxyboy
• Jim Bednar / Anaconda / jbednar@utexas.edu 
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Orhan Eroglu / NCAR / @erogluorhan
• Wesley Banfield / CEREGE / @WesleyTheGeolien

60 Second Updates:

• Rich:

• Converting the whole 26 year National Water Model v2 reanalysis on AWS (230,000 hourly netcdf files, 10TB) to Zarr, will push to a personal bucket then hopefully copy to the NOAA Big Data Program bucket.  Doing this by downloading the NetCDF files, then appending each time step to a Zarr dataset, then using rechunker.

• https://github.com/pangeo-forge/staged-recipes/issues 

• Also trying to figure out how to use kubessh to execute long-running jobs on a gateway cluster (without JupyterLab).
• https://discourse.pangeo.io/t/pangeo-batch-workflows/804/4 
• Yuvi’s proposal: https://words.yuvi.in/post/kbatch/

• Julius:

• Bunch of small stuff in relation to cmip6 OMZ paper

•     xesmf wrapper for cmip6-preprocessing (maybe cf-xarray is helpful?)

• https://github.com/pangeo-data/xESMF/issues/16 

•     Using cf-xarray for cmip6-preprocessing
•  sign agreement hatching for e.g. multi-model mean  Example xarrayutils

• Martin:

• Specification of remote references (see agenda)
• PR to Zarr to fix things (what things?)

• Tom:

• Onboarding at MS
• Dask / Jupyterhub updates

• Scott:

• Post about COGs on discourse:
  Cloud Optimized Geotiffs + Pangeo best practices - Data - Pangeo 
• Rich: “dask median issue”(Scott says would be a good contribution to the COG page)

• Takaya:

• Working on a pitch to make (~100Gb) data from several basin-scale submesoscale permitting simulations at the SWOT cross-over regions during the fast-sampling phase and their inter-comparison analysis code publicly available.
• Follow up with fbriol: https://github.com/pangeo-data/pangeo-datastore/pull/58

• JimB:

• hvPlot to be released any hour now, which will make hvplot.holoviz.org/user_guide/Interactive.html be live (for now only on github). Shows how to use .interactive with xarray objects for one-line GUIs.

• Charles:

• PR merged in STAC browser to render Zarr data; next steps:

• Version handling for Zarr v2 and v3
• Figuring out short term solution to make Zarr in requester pays buckets public
• Thinking about how we want to generate STAC catalog moving forward

• Pangeo on STAC index: https://stacindex.org/catalogs/pangeo#/?t=catalogs
• Settling on tools for indexing cloud buckets (Rclone, S3P), reworking Ryan’s notebooks that generate STAC catalogs

• Orhan:

• Wesley:

• Refactoring app to fully use Dynamic Maps
• Add undo / redo
• Automatically fill depressions in DEMs
• Add template to App
• Repo: https://github.com/CEREGE-CL/netcdf_editor_app 
• Instance: https://netcdf-editor.ew.r.appspot.com/app 

• Zac (proxied by Rich): ERA5 1-click deploy of Dask & Jupyter notebooks into an AWS Account.  Got burned by new dockerhub pull rate limits accessing miniconda container, so switched to pulling “miniforge” container from quay.io (thanks to tip from Filipe), which worked great! -- drop in replacement for miniconda)
• Joe

• doing work on scikit-downscale, cmip6 in cloud (from azure)

• Could we implement cross-cloud caching?
• Pangeo “resolver” service, i.e. pangeo://cmip6/etc -> points to gcs or s3 appropriately
• Link to PR in intake that generates presigned urls?
• Nasa Cumulus proxy service

• Numpytiles and zarr: https://github.com/zarr-developers/community/issues/37

• Ryan

• Doing new science, finding patterns that don’t blow up Dask, rechunker, map blocks, like diffusion and smoothing.
• CMIP6-in-the-Cloud meetings, with Balaji and Kershaw.  CIDA is going to store CMIP data in Cloud.  ESGF funding is going away, LLNL is losing funding: http://www.bnlawrence.net/computing/bigdata/2020/11/the-future-of-the-esgf/ 

Agenda:

• Demo frow Wesley

• Repo: https://github.com/CEREGE-CL/netcdf_editor_app 
• Instance: https://netcdf-editor.ew.r.appspot.com/app 
• Follow up: provide a link to cloud-based netcdf / zarr data

• Martin:  discuss proposed spec for new fsspec referenceFileSystem.

2020-11-11 2100 GMT (1pm PST, 4pm EST)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Jim Bednar / Anaconda / @jbednar
• Orhan Eroglu / NCAR / @erogluorhan
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Julius Busecke / LDEO / @jbusecke
• Paige Martin / LDEO/ANU / @paigm
• Lee Bardon / Univ. Southampton, UK / @teatauri
• Martin Durant / Anaconda / @martindurant
• Zac Flamig / AWS / @zflamig
• Ryan Abernathey / LDEO / @rabernat

60 Second Updates:

• Rich: Still messing with Qhub, getting “Coastal Coupling” group with USACE, NOAA going, like the linux permissions, groups, github actions

• Ability to have group only notebooks and data

• Jim: Working on new Datashader and Panel releases

• HoloViz: Just so people know our situation, we are well supported but we are generally paid to add features (sometimes including docs, but usually software features).  As a result, we create lots of features! But every feature has the potential to break things, and nobody pays us for general polishing and infrastructure. General library support has to fit in after hours or when it gets really desperate. If anyone cares more about stability than about features, we’d love if someone put funds onto a grant proposal for the explicit purpose of bug fixes, better test infrastructure, and improving stability. We could make some pretty big advances in stability with not much money this way...

• Orhan:

• GeoCAT plotting gallery growing (now about 100 examples): https://geocat-examples.readthedocs.io/en/latest/
• Slightly turning our attention to unstructure grids
• Also spending more time on GeoCAT computational functionality

• Charles:

• Coordination of CMIP6 cloud data between Google, S3, NOAA-GFDL
• Syncing repositories, how to list buckets

• Julius: Lurking here, working on paper
• Paige: new postdoc with Ryan at LDEO - first Pangeo telecon!

• Working from Australia (ANU).
• New to pangeo, started working on xrft

• Zac: lurking as always
• Martin:

• Working on a way to read hdf5 with existing fsspec and zarr libraries.
• Able to put hdf5 data and metadata into an intake catalog, no copying/rechunking needed. Just a new metadata file. Nice usecase for pangeo-forge
• Working on blog post
• https://github.com/intake/fsspec-reference-maker 

• Ryan:

• Pangeo-forge has been idle for a while but now attention is back.
• Getting the community to align on specifications as a priority.

• Generalized case for hdf and microscopy data as a starting point

• Zarr v3: Soon to be open for comments. Plans for extensions to enable customization.

• Lee:

• First time at pangeo meeting.
• About to start PhD applications
• Working on paper with folks at NOC
• Happy to work on projects that need some more hands/eyes.
• Interested in physical oceanography + biogeochemistry. Loves programming, wants to improve software building skills.
• Maybe contribute to
  http://gallery.pangeo.io/repos/pangeo-gallery/physical-oceanography/ 

• Takaya:

• Also first timer here
• Did his PhD with Ryan, now in France
• Wants to learn more about newest developments of the community.

• [name]:

Agenda:

• Jim: Demo of taking cross-sections of a mesh -- does this approach achieve what Ryan is after? Would need to be a cross-section of a vector field, but that seems doable, as I think extracting the cross-section is custom anyway.
• 2i2c Launch:

• https://github.com/pangeo-data/pangeo-cloud-federation/issues/873 
• https://2i2c.org/posts/hello-world/ 
• https://github.com/2i2c-org/2i2c-pangeo-hubs 
• Non profit unique in this space
• 100% open source, and users are free to take their infrastructure and move it elsewhere
• Plans to migrate current hubs to 2i2c

• Wants to hire somebody to migrate and operate cloud hubs to 2i2c, recommendations to Ryan.

• New direction for pangeo: Shift back to software and science use cases, instead of infrastructure.

2020-11-04 1700 GMT (9am PST, 12pm EST)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Ryan Abernathey / LDEO / @rabernat
• Hillary Scannell / LDEO / @hscannell
• Jim Bednar / Anaconda / @jbednar
• Martin Durant / Anaconda / @martindurant
• Tom Augspurger / Anaconda / @TomAugspurger
• Julius Busecke / LDEO / @jbusecke
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Wesley Banfield / CEREGE / @WesleyTheGeolien
• Chelle Gentemann / FI / @cgentemann
• Chiara Lepore / LDEO / @chiaral
• Scott Henderson / UW /

60 Second Updates:

• Rich

• Helping ESIP with Qhub deployment for a coastal cloud experiment with USACE sensor data, NOAA National Water Model and USGS Coastal Model output
• Worked with Martin on HDF5 access with fsspec

• Ryan

• Cloud-Native Repositories for Big Scientific Data
• xrft

• Hillary

• New postdoc; Ph.D. from UW
• Working on tracking of marine heatwaves
• Trying to release python package soon

• Martin

• intake regression fixes
• xarray PR ready/zarr-write PR in
• CI overhaul to come

• Jim

• New HoloViz package hologridgen released, for folks wanting to construct quadmesh grids. (HoloViews GUI for calling pygridgen)

• https://llnl-conduit.readthedocs.io/en/latest/blueprint_mesh.html#mesh-blueprint 

• New HoloViz package lumen still progressing quite quickly, mainly for monitoring deployments and other servers but also for building dashboards using YAML.

• Tom

• Moving to Microsoft AI4Earth!
• Changes to Zarr graph in dask https://github.com/dask/dask/pull/6773
• Hub work, batch workflows, prometheus.

• Julius:

• Pangeo Blogpost about xgcm transform
• Fixed small but annoying bug in `nc-time-axis`
• Working on making xarray.weighted more dask-friendly

• Charles

• Conversations with people from NOAA to sync up google cloud + S3 data
• Learning about prefect

• Wesley

• Creation of interactive web app using panel to modify NetCDF files: https://cerege-cl.github.io/netcdf_editor_app/
• https://discourse.pangeo.io/t/interactive-netcdf-editing/892/2 

• Chelle

• Gave open science talk to ~100 @ nasa hq, E.8, inclusive, metric, award
• Karen St Germain (new Earth Science Division Director)  was interested...
• Open (inclusive) team organization, planning , budgeting, ...
• Trying to finish up BAMS paper
• OHW MHW    (marine heat wave group, they have a Slack channel)

• Chiara

• Crunching through CMIP6 6/3hourly data, lots of lessons learned about some limits and issues of pangeo google cloud deployments.
• Hopeful fix: https://github.com/dask/dask/pull/6779
• We should think about a detailed page guiding users in choosing the right size (mem and workers) for their workflow - to avoid waste and optimizing computations. (start here - https://github.com/pangeo-data/pangeo/pull/799 )

• Scott

• Working with Sentinel-1 SAR public dataset on AWS
• New demos of COG
• Planning post on Discourse around COG best practices

Agenda:

• Demo of new fsspec capability to access HDF5 files using zarr without needing FileChunkStore to be implemented in Zarr.  Click the 3rd binder button on this Hurricane Ike repo to check it out for yourself.
• Batch workflows again
• Async Zarr: https://gist.github.com/rabernat/21b17440ffc3c910758a35731a9dbf57 
• Catalog stuff

2020-10-28 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Tom Augspurger / Anaconda / @TomAugspurger
• Jim Bednar / Anaconda / @jbednar
• Deborah Khider/USC-ISI/@khider
• Martin Durant / Anaconda / @martindurant
• Charles Blackmon-Luca / LDEO / @charlesbluca
•  Zac Flamig / AWS / @zflamig

60 Second Updates:

• Ryan:
• Rich:

• Rechunking again, not sure how to rechunk from Xarray
• using 450 cores to write NetCDF to Zarr
• added an ERDDAP notebook to the Pangeo Gallery:

• Tom:

• Worked on some “ordering issues” (from zarr rechunk store: https://github.com/dask/dask/issues/6745. This is hard.
• https://github.com/dask/dask/pull/6773. New keyword to help with from_zarr / concatenate ordering / memory usage
• Put ML flow on jupyterhubs

• Deborah:

• Release of Pyleoclim (time series data analysis) ->
• next step: integration with field data (through xarray)

• Power spectra: https://xrft.readthedocs.io/en/latest/ 
• EOFs: https://ajdawson.github.io/eofs/latest/ 

• Jim:

• Mainly been recording various talks and tutorials, e.g. for PyData Global.
• Panel 0.10 formally released, with tons of new features.
• HoloViews 1.13.5 released, improving linked selections.

• Martin:

• did a webinar earlier today
• PR on intake for dealing with environment variables in intake templates
• fsspec is getting a compressed archive implementation
• spent time playing with awkward array
• PR for zarr parallel writes

• Charles

• PR to radiant earth stac browser to view Zarr metadata
• Learning about prefect

• Zac

• Questions about metadata APIs

• Static metadata in json on objection storage; cataloged with STAC
• ISO metadata
• NASA Earthdata
• Data.nasa.gov, data.usgs.gov

Agenda:

• ML Flow Demo / Discussion
• Paleoclim demo
• Blog post on “Pangeo Pivot”:

• Private draft link:
  https://medium.com/pangeo/pangeo-2-0-pivoting-our-community-2bedf099582d (DO NOT SHARE ON SOCIAL MEDIA YET)

• Dask gateway discussion

• https://github.com/pangeo-data/pangeo/pull/799 

2020-10-21 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Tom Augspurger / Anaconda / @TomAugspurger
• Martin Durant / Anaconda / @martindurant
• Sebastian Alvis / UW / @salvis2

• John Clyne / NCAR / @clyne
• Zac Flamig / AWS / @zflamig
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Julius Busecke / LDEO / @ jbusecke
• Jim Bednar / Anaconda / @jbednar
• Deepak Cherian / NCAR / @dcherian
• Chelle Gentemann / FI / @cgentemann

60 Second Updates:

• Ryan
• Rich:

• Excited about Qhub, buddy Josef Kellndorfer (SAR dude) got it going yesterday on AWS.
• Trying to work with a 320GB COG on Pangeo, experimenting with writing TileDB for big image data.
• Updated the Pangeo video playlist. Need to add Ryan’s big picture talk from JupyterCon 2020.

• Tom:

• Reshaping dask arrays: https://github.com/dask/dask/pull/6748, https://github.com/dask/dask/pull/6753
• Indexing issue
• Pangeo forge work

• Martin:

• Fsspec configuration for defining default arguments for any of the backends

• Sebastian

• Wrapping up; last week. Trying to get icepyx on pangeo gallery, update monitoring on the binderhubs

• John Clyne

• Had kickoff meeting for project pythia; assembled working groups (infrastructure, content, etc.); will soon be posting links to open meetings; medium post

• John C.:

• Project Pythia launched two working groups, one for infrastructure and one for content. Stay tuned for a posting on how to get involved with this educational resource

• Zac: Just listening
• Charles:

• Working on Vue for Zarr metadata for stac-browser (issue on build errors)
• Problems with hanging GitHub Actions workflow

• Jim B:

• Panel 0.10 being released imminently; release notes (still in prep) are at http://blog.holoviz.org/panel_0.10.0.html.  Can now easily build fancy authenticated apps, if anyone needs that!
• Also looking at datashader profiling for trimesh rendering, for UCAR. Initial results: 958 seconds for rendering with mpl, 6.7 for Datashader. SciPy Delaunay triangulation is also about twice as fast as mpl’s.

• Deepak : thinking about NASA Open source call (Discourse Link) + Call PDF
• Julius:

• Looking for CMIP6 grid files (‘metrics’) discourse entry. If you know anyone working at a modelling center, please point them there! This will be integrated with cmip6_preprocessing.
• Xgcm Quickstart Guide
• Highlight this amazing video using xgcm.transform 😍 by Dhruv Balwada 

• Chelle

• NASA EOSDIS made an announcement that Chelle will be helping to coordinate cloud strategy across DAACs
• Open source call from NASA
• Presentation to HQ on ‘open science’ - looking for any slides people might have prepared describing ‘spectrum of openness’ for both software, data, compute, ...

• Scott H

• Wrapping up NASA ACCESS project
• Satellite swath data

Agenda:

• QHub discussion. How do we want to interact with this project?

• Announcing QHub - blog post
• https://qhub.readthedocs.io/en/latest/ 
• Yuvi did some early work on QHub
• Quansight developed this internally for their customers

• Anaconda is contributing to it

• Pangeo does get some credit:
  https://qhub.readthedocs.io/en/latest/docs/aws/theming.html?highlight=pangeo 
• Chelle: there are lots of things in this space now (infrastructure as a service). Pangeo is well suited to providing guidance through this space.

• 2i2c update (Ryan)

• https://www.youtube.com/watch?v=YjonPLxDiwM 
• Can we hand maintenance / service off to 2i2c, who might use qhub to deploy our hubs?

• Dealing with big COGS (e.g. 300GB)

• Works great for visualization
• But not for processing
• Issues with GDAL?
• Have them post a question of Pangeo Discourse

• Can we help these people?
  https://discourse.pangeo.io/t/running-pangeo-with-sentinel-2-datasets-on-google-cloud/955/2 

• Affiliated with Vijay Modi’s group (remote sensing / sustainable development)
• Problem: there is no STAC catalog for Sentinel 2 on GCS

• GEE has STAC catalogs, but they aren’t on GCS
• Not straightforward to just copy the STAC catalog from AWS.

• Possible suggestions:

• Maybe use compute in GCP but get the data from AWS (slower, but it’s a public dataset)?
• Maybe set up compute in AWS?

• Updates to Pangeo website
• NASA open source

• Deepak’s post:
  https://discourse.pangeo.io/t/nasa-funding-opportunity-support-for-open-source-tools-frameworks-and-libraries/946/4 
• Jim: Bokeh is thinking about applying; there are a limited number of awards available.
• Scott: not a lot of $; intake stac; legacy HDF in the cloud

2020-10-14 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Sebastian Alvis / UW / @salvis2
• James Bednar / Anaconda / @jbednar
• Orhan Eroglu / NCAR / @erogluorhan
• Martin Durant / Anaconda / @martindurant
• Jim Crist-Harif / Prefect / @jcrist

60 Second Updates: started

• Sebastian Alvis: Wrapping up
• James Bednar: No updates. Been working on Lumen.
• Orhan Eroglu

• Working more on Unstructured Grids recently
• GeoCAT Contributor’s Guide: https://geocat.ucar.edu/pages/contributing.html

• Martin:

• Minor fsspec release
• Intake-xarray fixes

• Jim:  Working on dask-gateway next release (hopefully next week or two). Will give deployment flexibility and bug fixes for Kubernetes.

2020-10-07 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Chelle Gentemann / FI / @cgentemann
• Tom Augspurger / Anaconda / @TomAugspurger
• Jim Bednar  / Anaconda / @jbednar
• Ray Bell / Royal Caribbean / @raybellwaves
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Martin Durant / Anaconda / @martindurant
• Zac Flamig / AWS / @zflamig
• Sebastian Alvis / UW / @salvis2

60 Second Updates: started

• Rich: 1st ML training on pangeo a disaster, 100 people, instructor set up to read data from a file system (no fspec/fuse), first step was to download data from a s3 bucket, overran something, servers crashing. Having people go through a notebook was really difficult and we need to think about new ways to communicate.
• Chelle: looking for dev ideas ML remote sensing, chat Chelle on gitter, CCMP AVISO data on Pangeo
• Tom: getting pods packed into fewer nodes to save costs (GCP).  debugging logs (need admin access to work on this). New page on scheduler that gives access to logs.
• Jim: There’s a new  HoloViz project tentatively called Lumen (github.com/holoviz/lumen):

• Lumen makes it simple to make Panel dashboards to monitor stuff (pretty much any stuff) from a YAML config file (with no coding).  We use it to monitor our deployed notebooks and dashboard examples, but it can be used to keep tabs on more or less anything, whether that’s server processes or hardware devices or data (including Intake data sources).  It’s having its first release this week and has development funding for the next few months, so if this general category of usage is interesting please chime in on issues so you can guide how it develops!
• Discussion on creating tutorial on pangeo gallery for panel / lumen / heroku. Would be good to highlight how useful panel is and pangeo gallery is a good platform. 

• Ray: First meeting. If ok just listening.  Worked at RSMAS , developed xskillscore
• Charles: solution using github actions to keep GCP/AWS CMIP6 in sync.  Working to render zarr data in stack browser.  
• Martin: started work on async write with zarr
• Zac: just lurking
• Sebastian: getting GitHub Actions set up for AWS hub deployment and image building, adding to hubploy documentation.

Agenda:

• Dask Array indexing intuitions (5-10 minutes) , (what do users expect indexes to mean?)

• Using FUSE on JupyterHub:  

• fsspec has FUSE?

• from fsspec.fuse import run; run(s3, “<bucket>”, “/mount/dir”)

•  InformaticLabs lab approach with goofys  

• Jacob at one time said “flex volumes are

• Brainstorm on how to improve notebook tutorials  - lessons learned doc? Spinup doc. Automated testing ?  http://gallery.pangeo.io/repos/pangeo-data/pangeo-tutorial-gallery/#  best pract?  Highlight https://blog.dask.org/2020/08/21/running-tutorials .  

• Community of helpers, Rich, Tom, etc.  

• Zac offered AWS resources, help with videos, production, dissemination.

• Force re-run notebooks CI to test that the, can be reduced to use smaller chunks of data to make easier, or make stripped down versions that you can run CI (papermill - notebook automation tool).  Jim brough up how to test stripped down versions / dev large version.
  
   

2020-09-30 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Martin Durant / Anaconda / @martindurant
• Julius Busecke / LDEO / @jbusecke
• Ryan Abernathey / LDEO / @rabernat
• Sebastian Alvis / UW / @salvis2
• Scott Henderson / UW / @scottyhq
• James Munroe / MUN / @jmunroe
• Tom  Augspurger / Anaconda / @TomAugspurger
• Rion Dooley / Chapman / @deardooley
• Zac Flamig / AWS / @zflamig
• Deepak Cherian / NCAR / @dcherian

60 Second Updates:

• Martin: working on integrating Zarr + Xarray + Fsspec with async features
• Julius: released xgcm 0.5.0 Docs for transform feature, already deployed to aws; working on xgcm; working on parcels on the cloud; stuck on dask workers not having package installed

• https://oceanparcels.org/ 
• Tom will explain some hacks
• James: Callum Shakespeare is interested

• Rich: Bugging Alex Jelenak to engage with Zarr FileChunkStore, got my first rechunker use case going (after stumbling with max_mem and setting chunk size larger than my actual dimension); at ESIP cloud cluster, Matt Hanson complained about 100Mb Zarr chunk size for visualization in the browser, leading to a discussion about the different use cases of visualization in browser vs number crunching in python. Ryan says maybe tileDB can give us best of both worlds.
• Sebastien: maintenance and troubleshooting for Pangeo Cloud Federation; terraform configuration
• Scott: STAC sprint, will post links to summary documentation; worked on intake-stac, master branch now works with STAC 1.0;

• https://www.youtube.com/channel/UCfEmhf6e_yOzvh5FzVDmZPg 
• https://github.com/radiantearth/community-sprints/tree/master/08182020-remote 

• James Munroe: secretly lurking; lots of teaching, training

• http://gallery.pangeo.io/
• Pythia
• Check out this Bokeh GUI editor for CTD data

• Tom:

• Cloud federation:

• very basic tests for GCP deployment (deploy + simulate actual workflow)

• Maybe integrate with binderbot / pangeo gallery

• Dask gateway has mapping widget to set environment variables, dask.config values, etc. Docs 

• Stalling computations issue:

• https://github.com/pangeo-data/pangeo/issues/788. Fix at https://github.com/dask/distributed/pull/4135, looking for additional examples.

• Dask array indexing

• Rion:

• PI for Agave platform
• https://docs.agaveplatform.org
• https://slackin.agaveplatform.org/
• https://agaveplatform.org/

• Zac

• Helping convert some grib data to zarr (HRRR). Running on university computers, working well locally, but slower when using S3. They may reach out to us.

• Is pangeo forge on your radar?
  https://github.com/pangeo-forge/roadmap/ 
• https://github.com/pangeo-forge/staged-recipes/issues/new/choose
  → example pipeline

• Deepak

• Not much; xgcm cf-xarray summer internship program

• Ryan

• Xgcm; Pangeo gallery updates

Agenda:

• Agave platform demo / discussion

• What does the interaction between Pangeo and Agave look like?

• Agave is good at streamlining interactions between different platforms.
• Wouldn’t “deploy” pangeo on Agave. Would instead work alongside.
• Ex. multi-domain experiments

• Dask Array indexing intuitions (5-10 minutes, if time)

2020-09-23 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Martin Durant / Anaconda / @martindurant
• Jeff Sadler / USGS / @jsadler2
• John Clyne / NCAR / @clyne
• Tom Augspurger / Anaconda / @TomAugspurger
• Jim Bednar / Anaconda / @jbednar
• Zac Flamig / AWS / @zflamig
• Sebastian Alvis / UW / @salvis2
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Matthew Rocklin / Coiled / @mrocklin
• Deepak Cherian / NCAR / @dcherian
• Julius Busecke / LDEO / @jbusecke

60 Second Updates:

• Rich: Experimenting with accessing netcdf on a server; h5netcdf was much more efficient than the new netcdf4 `url#mode=bytes` direct access method
• Martin: Connecting fsspec with zarr for xarray. Should now be possible to open a zarr url directly from xarray.
• Jeff: Working on parallel ML training. Should we be putting GPUs on the Dask workers? Could run the notebook run as CPU, but then farm tasks out to the GPUs; seems cheaper, especially if the GPU nodes are spot instances. Issue that I started a few months ago (currently stale) (https://github.com/pangeo-data/pangeo/issues/772). Tom: could do via dask-gateway, but may get complicated. Maybe simpler to do it via kube spawner?
• John: Project Pythia (community Python training resource for the geosciences) web site up: https://ncar.github.io/ProjectPythia/
• Tom: Looking into DaskArray indexing issue affecting xarray users.
• Jim: New GeoViews bugfix release out. Been working on streamz+hvPlot a bit.
• Zac: Starting to play around with converting hrrr datasets to zarr.
• Sebastian: Putting together a talk on Pangeo Gallery for JupyterCon. Also working on terraform for Google Cloud.
• Charles: Working on rendering zarr metadata in the STAC browser. Trying to get more Google Cloud running functionality.
• Deepak: new xarray release fixes indexing with datetime[ns] with pandas>1.1. Chatted with Matt R & Hugo (Coiled) about Dask for oceanography.
• Julius:  xgcm transformation feature is pretty much ready to merge PR docs
• Ryan: Working on NSF on a 10-year plan.
• Brendan: xarray-spatial released with A* pathfinding on xarray objects, plus better focal analyses with different kernels

Agenda:

• Rich: for interpolating an oblique photo of a beach into a regular x/y grid in space, could we use Datashader? (https://datashader.org/user_guide/Grids.html) Jim: if you only want raster as output, perhaps; Datashader only generates regular raster output, though it does accept many input types.  Datashader can rasterize quadmesh data using the GPU and also trimesh data (not using GPU).  Jim:  so you may be able to specify your photo as a quadmesh then sample it into the output raster.  I don’t think anything besides raster input supports any interpolation other than nearest neighbor, so you’ll see the shape of the input pixels as quadrilaterals in the rendered raster, without any smoothing.

2020-09-16 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Tom Augspurger / Anaconda / @TomAugspurger
• Martin Durant / Anaconda / @martindurant
• Julius Busecke / LDEO / @jbusecke
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Jim Bednar / Anaconda / @jbednar
• Philipp Rudiger / Anaconda/ @philippjfr
• Zac Flamig / AWS / @zflamig
• Sebastian Alvis / UW / @salvis2
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Rich Signell / USGS / @rsignell-usgs
• Haiying Xu / NCAR/ @halehawk
• Chelle Gentemann / FI / @cgentemann
• Ryan Abernathey / LDEO / @rabernat

60 Second Updates:

• Tom: Dask changes in 2.26.0 that may impact Xarray. Changes to comparison operations (to accommodate pint (https://github.com/dask/dask/pull/6632). Indexing with a list of integers (https://github.com/dask/dask/issues/6646). Pangeo-forge stuff. Some movement on (de)serializing Numba’s dynamic ufuncs (https://github.com/numpy/numpy/pull/17289, https://github.com/numba/numba/pull/6234)
• Martin: async coming to zarr… slowly PR.
• Julius: xgcm transform (transform e.g. different vertical coordinates in ocean models) almost ready...Please try it out! (PR)(Issue)
• Joe: Pangeo Forge prototypes and schemas. And Scikit-downscale development. Oh, and Pangeo on Azure.
• Jim B: Ask Philipp. :-)  Well, ok, sending final Columbia/Anaconda report to Ryan today; ask him if you want a copy to see what our team was up to over the past year.
• Philipp: Fixing zooming issues in HoloViews and working on releasing hvPlot with exciting new features (see Agenda)
• Zac: Recently added NOAA World Ocean Database
• Sebastian: Got aws-examples set up on terraform-deploy, links for that set up in Z2JH’s Community Section, monitoring mostly ready on hubs / binderhubs. Looking to do some GCP / Terraform things.
• Charles: Working on CMIP6 documentation, set up basic GHA sync for GCS->S3 CMIP6 data, looking into Vue to make Zarr metadata viewer for STAC browser
  https://github.com/radiantearth/stac-browser/issues/44 
• Rich. No progress on Pangeo Forge coastal modeling flow because Filipe was moving this week.  Discovering there is lots of training.  Like this: https://lpdaac.usgs.gov/resources/e-learning/
  https://lpdaac.usgs.gov/resources/e-learning/getting-started-gedi-l2b-data-python/ (jbednar: nice!)
• Haiying: Want @rabernat and @jhamman approve numcodecs.zfpy PR at https://github.com/zarr-developers/numcodecs/pull/229 , Pangeo benchmarking is done, got results of NetCDF vs. Zarr on NCAR S3 object store
• Chelle: paper help with examples, aviso, Goddard citizen science cloud Joe Foster/Andrew Pawloski

• Looking for examples that use cloud data and OSS and cloud computing -- with published science results/impact --

• Ryan:

Agenda:

• Philipp: Demo .interactive

• 🙌🙌🙌
• Would be great to have an example gallery at http://gallery.pangeo.io/ 
• PR: https://github.com/holoviz/hvplot/pull/505
• More real-world examples

• Julius: CMIP6 examples
• World ocean database

• Julius: This would be a killer application for the interactive + link_sharing (I am thinking of visualizing watermasses based on a T/S selection)

• Xcube: https://xcube.readthedocs.io/en/latest/index.html 
• More ZFP discussion?

2020-09-09 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Martin Durant / Anaconda / @martindurant
• John Clyne / NCAR / @clyne
• Tom Augspurger / Anaconda / @TomAugspurger
• Julius Busecke / LDEO / @jbusecke
• Jeff Sadler / USGS / @jsadler2
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Anderson Banihirwe / NCAR / @andersy005
• Rich Signell / USGS / @rsignell-usgs
• Deepak Cherian / NCAR / @dcherian
• Chelle Gentemann / FI / @cgentemann
• Jim Bednar / Anaconda / @jbednar
• Orhan Eroglu / NCAR / @erogluorhan
• Matthew Rocklin / Coiled / @mrocklin

60 Second Updates:

• Charles: STAC sprint, working on STAC stuff, google cloud pub / sub + AWS SNS
• Martin: new release of fsspec / gcsfs / s3fs including bug-fixes after async rewrite, concurrent up/download for caching, etc.; paying attention to STAC / auth; intake base classes rewrite, make it easier to write new drivers (martin will write blog posts!)
• John Clyne: ~70 plotting examples (Jupyter Notebooks) on GeoCAT gallery:
  https://geocat-examples.readthedocs.io/en/latest/. Turning attention to plotting  unstructured grids (e.g. MPAS, IKON, CAM-SE)

• Question: integration w/ holoviews + datashader?  
  See Rich Signell’s Hurricane Ike notebook (triangular mesh) and also the EarthSim User Guide (HoloViz-based)

• Tom: broke and fixed binder, need to update dask gateway

• How to update pangeo gallery examples
• Ready to redeploy gcp jupyterhub
• Work on dask array slicing

• Julius working on vertical regridding in xgcm

• Needs atmospheric use cases
• Talk to GeoCAT team
• Discussion about the new feature: https://github.com/xgcm/xgcm/issues/222
• WIP PR: https://github.com/xgcm/xgcm/pull/205

• Jeff: doing deep learning experiments with on-prem cluster; wants to do a pangeo training for USGS; binderbot
• Joe: working on Pangeo-forge...

• Demo feedstock hooked up to CI/Prefect:
  https://github.com/pangeo-forge/terraclimate-feedstock
• Example pipeline and meta.yaml PR:
  https://github.com/pangeo-forge/staged-recipes/pull/12

• Anderson:

• gave lighting talk on Zarr at STAC sprint
• working on merging custom spec for intake-esm into  , revive work from a few months ago
• Working on cheyenne jupyterhub deployment, decouple hub from compute nodes (rather than login node)

• Rich

• Met with folks working for FEMA working on unstructured flood monitoring
• How can we go serverless?

• Deepak: Got a PR into dask which eagerly slices numpy arrays, maybe solves some memory blowup issues
• Chelle:

• working with NASA HQ on series of NASA PI workshops to train PIs to use python, starting Jan. 1 (contact Chelle for more details)
• updated the AVISO dataset, more data, different chunk scheme
• What was funded from Pangeo from ACCESS grant

• Jim: The HoloViz team is preparing new releases of HoloViews, Panel, and hvPlot. Philipp Rudiger to demo new functionality here next week (was on holiday today). Very interested in following up about NCAR’s needs for plotting unstructured grids.
• Orhan: now leading GeoCAT team!

Agenda:

• STAC Sprint Update (Charles, Rich, Anderson, Joe)

• PySTAC: pythonic way of making stac objects
• STAC index: we are already in it!
  https://stacindex.org/catalogs/pangeo#/?t=catalogs
• Google Earth Engine public catalog/
• How do we want to use STAC?
• Something like zarr: https://www.npmjs.com/package/@planet/dynamic-tiles

• Performant rendering of chunked arrays
• Very similar encoding to zarr

• Vector tiles
• STAC data sprint starts tomorrow

• XGCM Vertical Regridding Update

• Extensive discussion of vertical regridding algorithms:
  https://github.com/xgcm/xgcm/issues/222 
• PR in progress:
  https://github.com/xgcm/xgcm/pull/205 
• Two basic algorithms

• Linear interpolation

• Requires monotonic coordinate

• Conservative regridding

• Handles non-monotonic coordinates
• Conserves extensive quantities
• Requires “cell bounds”

• Challenge: how to make it “automatic” for user while also being accurate
• What would be useful for us: use cases from the community. We already have several use cases for ocean data; need atmosphere (e.g. pressure). Weigh in here: https://github.com/xgcm/xgcm/issues/222   
• Deepak: is there a way to dispatch to GPU vs. CPU?

• Jim: you can use atomic primitives supported by numba and cupy. Ask Jon Mease or look at his code in Datashader for more details.

• IRIS work on vertical regridding
•  https://github.com/SciTools-incubator/python-stratify
• Guvectorize does not work on Dask distributed
• Orhan update from GeoCAT:

• Use case: vertical interpolation of hybrid coordinates to pressure coordinates

• Uses NCL fortran code: See
  https://www.ncl.ucar.edu/Document/Functions/Built-in/vinth2p.shtml
• Main challenges are “argument management” and parallelization

• Suggested to grab some NCL example use cases

• (See link above)

2020-09-02 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Martin Durant / Anaconda / @martindurant
• Jim Bednar / Anaconda / @jbednar
• Sebastian Alvis / UW / @salvis2
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Tom Augspurger / Anaconda / @TomAugspurger
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Alex Rybchuk / CU Boulder & NREL / @rybchuk
• Deepak Cherian / NCAR / @dcherian
• Matthew Rocklin / Coiled / @mrocklin
• Ryan Abernathey / LDEO / @rabernat
• Zac  Flamig / AWS / @zflamig
• Tim Crone / LDEO / @tjcrone

60 Second Updates:

• Rich: testing out AWS “no tears” HPC cluster, see 10 min demo here.
• Martin: multifile context and cache-download; find() shortcut
• Jim: Philippe has completed the “.interactive” functionality for xarray, pandas, etc.; see https://github.com/holoviz/hvplot/pull/505 . He’s scheduled to come to the next Pangeo meeting to demo it once it’s released. Also added support for using Bokeh’s color mapping with Datashader output, to get proper colorbars, hover, etc., even when using Datashader features like histogram equalization and “spreading” (making isolated pixels visible). Should be available in next Bokeh and Datashader releases.
• Sebastian: upgrades for pangeo cloud federation, examples for TerraForm
• Joe: Vacation last week, working on Pangeo on Azure and Pangeo Forge,
• Tom: working on design issues for pangeo forge , merging staging to prod on cloud federation , reshape issue to allow user-controlled rechunking
• Charles: attending pangeo-forge meetings, Rhodium group collab, working on documentation for how to access CMIP6 data on GCS / AWS
• Alex: works at NREL, primarily user, wants to keep tabs on user side of things [Ryan suggests contributions to the Pangeo gallery would be very welcome]
• Deepak: merged support for duck dask arrays. This will make Xarray(Pint(dask)) work. Also see others’ efforts at https://github.com/xarray-contrib/pint-xarray
• Ryan:  posted a new idea for a small project, proxied access to cloud storage
  https://discourse.pangeo.io/t/get-involved-in-pangeo-entry-points-for-new-contributors/643/8 
• Zac: Just lurking
• Tim: Teaching Ryan’s class this semester.  OOI camera data => Azure.

 Agenda

• Update our working groups (some getting a bit stale -- or dead)

• http://pangeo.io/meeting-notes.html
• https://discourse.pangeo.io/t/updating-the-pangeo-working-groups/819
• Data WG has met its goals

• Outcome is ESIP best practices
• Time to wind down

• Education working group

• Ping Rob Fatland, he was roughly in charge

• Do we need a viz working group?

• Updates on Pangeo-related funded projects

• 3 NASA ACCESS proposals, machine learning one funded, infrastructure one from UW and devseed or element 84 one not fund
• Jupyter

• NSF report due for original Pangeo NSF grant (no cost extension has been requested)
• Updates on cloud operations
• Pangeo-Forge?
• Cloud access proxy

• Zac worked on this problem at NIH
• Pre-signed URLs

• Bake a new URL with credentials
• Gen3 supports this workflow through Fence/Indexd (https://gen3.org; https://github.com/uc-cdis/fence ; https://github.com/uc-cdis/indexd)
• Use FUSE to work with these

• https://github.com/mitre/fusera
• https://github.com/uc-cdis/gen3-fuse 

• Works on azure and google as well!
• But you have to have an external catalog
• Or run a FUSE service
• Could be a fsspec spec as well
• These urls expire and need to be refreshed

• There is a proposal to allow Intake server to do signing of http links, act as S3/gcs  signer/broker?
• Zac: this mode of access doesn’t exactly qualify as “open data”
• Matt: not hard to just do IAM roles

• Chunking:  

• Rich: why does Pangeo use 100MB chunks and COG folks use 0.1MB chunks?
• Martin: workers submit requests to read a chunk of zarr from s3, wait until that is read, then read another.  Would be more efficient for workers to submit concurrent requests.  
• Ryan: COG does byte range requests, while Zarr reads entire chunks.  TileDB is a hybrid  with some advantages.  

• STAC!!!

• Stac sprints happening, desire to  transition our catalogs to stac/intake
• https://github.com/charlesbluca/pangeo-datastore-stac/ 
• Obstacles: lack of representation for zarr
• Where does esm / datacube STAC extension stand?

• https://github.com/radiantearth/stac-spec/pull/361
• https://github.com/radiantearth/stac-spec/issues/781
• https://github.com/pangeo-data/pangeo-datastore/issues/107 

• https://medium.com/radiant-earth-insights/join-the-cloud-native-geospatial-outreach-day-and-sprint-286f6fd553c3 

2020-08-26 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• James A. Bednar / Anaconda / @jbednar
• Jessica Scheick / UMaine (UW unofficially) / @JessicaS11
• Sebastian Alvis / UW / @salvis2
• Chelle Gentemann / FI / @cgentemann
• Tom Augspurger / Anaconda / @TomAugspurger
• Matthew Rocklin / Coiled / @mrocklin
• Jeff Sadler/ USGS / @jsadler2
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Rich Signell / USGS / @rsignell-usgs
• John Clyne / NCAR / @clyne

60 Second Updates:

• Jim: Starting a new project with Rich Signell on interactively specifying curvilinear quadmesh grids in a web GUI (Panel/HoloViews/Bokeh)

• Ryan: see https://github.com/cf-convention/discuss/issues/5

• Jessica: working with Sebastian on Pangeo gallery from icepyx

• https://github.com/icesat2py/icepyx/pull/125

• Sebastian: working on prometheus / grafana
• Chelle: CCMP in master catalog (atmo), testing updated aviso right now. Rich and I connected re: USGS/NASA community ‘how to cloud’.
• Tom: grafana / prometheus for hubs; map-overlap, upstreaming pangeo help chart; looked into Prefect (it’s somewhat complicated); released rechunker
• Matt: working on Coiled things; trying to release Dask more frequently -- is this good?
  https://github.com/dask/community/issues/84 
• Jeff: ML training in parallel on HPC using GPU --can it be done on cloud?

• Could put GPU’s on the workers ... but would be very expensive

• Charles: working on documentation for accessing CMIP6 data in GCS and S3

• Documentation website: https://pangeo-data.github.io/pangeo-cmip6-cloud/
• Documentation repo: https://github.com/pangeo-data/pangeo-cmip6-cloud

• Rich: chunk-store xarray PR was merged, can move forward with the netCDF / Zarr layer

• Links to PRs: Zarray chunk store: https://github.com/pydata/xarray/pull/3804
  Need more community support for this concept on the Zarr side here:  
•  https://github.com/zarr-developers/zarr-python/issues/556

• John Clyne

• NCAR + Albany kicking off new python training project:
• https://docs.google.com/document/d/1HzOYGRsMnnE8KQIeU_nR6DSjLNj7pSoK8V9_kn0_rHw/edit?usp=sharing
• http://gallery.pangeo.io/

• Ryan

 Agenda

• Update on Pangeo Forge meeting

• Pangeo Forge Coordination Meeting
• Contribute use cases:
  https://github.com/pangeo-forge/staged-recipes/issues/new/choose
• NASA DAAC Use Cases (@chelle will dig them up)

• L2 Data

• How do you put L2 data together in a meaningful way when the lat/lon do not line up in a regular way.
• Should it just be left in NetCDF?
• There is a Swath common data model for NetCDF: See slide 17 in this Unidata presentation
• A granule = 5 minute space of time in the satellite’s path (~100-200 MB?)
• Fixed number of scans and granules per scan
• Possible dimensions (cross-track, long-track, and orbit)
• Similar to IceSat datasets - Element 84 and Development Seed have been working on putting ICESat-2 datasets into cloud optimized formats
• Or maybe we *want* a dataframe format (like parquet)

• Parquet can hold images (arbitrary binary data)
• Wouldn’t index the image

• Two use cases:

• People creating products: want all the data to create a derived product (e.g. L3, new algorithm, etc.)
• People who want to examine a specific region

• Discourse Get Involved in Pangeo: Entry Points for New Contributors
• How to get funding to support student internships

• Brian Arbic proposal addendum
• USGS convenience checks
• Corporate sponsors (Coiled, etc.)
• Dask/NumFocus

• GOES netcdf

• https://nbviewer.jupyter.org/github/oceanhackweek/ohw20-tutorials/blob/master/10-satellite-data-access/goes-cmp-netcdf-zarr.ipynb
• Rich’s Zarr-hacking approach: https://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314
• Ryan’s GOES notebook: https://github.com/awslabs/amazon-asdi/pull/12
  https://nbviewer.jupyter.org/github/awslabs/amazon-asdi/blob/d59987afaa4c928456413e3e4ac2d18f8d07c5d8/examples/goes16-precip-pangeo/notebooks/GOES-16_precip_Pangeo.ipynb

2020-08-19 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Tom Augspurger / Anaconda / @TomAugspurger
• Sebastian Alvis / UW / @salvis2
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Jim Bednar / Anaconda / @jbednar
• Martin Durant / Anaconda / @martindurant
• Matthew Rocklin / Coiled / @mrocklin
• Chelle Gentemann / Farallon Institute / @cgentemann
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Rich Signell / USGS / @rsignell-usgs

60 Second Updates:

• Tom: PR to dask to fix xarray align issue; Pangeo helm chart--move to Dask?; cloud working group call on batch workflows
• Sebastian: looking at updates to hubploy, Z2JH Terraform examples, icepyx examples on pangeo-gallery
• Charles: copying CMIP6 public dataset from GCS to S3, working on synchronization and cataloging challenges; almost ready to announce
• Jim: new funding for Bokeh and Panel; Philipp finally ready to start working on “interactive” functionality for xarray. See the PR at xarray/issues/3709 and a draft example notebook and chime in if you have opinions or want to try out prototypes.
• Martin: async in s3fs + gcsfs moving forward; fsstore in zarr PR needs review; intake improvements, refactoring
• Matt: pass
• Chelle: working with Charles on CCMP data
  https://nbviewer.jupyter.org/github/oceanhackweek/ohw20-tutorials/blob/master/10-satellite-data-access/goes-cmp-netcdf-zarr.ipynb
  Met with NASA / NOAA cloud strategy leads, project being approved to help scientists move onto cloud (openscapes model); NOAA EDMC q from Robert McGuire: how do we move onto cloud? Pangeo is what people want; pushback from some people about infrastructure, some want to just provide credits
• Joe: Pangeo forge roadmap, zarr appending bug, STAC sprint,
  Benoit Bovy updated xpublish a lot  https://github.com/xarray-contrib/xpublish
• Rich: sprint with the ESIP cloud-optimized data best practices group, working on Cloud Hosted Data Optimization Knowledge Base, document.   NOAA / USGS contracting, want to get on the cloud, need to get together with Chelle
• Ryan: https://medium.com/@rabernat/advising-and-collaborating-during-a-pandemic-and-sabbatical-ca9531b82b6d

 Agenda

• Install prefect server on our Hubs?
  https://github.com/pangeo-data/pangeo-cloud-federation/issues/696

• Chelle: “Trond” might be able to help
• Tom:

• What about user management? That’s cloud feature
• Could do this as part of Jupyterhub as a service
• Matt: should jupyter support batch scripts?

• Rstudio Pro: https://rstudio.com/products/rstudio-server-pro/ 
• Talk to Chris White

• Catch up for JHub user storage limits:
  https://github.com/pangeo-data/pangeo-cloud-federation/issues/654

• Deploy our own NFS server
• Get UIDs for all users

2020-08-12 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Sebastian Alvis / UW / @salvis2
• Orhan Eroglu / NCAR / @erogluorhan
• Matthew Rocklin / Coiled / @mrocklin
• Anderson Banihirwe / NCAR / @andersy005
• Nicholas George/USGS/@nicholaskgeorge

60 Second Updates:

• Rich:

• OceanHackWeek. Going to copy PR to get RStudio onto USGS Jupyterhub.
• White paper for best practices on cloud-optimized data. Please join the hack session this afternoon 3-5pm ET:

ESIP - Cloud Data Optimization Knowledge Base Workshop

• Wed, Aug 12, 2020 3:00 PM - 5:00 PM (EDT)
• Please join my meeting from your computer, tablet or smartphone.
• https://global.gotomeeting.com/join/333323397
• You can also dial in using your phone.
• United States: +1 (872) 240-3212
• Access Code: 333-323-397
• Pangeo-forge got funded! The above hack session can turn into a user guide.

• Orhan:

• Revising documentation for GEOCAT, putting in Contributors Guide for GEOCAT Analysis Toolkit. Will provide links to that when it is done. For now, check the following GeoCAT examples gallery with a lot of Xarray usage:

• https://geocat-examples.readthedocs.io/en/latest/

• Matthew:

• Hiring a life sciences fellow.
• Doing bi-weekly tutorials for dask. Could we do something like that for Pangeo (xarray, Anderson nominated)? Using Binder.

• Anderson:

• Working on Intake / Intake-ESM, hope to release new version this week / early next week. Dask-jobqueue, streamline building process of Docker images, open PR under review.

• Nicholas:

• Visualize geographical data, merging different sources.

• Sebastian:

• OceanHackWeek.
• Pangeo-binder PR to put Prometheus / Grafana on the binderhubs, maybe the Pangeo Hubs later.
• Looking into Z2JH-K8s Terraform Contribution

 Agenda

• Intake / Intake-STAC

• In the process of merging with STAC. Made a different spec before knowing about STAC.
• Creating ESM extension to STAC
• Upcoming STAC sprint

• Dask Scheduling Updates

• Someone thinking about different architecture? Out-of-notebook memory?
• https://blog.dask.org/2020/07/21/faster-scheduling

• Setting up JupyterHubs with Terraform

• https://medium.com/pangeo/terraform-jupyterhub-aws-34f2b725f4fd
• https://github.com/pangeo-data/terraform-deploy/tree/blog-post
• https://hubploy.readthedocs.io/en/latest/howto/hubploy-deploy-jupyterhub-repo-setup.html

• Live Training Thoughts

• Is great and all, but what if we had something like Khan Academy? Mostly want problems to work through, not prebuilt notebooks. Feedback (ie are you wrong are right) would also be nice.
• Dask is doing some online tutorial thinking https://github.com/dask/community/issues/57

• Matthew thinks it’s been going well for Dask, encourages Pangeo involvement.

• Should be money for NCAR to make training materials. Geoscience focus. Matthew knows some people that he can connect to for help on an online learning platform.

2020-08-05 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Tom Augspurger / Anaconda / @TomAugspurger
• Jim Bednar / Anaconda / @jbednar
• Sebastian Alvis / UW / @salvis2
• Martin Durant / Anaconda / @martindurant
• Julia Kent / NCAR / @jukent
• Nicholas George/USGS/@nicholaskgeorge

60 Second Updates:

• JimB: Writing docs for Param. Preparing presentation for Oceanhackweek.
• Sebastian: Oceanhackweek coming up. Preparing terraform/grafana/prometheus/ binder infrastructure.
• Martin: Working on getting async stuff and s3fs work ready - needs docs before release. Dask group is running online binder-based tutorials, including one later this week (see twitter @Dask_dev).
• JuliaK: NCAR finished SIParCS summer internships translating NCL gallery into Python, growing the Python gallery 10X. Sad to see them go!
• Nicholas: Been working on Panel DEM dashboard with Rich, somewhat helped by Jim.
• Tom: Debugging rechunker performance.

2020-07-29 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Jim Bednar / Anaconda / @jbednar
• Matthew Rocklin / Coiled / @mrocklin
• Sebastian Alvis / UW / @salvis2
• Tom Augspurger / Anaconda / @TomAugspurger
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Brendan Collins / makepath / @brendancol
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Scott Henderson / UW eScience / @scottyhq
• Chelle Gentemann / FI / @ChelleGentemann
• Deepak Cherian / NCAR / @dcherian
• Martin Durant /
• Daniel Rothenberg / ClimaCell / @darothen (daniel@climacell.co)

60 Second Updates:

• JimB: Presented HoloViz at ESIP, now preparing a HoloViz presentation for ocean hackweek.
•  Sebastian: went to ESIP, now setting up some stuff for escience ocean hackweek
• Tom: updated to dask-gateway 0.8, all of the envs need to be updated, worker dashboards now working. Public grafana dashboard (http://grafana.us-central1-b.gcp.pangeo.io/grafana/d/a_khcwhWk/dask?orgId=1&refresh=5s). Looking into rechunker performance issue.
• Joe: gave talk at ESIP about xarray roadmap, Earthcube jupyter meets earth workshop; pangeo-forge
• BrendanC: working on Xarray-Spatial and feeling guilty about not attending community meetings.

• Follow up with Ryan + Jonny Kingslake

• Charles: working on CMIP6 data pipeline and organization
• Scott: ESIP, NASA EOSDIS overview of work on NASA ACCESS project (link)
• Chelle: enjoyed ESIP! Presenting to NASA / NOAA leadership on cloud strategy--how to enable their scientific communities to take advantage of cloud data; working on MUR rechunking

• NOAA wants to figure out what NASA is doing so that they can invest properly.  Also talking with NCAR.
• Should we write a whitepaper?  Review monday 8/3, meeting date just updated to 8/12, so a little more time to prepare.  Maybe a one-pager?  Yes.  suggestion - what are the 3 biggest barriers to wider adoption of scientists using cloud datasets?  Access to compute? Python? Knowledge gap?
• Closest existing thing: https://medium.com/pangeo/closed-platforms-vs-open-architectures-for-cloud-native-earth-system-analytics-1ad88708ebb6
• Biggest barrier is access to cloud.  It takes a while today to get access to HPC resources.  Increasing access here would be welcome.
• 2i2c could be relevant here: http://2i2c.org/

• Deepak: mostly listening
• Daniel Rothenberg: jumping in after long absence
• Matthew Rocklin: working on coiled (let me know if you want invites).  Working on design for lower overhead: Faster Scheduling
• Martin Durant: async s3fs is in.  This should help make things like rechunker faster.  We probably need to change downstream libraries like Zarr

 Agenda

• Pangeo Forge updates (Ryan and Joe)

• https://github.com/pangeo-forge/pangeo-forge
• We are seeking input on use cases for pangeo-forge:
  https://github.com/pangeo-forge/staged-recipes/issues/new/choose
• Started a couple of weeks ago
• Using Prefect for ETL management
• If anyone who want data in the cloud can help fill out one of these issues it would help us understand the kind of workflows that this should support
• Should we also be thinking about other datasets that are being put into the cloud in non-optimized formats?  For example by NASA

• Yes, that’s probably in scope.  This is especially important if the data isn’t in a cloud-optimized format.  If it was already optimized then we would just drop it into a catalog.
• If there is a dataset that you care about and would like to be in the cloud, then go ahead and raise an issue

• Rechunker updates (Ryan)

• https://github.com/pangeo-data/rechunker
• Apache Beam support! https://github.com/pangeo-data/rechunker/pull/34
• Puzzling performance problem on s3:
  https://github.com/pangeo-data/rechunker/issues/35

• Dask tasks for reading / writing from s3 are about 10x slower than they should be
• Need some sort of worker profile information.  Tom is planning on looking at this later today.

• Update on CMIP6

• CMIP6 data is currently in google cloud, oriented primarily around access using intake-esm
• Enumerated using CSV files / esm collection spec
• CSVs have limitations for cataloging
• Hopefully this is eventually a special case of pangeo-forge, but CMIP6 is special enough to special-case

• Brendan: Regridding needs in Pangeo? (Agenda item added by Jim after hearing Brendan saying he was working on some interpolation/regridding support in xarray-spatial)cc

• Interested in numba-ising ESMF.
• Brendan doesn’t have much experience with ESMF, but does have plenty of experience converting things into Numba.  Would be happy to have a follow up call with people who have more context
• We should document the algorithms that we need, which might require going through the Fortran code
• Good reference:
  http://www.earthsystemmodeling.org/esmf_releases/regridding_status_noheader/esmf_5_3_0_regridding_status.html
• One challenge is that the information you need in order to do regridding well goes beyond Xarray’s regridding model.  Elements like faces and such.  Some sort of data structure beyond Xarray is required.  This also comes up in xgcm, which needs to integrate different quantities.  There is some space here for a more general convention to this problem.
• The algorithms are pretty complex and there is a lot of inertia here.  This might be a big lift.  You might also want to look at reprojection stuff in GDAL and rasterio warp module. This might be a good place for xarray-spatial to start.  There is also a rio-xarray project, which has maybe wrapped in warping, but hasn’t reimplemented it in a Numba or Dask-friendly way.
• General question about Numba.  It’s neat that your project provides a good example of Numba-accelerating a domain-specific algorithmic package.  We’ve run into issues with Numba gufuncs and serialization

• https://github.com/dask/distributed/issues/3450
• https://github.com/numba/numba/issues/4314
• https://discourse.pangeo.io/t/numba-guvectorize-and-distributed/758
• Lots of our functions work well on dask arrays due to shared API, or by using map_overlap, not using guvectorize or anything similar yet.  We have our mechanisms to spread custom numba functions around with Dask.
• This really walks through what works and what doesn;t:
  https://nbviewer.jupyter.org/github/xgcm/fastjmd95/blob/master/doc/fastjmd95_tutorial.ipynb
  Deepak, does this gel with your experience?

2020-07-22 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Ryan Abernathey / LDEO / @rabernat
• Tom Augspurger / Anaconda / @TomAugspurger
• Jim Bednar / Anaconda / @jbednar
• Thomas Moore / CSIRO/ @Thomas-Moore-Creative
• Scott Henderson / UW eScience / @scottyhq
• Patrick Quinn / NASA EED-2, Element 84 / @bilts
• Dona Banerjee / UC
• Zac ?

60 Second Updates:

• Rich: ESIP session tomorrow (link?)
• Jim: Presenting at ESIP tomorrow. Lots of work happening on
  makepath/xarray-spatial: Raster-based Spatial Analysis for Python (Datashader-friendly, xarray-based raster analysis/computation tools)
• Tom: working on ESIP talk, rechunker, public ingress for grafana on GCP cluster
• Thomas saying G’day from Australia - checking in. Thanks for RECHUNKER!!!, continuing to advocate for Pangeo across Australian data science space and leveraging great work by CSIRO folks like Dougie Squire and Paul Branson.  Action item: explore “hero” rechunker example on DCFP forecast datasets (CSIRO HPC or NCI)
• Scott: Gave ESIP presentation on intake-stac last week (bit.ly/esip-intake-stac). Giving NASA EOSDIS webinar next week.

• Ryan: look at https://github.com/charlesbluca/pangeo-datastore-stac

• Patrick: ESIP cloud computing cluster
• Ryan:

• https://github.com/awslabs/amazon-asdi/pull/12
  https://github.com/awslabs/amazon-asdi/blob/d59987afaa4c928456413e3e4ac2d18f8d07c5d8/examples/goes16-precip-pangeo/notebooks/GOES-16_precip_Pangeo.ipynb

 Agenda

• Hub User Management
  https://github.com/pangeo-data/pangeo-cloud-federation/pull/675 

• Current process: fill out form, then manually add to github teams
• New script seems to work!

• Pangeo-forge:

• https://github.com/pangeo-forge/pangeo-forge

• 4:30pm ET: Strategy for cloud-optimized data “best practices” document: issues arising from the ESIP Cloud Computing Cluster (Ana Privette, Patrick Quinn, Rich Signell).   Notes from Pangeo Data Working Group, Notes from  ESIP Cloud Computing Cluster.  Repo for cloud optimized data best practices document.  

• http://gallery.pangeo.io/repos/earthcube2020/ec20_abernathey_etal/cloud_storage.html

2020-07-15 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Rich Signell / USGS / @rsignell-usgs
• Jim Bednar / Anaconda / @jbednar
• Ryan Abernathey / LDEO / @rabernat
• Tom Augspurger / Anaconda / @TomAugspurger
• Jeff Sadler / USGS / @jsadler2
• Nicholas George/USGS Intern/@nicholaskgeorge
• Chiara Lepore/LDEO/@chiaral

60 Second Updates:

• Rich: Sessions at ESIP meeting, Introducing Nicholas
• Nicholas: Computer and Electrical Engineering at Cornell, working on dashboard solution for visualizing terrain data from COGs
• JimB: New work from Philipp: Added basic 3D volume regridding to Datashader; meant for server-side regridding of volumetric data. Panel now supports ipympl, allowing interactive Matplotlib plots in Panel apps and dashboards.
• Tom: Working on deployment stuff, dask distributed + gateway, should go directly to distributed 2.21; grafana dashboard doesn’t show info for workers; rechunker
• Jeff: Scipy poster (Streamflow prediction using network-aware deep learning through Python in the cloud). Was looking into Prefect for parallel GPU workflows (replicates of expensive training tasks).

• Some discussion on serializing tensorflow linked at https://github.com/dask/dask-ml/issues/696

• Chiara: Working on new ocean.pangeo.io (https://us-central1-b.gcp.pangeo.io/) dask gateway issues compared to kubernetes
• Ryan:

 Agenda

• 5 min demo by Nicholas George, new Pangeo intern, on progress visualizing USGS terrain data in COG format

• Link to discourse post?
• Link to notebook is here  https://discourse.pangeo.io/t/get-involved-in-pangeo-entry-points-for-new-contributors/643/4?u=rsignell 

• Pangeo training: Lessons learned from SciPy xarray tutorial (and other tutorials)

• Blowing through Jupyter cells was too fast for people (what Joe did)
• Writing out the cells covered very little material (what Ryan did)
• Use Binderbot as tool for training on their own?
• Ryan’s class

• Full class starting from nothing

• https://earth-env-data-science.github.io/intro
• Python
• Matplotlib
• Pandas
• Xarray
• Doesn’t even get to Pangeo
• Types cells live and pauses for exercises
• Evidence that it is effective

• There’s a mismatch between the effort what it really takes and what people think it takes

• It can’t be done in a 1-4 hour tutorial
• It may take a college class a semester

• 1 hr tutorial

• Maybe just able to inspire people to go try to do it on their own

• What potential is there for self-paced instruction?

• Right now there is no feedback

• Some people are focused on training and education

• Kevin just got a large Earth Cube proposal to

• We should be using scientific knowledge of how to effectively teach and teach online
• A lecture is the wrong pace for everyone
• We should probably not start people by creating an xarray dataset by scratch

• Give them a cool dataset and have them plot within a few minutes
• Maybe start with Rocklin’s 3 minute video

• Status check on new cluster deployment

• Dask gateway issues

• Demo from Chiara
• Heavy user of ocean.pangeo.io
• Doing a complicated workflow

• Developed xcape: https://github.com/xgcm/xcape/
  (wrapped fortran code for processing gcm output)
• Workflow involves preprocessing
• Operate on one year of 3-hourly atmospheric data; pretty smooth with dask_kubernetes
• Same analysis on dask_gateway (same number of workers and memory): time would vary between 500 and 2000 s
• 
• All tasks stuck in one worker; cluster stalls
• Occurs with other separate tasks that are simpler too
• Works okay for some years so it doesn’t seem to be the workflow that is the problem
• Sometimes it works through all the tasks getting stuck on the time but not most of the time
• Didn’t happen with Dask Kubernetes
• The memory is not filling up…
• Stalling was delayed by increasing memory but still occurred
• Maybe some leaking of some sort - some sort of cumulative thing
• It doesn’t crash so the cluster just keeps trying and costing money
• Other issue: feels like there is less control over the cluster compared to the previous deployment. No there is no “shutdown” button

• “Create cluster” should be easy to fix but “list of cluster” is not as clear

• So much has changed so it’s hard to know what it is
• Could be the scheduler that is hanging

• Scheduler has 2 cpu cores and 4 GB of memory. Maybe small?
• Maybe a memory leak in the scheduler: https://github.com/dask/distributed/issues/3898

• Temporary bucket
• Packages

• https://discourse.pangeo.io/t/gibbs-sea-water-module-missing/739/2

• Binder nbgitpuller is broken:

• https://github.com/pangeo-data/pangeo-binder/issues/154

• Rechunker update

2020-07-08 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Martin Durant / Anaconda / @martindurant
• Sebastian Alvis / UW / @salvis2
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Deborah Khider/ USC ISI/@khider

60 Second Updates:

• Sebastian: icesat2 hackweek post mortem (insights)
• Martin: working async filesystem, prototype was http but backtracking a bit, working on gcsfs, and s3 is next
• Deborah: teaching a capstone on datascience

• Looking for a guest lecture on geoscience / data science collaboration, post here: https://discourse.pangeo.io/

• Joe: gcp cloud migration, other cloud migration / pipelining, prepping xarray tutorial for SciPy

 Agenda

• Hackweek infrastructure / insights for hackweeks / classes

• NCAR ran a hackweek recently (CMIP6), one-off for now, open to collaboration
• Deborah thinks science-specific programs might be more interested, general data science programs might have too broad a focus

• NFS Quota:

• https://github.com/pangeo-data/pangeo-cloud-federation/blob/49ae1cf8e9496b07ff2fd181cbc0a960418c9dbe/deployments/gcp-uscentral1b/config/common.yaml#L13-L22

2020-07-01 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Kevin Paul / NCAR / @kmpaul
• Jim Bednar / Anaconda / @jbednar
• Sebastian Alvis / UW / @salvis2
• Chelle Gentemann / FI / @ChelleGentemann
• Tom Augpsurger / Anaconda / @TomAugspurger
• Kevin Donkers / UK Met Office / @kaedonkers

60 Second Updates:

• Kevin P:  Plug for Anderson’s latest blog post

• https://ncar.github.io/xdev/posts/writing-multiple-netcdf-files-in-parallel-with-xarray-and-dask/

• JimB: Went camping last week, so not much to report. Preparing for SciPy next week, including some blog posts.  Working on CUDA/Numba issues for Datashader.
• Sebastian: Start up search on how to enforce user storage limits on hubs
• Chelle: did interview on open science / cloud science for AWS last week.  Working on talk for ESIP.  Working on a NASA project to get scientists in the cloud.

• Pangeo Gallery
• NASA initiative (2i2c)
• NSF EarthCube Pythia Project

• Tom:

• Wrapping up ocean.pangeo.io transition
• Ryan is rewriting the main site’s content:
  https://github.com/pangeo-data/pangeo/pull/780

• Kevin D:

• Not much to report from UKMO
• Pursuing Microsoft Azure to provide aspects of Pangeo as a service
• Investigating dataverse (data repository tech: https://dataverse.org/)

• Provides citable DOI for datasets
• Seems to support geo data, but not sure about scaling
• Maybe not sure about whether it is designed for big data
• Works well for tabular/csv data, but not fully functional for large netcdf
• Can link out to data stored elsewhere (might be a solution for big data)

 Agenda

• Education resource for moving scientist to the cloud

• Who’s interested?  Kevin P, Chelle, Sebastian, JimB (e.g. happy to review some materials and upgrade/simplify their viz code) …

• Kevin P will put together email and meeting

• What needs to be done?  More than just the Pangeo platform

• How do you enable community change?  Make the transition?

• Hackweek?

• Barrier:  Where do you work?  → Hub
• Barrier:  Diversity in languages for analysis (R, IDL, Matlab, etc.)

• Tutorials that demo how fast/efficient Python can be in cloud
• Develop tutorials around datasets
• NCAR makes “real-world example” tutorials, but want a stronger transition from software users to software contributors

• Get “enthusiastic, technically reluctant” scientists to develop the tutorials / modules themselves

• 2 types of tutorials: “this is how this thing (Xarray, Dask, etc.) works”, and “this is how you do this on this data”
• Addresses 2 different communities: contributors, and uses

• How to interact with and connect to the Pangeo community:

• Stack, Discourse, etc.

• How NASA and NSF funds science encourages siloing

• Leadership is invested in changing this model

• NASA science mission divisions is pressing open source software
• Implementing, starting with Earth Sciences
• This education effort (moving scientists to cloud) is one of their first efforts
• Want to develop a funding mechanism for open source projects
• There is resistance to sharing code

• Social platform can be useful to get scientists to work together to find solutions

• Office Hours (continued)

2020-06-24 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle:

• Kevin Paul / NCAR / @kmpaul
• Ryan Abernathey / LDEO / @rabernat
• Martin Durant / Anaconda / @martindurant
• Deepak Cherian / NCAR / @dcherian
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Rich Signell / USGS / @rsignell-usgs
• Tom Augspurger / Anaconda / TomAugspurger
• Julia Kent / NCAR / @jukent
• Sebastian Alvis / UW / @salvis2
• Orhan Eroglu / NCAR / @erogluorhan

60 Second Updates:

• Kevin:  Possibly more good news on the grant front… NCAR may be awarded a new EarthCube award to build a centralized Python training site.  Still contingent on some things, though.  Partnership with Brian Rose and Kevin Tyle at UAlbany.
• Ryan: OGC Presentation (Plenary and MetOcean working group), ESGF Presentation, Cloud refactor, 2i2c
• Martin: reported on async of HTTP filesystem; plan to backport to fsspec; interesting PR about accessing a zip file on S3 containing 120K images as an intake dataset; planning on some separation of concerns work in the base intake package to simplify plugin development (next week or later)
• Deepak: CF-Xarray; Office Hours Discourse post
• Joe: Cloud refactor, intake-stac -> intake, xpublish -> xarray-contrib, xarray-tutorial for scipy, presented at EarthCube All Hands Meeting
• Rich: Found out at ESIP Cloud Computing Cluster that Chris Lynnes (NASA) and Patrick Quinn and Andrew Pawloski (Element84) have further developed the NetCDF4/HDF5-via-Zarr for NASA use,  also this cluster just started  working on a cloud data optimization best practices document and are hoping for community input.  Pangeo tutorial checking?

• This has been merged (zarr):
  https://github.com/zarr-developers/zarr-python/pull/557 
• This is stalled (xarray):
  https://github.com/pydata/xarray/pull/3804

• Tom: Drafted blog post on pangeo multi-cloud, updated environment for pangeo-notebook, adding metrics to pangeo deployments.
• Julia: NCAR Python Tutorial for (almost absolute) Beginners

• https://ncar.github.io/python-tutorial 

• Sebastian: pangeo gallery (waiting for landsat 8 notebook, starting coordination on icepyx examples), improving cluster / jupyterhub configurations for hackweek template, looking into prometheus / grafana outputs

 Agenda

• Office Hours Discussion: see https://discourse.pangeo.io/t/pangeo-office-hours/669

• Getting people of diverse backgrounds to participate

• Survey of existing Pangeo community?
• Internships

• Rich: can write a check to pay for an intern, but how do you find candidates?  SciPy collects ethnicity information, maybe we can reach out to BIPOC participants and create an internship at SciPy
• Can possibly learn from the SIParCS program at NCAR
• See Ryan’s project ideas post in Discourse (link?)

• Need to increase gender diversity, too; just increase participation

• Ryan: Let’s just start the Office Hours and get the ball rolling, then we can adjust based on what we learn
• Deepak:  Will work on a plan for the 1st Office Hour meeting

• Ongoing cloud refactor update

• All the hubs are down; most have been archived and won’t come back; consolidating the hubs to reduce cost; transitioning to a new type of K8s cluster; transitioning to a new auth system (moving to auth0)
• Metrics / Monitoring (Tom)
• User Storage Limits (Sebastian)

• Linux hack has not been working.
• Google Filestorage has a provisioner tutorial that uses the NFS-Client Provisioner to dynamically provision PV / PVCs. Within this, there appear to be options for enabling quotas, but they are tricky. See this issue and these docs. The docs show an argument that looks promising and there is an example that uses a flag to enable xfs quotas, but it doesn't appear to be Helm'd.
• sebialvis@gmail.com, you are now a project owner on  GCP project pangeo-181919

• 2i2c

• https://2i2c.org/

2020-06-17 2000 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle:

• Deepak Cherian / NCAR / @dcherian
• Sebastian Alvis / UW / @salvis2
• Tom Augspurger / Anaconda /@TomAugspurgeri
• Martin Durant / Anaconda / @martindurant
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Ryan Abernathey / LDEO / @rabernat
• Jim Bednar / Anaconda / @jbednar
• Rich Signell / USGS / @rsignell-usgs
• Kevin Donkers / UK Met Office / @kaedonkers

60 Second Updates:

• Deepak: working on CF xarray
• Sebastian: pangeo gallery + IceSat2 Hackweek
• Tom: last week worked on temporary bucket storage
  Bokeh 2.1 release broke Dask dashboard; should work with 2.0.2
• Martin: async backend for fsspec; how to maintain current blocking API? https://github.com/intake/filesystem_spec/pull/310
• Charles: working on moving CMIP6 to AWS via public dataset program
• Ryan: #ShutDownSTEM
• Jim: Working on a new HoloViews release and accompanying blog post (in progress)
• Rich: posted a project on New Contributors page (1m terrain data in COG on AWS) for using xarray-spatial, working on moving IOOS model output to cloud
• Peter K will give a talk comparing TileDB to Zarr at ESIP meeting, user stories for Azure ML
• Deborah Khider - Data Science at Information Sciences Institute; recently working with Xarray--automatically couple models of hydrology and ?; another project related to organizing Paleo data: LPiD

 Agenda

• Deepak: new package cf-xarray following up on discussion in
  https://github.com/pangeo-data/pangeo/issues/771

• https://github.com/xarray-contrib/cf-xarray
• https://cf-xarray.readthedocs.io/
• Example notebook:  
  https://cf-xarray.readthedocs.io/en/latest/examples/introduction.html
• PRs welcome!
• Kevin: overlap with some Iris features.
• Refactor xgcm, xesmf
• People like the design

• Follow up from #ShutDownSTEM:

• https://medium.com/pangeo/pangeo-and-shutdownstem-b234643290e0?source=activity---post_recommended_rollup
• https://discourse.pangeo.io/t/get-involved-in-pangeo-entry-points-for-new-contributors/643/5
• Sebastien: UW hackweeks have been effective; could we do one
• Office hours:

• Deepak volunteers
• What is a good model?

• Kubernetes has them: https://github.com/kubernetes/community/blob/master/events/office-hours.md (starting in 30 min)

• KubeCluster permissions issue:

• https://github.com/pangeo-data/pangeo-cloud-federation/issues/619
• Tldr:

• Dask Kubernetes needs special kubernetes permissions to create worker pods.
• We gave these permissions to the “daskkubernetes” Kubernetes service account.
• We just switched to the “pangeo” Kubernetes service account to support temporary bucket storage
• Do we grant this permission to “pangeo” SA? Do we move ahead with Dask Gateway?

• Scratch Storage Bucket
• Migration of ocean.pangeo.io accounts

• https://discourse.pangeo.io/t/migration-of-ocean-pangeo-io-user-accounts/644/14

2020-06-10 2000 GMT (1pm PDT, 4pm EDT)

Special #ShutdownSTEM discussion on racism         

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Tom Augspurger / Anaconda / @TomAugspurger
• Jim Bednar / Anaconda / @jbednar
• Jessica Scheick / University of Maine; icepyx / @JessicaS11
• Deepak Cherian / NCAR / @dcherian
• Anna-Lena Deppenmeier /NCAR / @aldepp
• Julia Lowndes / NCEAS UCSB, Openscapes / @jules32
• Bia Villas Boas / Scripps Institution of Oceanography
• Alex Kerney / Gulf of Maine Research Institute / @abkfenris
• Dhruv Balwada / University of Washington / @BalwadaDhruv
• Erin Robinson / ESIP / @esip_Erin
• Diana Gergel / Rhodium Group/Climate Impact Lab / @dgergel
• Emilio Mayorga / University of Washington / @emiliom
• Thomas Bearpark/ Climate Impact Lab
• Sebastian Alvis / UW / @salvis2
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Jessica Hausman / PO.DAAC/JPL / @jhausman    (my internet hates me right now)
• Anderson Banihirwe / NCAR / @andersy005
• Rich Signell / USGS / @rsignell-usgs
• Simon Greenhill/Climate Impact Lab and Energy Policy Institute at UChicago /@simondgreenhill
• Gui Castelão / Scripps Institution of Oceanography / @castelao
• Ned Molder/ USGS FORT Science Center/
• Brian Westra/ University of Iowa
• Dylan Hogan / Climate Impact Lab / @dylanhogan
• Julius Busecke / Princeton University / @jbusecke

Notes:

If the Pangeo community wants to actually contribute to this effort a long term goal/strategy has to be devised NOW, or else people will get busy and this will end up simply being a lip service moment/zoom call (Remember that it took only 3 months for America to forget about CoronaVirus and move onto other things, even though ~800 people are still dying everyday in the US). It is important to come up with actionable and intentional plans, while there is still momentum. 

• Rich: sea change   https://seachange.aaas.org/  a US implementation of a successful UK program to certifySTEMM diversity (STEMM Equity Achievement)
• Erin: do group silent notetaking +1

Brainstorm on actions

Careers

• Use Pangeo network to help recruit Black people into geospatial / tech fields
  https://discourse.pangeo.io/t/get-involved-in-pangeo-entry-points-for-new-contributors/643/3  +1 +1
• …
• Empower graduate students and other minorities to effect change → take their reports seriously instead of gaslighting them +1+1+1
• Be involved in the whole review process for hiring candidates. Black people suffer from disproportionate imprisonment and this heavily affects their ability to get good jobs after. Don’t let people throw out applicants because of the “criminal” background if there’s a chance that it’s a part of the industrial prison complex. +1 +1
• How about a targeted Pangeo group for Black and other underrepresented groups where we set a goal of membership  and we try to be particularly welcoming and inclusive.  Inclusion must be cultivated, and we Pangeans should have the ability to do that.  At the USGS, we can write $2500 convenience checks.  Mini -internships instead of or in addition to a group?  (I’m thinking here how scary it is on SO for newbies, and even github issues can be intimidating) +2
•  Partner with orgs like outreachy/google soc to provide structured training/mentorship (with pay). Virtual opportunities might help reach a broader group. +1 +1 +1+1

• Is it possible to ask Pangeo’s google connections to allow Pangeo community leaders to become GSOC mentors? Why not use the framework that is already in place?

• How to actively recruit african americans graduate students in oceanography / marine sciences / geosciences, specifically for OceanHackweek? Are there institutions that have graduate Oceanography / Marine programs, Masters or PhD, that we should reach out? Are there lists of such institutions and specific departments?
• Pangeo presence at SACNAS +1
• How to educate ourselves? It’s easy to identify and recognize the prejudice and pressure that I suffered, but hard to identify what other groups experience. Some of us learned a lot in the last week what was wrong in a few things that were considered normal before, and now we can see as clearly wrong. I think that this click is crucial. We know that there are problems, but can we clearly point out what are the problems. Some people work and study racism, what could we learn from them. Where is this material that we could learn from. Maybe a solution is a way to share between ourselves good material on the subject. We can act with more efficiency if we are better prepared.

• Here are a few resources to get started

• https://qz.com/257474/what-riding-my-bike-has-taught-me-about-white-privilege/
• https://notimeforsilence.org/
• https://www.ibramxkendi.com/how-to-be-an-antiracist-1
• https://www.indiebound.org/book/9781580058827

Infrastructure

• Providing public-focused “easy ways in” to your particular research area, so that people who have not been on a pipeline into it their whole lives can get started with something that may excite them (binder, etc., with descriptions that can reach the general public and show what research is like for people who did not necessarily grow up in that environment).
• Mentorship for Black colleagues and groups of mixed colleagues to learn together and be allies (MPOWIR like mentor group aimed at BIPOC?)
• Reach out to HBCUs and offer to partner to provide cloud infrastructure (+1)+1 +1

• Jupyterhubs
• Data Hosting
• Science Gateways community institute https://sciencegateways.org/ 
• NSF Big Data Hubs

• Engage “external” communities who might otherwise not necessarily be considered as stakeholders in the research, or in the application of the research. So, communities with seismic risk, flood risk, exposure to industrial pollution… This may overlap with the next bullet
• Reach out to advocacy groups for BIPOC communities, offer services, data aggregation, computing resources, etc.
• Find out what some of the barriers to access are and think about how they can be mitigated (e.g. computer/internet access or reliability) +1 

Education

• Have you heard of skype a scientist? It’s an opportunity to connect to (lower income district?) schools, and when you say you’re a data scientist they are interested in examples for coding -> maybe a super easy pangeo tutorial could help people get into this  +3 +1 https://twitter.com/SkypeScientist, https://www.skypeascientist.com/ 

• At Gulf of Maine Research institute we’ve been doing our own version of this ‘Scientist to Go’. We have a large community and education team, so we’ve reached out to teachers that we usually work with for sessions with our research team. 

• Girls Who Code https://girlswhocode.com/

• There are multiple locations around the world, it’s focused on middle and high school females, but they’re always looking for volunteers to help

• Pangeo office hours? +1

• Mentor people in open source contributions to software they use;  +4 +1

• There is also a place for mentorship of folks using the tools. If they turn into contributors, that’s fantastic, but having a known person that they can turn to for help goes a long way. Mentoring folks on how to use the tools also is less likely to require as much monetary or time investment.

• Make the pangeo stack accessible to people that don’t have “local” expertise they can turn to?
• Individual invitations work
• Partner with folks from Vanguard STEM
• How do we find mentees? Are there Big Brother/Little Brother programs?
• Mentoring is time-intensive, how does it scale?

• Mentorship networks: mentor mentees to create more mentors. Mozilla Open Leaders model.
• Mentoring in the form of creating a safe and supportive space can be quite important - knowing that “if I speak up here, I will be listened to and empowered, not ridiculed” can be a huge resource for reducing fear of retaliation and anxiety +1

• Bystander training  
• Educational training videos hosted by people from underrepresented groups? +1
• Lab meeting discussions: here is a google doc scaffolding and reading list: https://twitter.com/gavin_fay/status/1270768571220623361 
• Take a long look at your field’s history and find if there are any prominent black figures that have been scrubbed from it and not given credit for the field’s development. Promoting them is only more accurate and can help others to feel represented and encouraged to take careers in those fields.
• Help undergrads understand the process of entering grad school. For example there is money available to fund grad school/research in geoscience. +1
• Work with VanguardSTEM: https://conversations.vanguardstem.com/ (leader of the #ShutdownSTEM strike today) +1
• Engagement with communities to support and encourage STEM at early levels (middle school), follow up with longitudinal outreach and awareness. More than “one-time” events. +1
• Language is important - how does the use of words like “urban” influence perception?
• Targeted hackweeks to underrepresented communities

Resources & Training, other groups taking action

• 500 women scientists: https://500womenscientists.org/updates/2020/6/1/take-action  +1000

• I learned to “Capitalize the “B” in Black when talking about Black people or communities” as one concrete thing from this post

• Earth and Space Science resources: https://docs.google.com/document/d/1pW7MbGmx3IgFl_mLfNlsGTNBY1W92r-0r6sK5_YBLdg/edit?fbclid=IwAR0rTXjeimAbdDUsatses2eXAJBYzfAhePO6I6mMWoBbJQNKeyVLwTLCF78
• Particles for Justice

Papers on diversity in geosciences:

https://www.frontiersin.org/articles/10.3389/fpsyg.2019.01305/full

https://www.jstor.org/stable/24861948?seq=1

 Things not to do:

• Don’t depend on Black colleagues to educate you +1+1
• Don’t do nothing and hope for the best.

When buid

2020-06-03 1600 GMT (9am PDT, 12pm EDT)        

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Tom Augspurger  / Anaconda / @TomAugspurger
• Sebastian Alvis / UW / @salvis2
• Haiying Xu / NCAR / @halehawk
• Martin Durant / Anaconda / @martindurant
• Kevin Donkers / UK Met Office / @kaedonkers
• Rich Signell / USGS / @rsignell-usgs
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Orhan Eroglu / NCAR / @erogluorhan
• Scott Henderson
• Jeff Sadler / USGS / @jsadler2

60 Second Updates:

•  Ryan: thinking mostly about diversity and inclusion.  
• Tom: AnacondaCon going on this week.    GPU on Google figured out.   Working on temp storage: https://github.com/pangeo-data/pangeo-cloud-federation/issues/610
• Sebastian: updating tutorial notebook, using gateway instead of dask-kubernetes (Ryan suggests moving stale notebooks (e.g. tutorial notebook) to the pangeo gallery)
• Haiying: pypi package added for numcodecs zfp,  working on osx builds. Need sync control between delete file and create a new file, otherwise cannot create a new file correctly. Here is the issue log: https://github.com/pangeo-data/benchmarking/issues/42
• Martin: New release of Intake and related packages, to be followed by blogpost of what’s new; starting working on async in fsspec (e.g. could be useful for commands like ls, but of most benefit in downloading large files in chunks and many files in parallel, as well as batch copy/rm operations)
• Kevin: air quality data added to Azure; working on blog post on ARD on cloud using Zarr + TileDB; running workshops with Microsoft to understand how Azure ML can be used with Pangeo
• Rich: working with intake, trying to figure out how to work with data from ERDDAP from the provider side to best serve Xarray users
• Joe: two blog posts:

• Environments (led by Tom):
• DL + GPU post (led by Jeff):
• GKE auto node pools
• Gave talk to NASA HQ about Pangeo, positive feedback from program managers
• How do you scalaby do shapefile clipping with rasters?

• Tried https://corteva.github.io/rioxarray/stable/examples/clip_geom.html, this worked best but wasn’t great.
• Maybe look at: https://github.com/pygeos/pygeos/?
• https://github.com/scikit-geometry/scikit-geometry

• Orhan: Back to working on z-to-sigma mapping
• Jeff: Blog post on ML environment out, trying to get this going on USGS pangeo, help from Filipe on conda build.  Sam Congdon from USGS will participate in Monday Cloud call.
• Scott:  working on auth on binder , trying auth0 on JH, all applications use the same auth, would prefer to use github auth? (Ryan, once we pick one, would facilitate tracking users across resources.   How close are we to shutting down the JHs, going to binder exclusively?   Scott: will take a while.   Need persistent storage.   Joe: need shared disk to avoid massive resource)

Agenda Items:

• Pangeo “entry points” and brainstorming about how to increase / support diversity within Pangeo community

•  Ideas to improve:  Pangeo interns at Pangeo orgs like ESIP, USGS, NCAR
• Generate a list of projects that would be suitable for pangeo internship projects
• Haiying: Utilize Google “Code In” for high school students?

• https://edu.google.com/code-with-google/?modal_active=none&story-card_activeEl=enhance-any-subject ?

• Orhan: Collaborate with Google Earth Engine?  Tyler Erickson?
• Sebastian:  Utilize Data Science for Social Good? https://escience.washington.edu/dssg/

• Ryan’s funded center:  Funding for 5 hires,  have 3 offers accepted,  Paige Martin?  Hillary Scanal(sp?)    Julius Bussecke,  starting in October time frame.   Wants to hire a devops person, but will contract if can’t find someone

2020-05-27 2000 GMT (1pm PDT, 4pm EDT)        

Name / Institution / @GitHub handle:

• Ryan Abernathey / LDEO / @rabernat
• Jim Bednar / Anaconda / @jbednar
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Rich Signell / USGS /@rsignell-usgs
• Tom Augspurger / Anaconda / @TomAugspurger
• Scott Henderson / UW / @scottyhq
• Orhan Eroglu / NCAR / @erogluorhan
• Sebastian Alvis / UW / @salvis2
• Jonathan MacCarthy / @jkmacc-LANL
• Martin Durant / Anaconda / @martindurant
• Deepak Cherian / NCAR / @dcherian

60 Second Updates:

• Jim: Released Datashader w/ Dask rectilinear and curvilinear quadmesh support.
  Now working on Panel for multiple (some non-geo) clients, polishing and improving templates, simple indicators, progress/busy display, etc.  hvPlot about to release ASAP.
• Charles: Working on migrating our intake catalogs to STAC,Assessing how hard this would be
• Rich: converted a stack of geotiffs to Zarr, 20x speedup of time series analysis of 180 image stack.  
• In-browser Zarr image views:  
  https://twitter.com/trevmanz/status/1265377097981329423  (Ryan, is this the actual browser of zarr images?  http://viv.vitessce.io/)
• Tom: wip with Rechunker, draft blog post
• Scott: worked on Pangeo GPU blog post, interested in scratch space
• Orhan: pass
• Sebastian: working on icesat2 hackweek stuff, hubploy documentation
• Jon MacCarthy: exploring zarr and xarray:

• Parallel with neuroscience:
  https://predictablynoisy.com/xarray-explore-ieeg

• Martin:

• Dask maintenance
• Intake release soon

• Ryan:  xarray CZI grant: https://twitter.com/i/status/1265680953701134338
• Deepak: starting a cf-xarray package.

Agenda Items:

• Scratch storage:

• https://github.com/pangeo-data/pangeo-cloud-federation/issues/610
• Rich: USGS has a solution, users can only write bucket/username tag, like usgs-pangeo/rsignell
• Short term solution:

• Global scratch bucket
• Attach to pangeo service account: https://github.com/pangeo-data/pangeo-cloud-federation/blob/6a8a702cf1257aedb86679ac7d76a43fe5845567/pangeo-deploy/values.yaml#L54
• Retention policy N days
• New environment variable PANGEO_SCRATCH, e.g.

• gs://pangeo-scratch/{{ username }}

• Discourse for Earth System Modeling in the cloud?

• YES! Free to use Pangeo discourse for this. Ryan can create a “topic”

2020-05-20 1600 GMT (9am PDT, 12pm EDT)        

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Tom Augspurger / Anaconda / @TomAugspurger
• Sebastian Alvis / UW / @salvis2
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Chelle Gentemann / FI / @cgentemann
• Rick Wagner / UCSD / @rpwagner
• Rich Signell / USGS / @rsignell-usgs
• Jim Bednar / Anaconda / @jbednar (late)
• Martin Durant / Anaconda / @martindurant (late)
• Jeff Sadler / USGS / @jsadler2

60 Second Updates:

• Ryan: rechunking
• Tom: EarthCube NB, rechunking
• Sebastian: EFS Drive for IceSat2 hackweek
• Charles: working actively on CI features for catalog, discovered intake-esm compatibility issues
• Chelle: developing talk on the velocity of science, pangeo-gallery demo dev failing at reading json file, ERA5 issues: Add some ERA5 data? · Issue #49 · pangeo-data/pangeo-datastore, MUR still slow, need to resolve, need to consolidate time chunk - Ryan will talk to Aimee/me about this
• Rick: Experience with globus, security, jupyterhub. Getting globus working with more kubernetes clusters.
• Rich: Trying to speed up median calculation, good talk with NATO undersea research institute, has extra money
• JimB: Datashader with Dask quadmesh finally ready for release after some minor cleanups; any day (hour?)  now!.  Full ipywidgets/Bokeh/Panel integration now finally ready; Philipp Rudiger starting blog post.
• Martin: NaN
• Rick: UCSD, formerly Globus, wants to help with auth and other stuff
• Jeff: finishing up Pangeo ML blog post

Agenda Items:

• Rechunking!

• Discourse issue:
• Tom’s package: https://github.com/TomAugspurger/rechunk
• Ryan’s package: https://github.com/rabernat/rechunkerc
• Issues:

• How to deal with large temporary datasets?

• Temporary bucket
• Dask filesystem (only for medium-sized datasets)
• Bucket with policy to delete files after time

• https://cloud.google.com/blog/products/gcp/optimizing-your-cloud-storage-performance-google-cloud-performance-atlas

• Fsspec layer to hash filenames

• xESMF update
• Globus / auth0

• Follow up with Globus

• Temporary buckets
• Intake ESM compatibility issues

2020-05-13 2000 GMT (1pm PDT, 4pm EDT)        

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Martin Durant / Anaconda / @martindurant
• Kevin Paul / NCAR / @kmpaul
• Tom Augspurger / Anaconda / @TomAugpsurger
• Sebastian Alvis / UW / @salvis2
• Rich Signell / USGS / @rsignell-usgs
• Joe Hamman / NCAR & CarbonPlan / @jhamman
• Orhan Eroglu / NCAR / @erogluorhan
• Anderson Banihirwe / NCAR / @andersy005
• Austin Kootz /NCAR/ @pilotchute

60 Second Updates:

• Martin: fsspec refactoring, dask-parquet interaction
• Kevin: https://github.com/NCAR/xtar

• https://github.com/mxmlnkn/ratarmount

• Tom:

• Working on earthcube demo notebook: https://github.com/pangeo-data/multicloud-demo
• Also working on zarr dask optimizations

• Sebastian: working on getting jupyterhub ready for icesat2
• Rich: running a fancy AWS parallel cluster based on Jiawei’s example, trying simulations in the cloud, liking Scott’s landsat notebook, playing with rasterio + dask
• Joe: working on Earthcube notebook for climate downscaling
• Orhan: Walking through Pangeo stack via notebooks and working on https://github.com/erogluorhan/remap_z_to_sigma

• Might want to look at:
• https://github.com/cspencerjones/xlayers
• https://github.com/jbusecke/xarrayutils

• Anderson: working on Earthcube notebook on intake-esm with Charles Blackmon-Luca, running containers on Cheyenne
• Ryan:

• https://github.com/JiaweiZhuang/xESMF/issues/98

Agenda Items:

• Presentation from Rick Wagner

• https://github.com/rpwagnerbuilds/z2jh-gcp/tree/master

• Update on cloud auth situation (Ryan)
• fsspec with ratarmount (ratarmount indexes tarball during mount)
• Debugging Dask issues on Pangeo JupyterHubs (Rich)

• Scheduler dies after completing task
• How can users learn to debug this sort of problem?
• Ryan: need better training material on dask, can’t expect it to be a magic black box

2020-05-06 1600 GMT (9am PDT, 12pm EDT)        

Name / Institution / @GitHub handle

• Joe Hamman / NCAR&CarbonPlan / @jhamman
• Ryan Abernathey / LDEO / @rabernat
• Tom Augspurger / Anaconda / @TomAugspurger
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Rich Signell / USGS / @rsignell-usgs
• Sebastian Alvis / UW / @salvis2
• Jeff Sadler / USGS/ @jsadler2
• Deepak Cherian / NCAR / @dcherian
• Scott Henderson / UW / @scottyhq

60 Second Updates:

• Joe:

• Got Pangeo deployed on fancy autoprovisioning k8s cluster
• EGU presentation on ML using Pangeo cloud deployment: https://meetingorganizer.copernicus.org/EGU2020/EGU2020-11332.html

• Tom: working on blog post about environments;
  https://docs.google.com/document/d/14m-TNi2R4VaTI0g2vy15LBRGDkur2B21wiAdrTt6nBg/edit?usp=sharing
  
• Charles: working on datastore, updating CI, cloud function for exposing metadata
• Rich: launched USGS pangeo
• Sebastian: published blog post, working on hubploy docs
• Jeff Sadler: working on Pangeo ML blog post
• Deepak: no updates
• Ryan: Zarr OGC

 Agenda:

• Getting to Pangeo 2.0: roadmap discussion

• Rich: sagemaker + dask cloudprovider: eliminate hubs
• Joe: two issues:

• Who’s going to do it?
• What is the best architecture?

• Colab / sagemaker are limited, not under our control
• Should have a sprint to try to implement binder with auth

• Scott: on a path to simplify AWS infrastructure

• Eliminate hubs
• Eliminate duplicate buckets
• Hubs have common environments, that’s an advantage

• Third option: eliminate hubs entirely, use local notebook + dask gateway

• Blockwise ensemble estimators (Tom)

• https://nbviewer.jupyter.org/gist/TomAugspurger/88ea870feb550e481652bbe701e5c750

• Rechunking discussion (Tom)

• https://discourse.pangeo.io/t/best-practices-to-go-from-1000s-of-netcdf-files-to-analyses-on-a-hpc-cluster/588/9

• Time to tear down hydro.pangeo.io, hub.pangeo.io (will cover at cloud ops WG call next week).
• IceSat-2 hackweek thoughts

2020-04-29 2000 GMT (1pm PST, 4pm EST)

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS/ @rsignell-usgs
• Jim Bednar / Anaconda / @jbednar
• Sebastian Alvis / UW / @salvis2
• Martin Durant / Anaconda / @martindurant
• Scott Henderson / UW / @scottyhq
• Deepak Cherian / NCAR / @dcherian

60 Second Updates:

• Rich S: tried out Bokeh 2.0 with notebooks
• JimB: New “interactive” functionality in preparation for xarray; see https://anaconda.org/philippjfr/xarray_interactive and https://github.com/pydata/xarray/issues/3709 
• Sebastian: wrote blog post for Terraform, getting ready for icesat2 hackweek
• Martin: Intake GUI visualizations, refactoring intake to remove diff btw. Entry and datasource, working on fsspec backends
• Scott: working on dask gateway hard cloud stuff, proxies, etc., needs some help
• Deepak: xarray team started biweekly dev meetings
• Ryan: Working on binderbot. Adding a cmips6 gallery, refactoring what Henry Drake started. Applying to formalize the zarr v2 spec as a community standard, with help from OGC.

 Agenda:

• JimB: Demo https://anaconda.org/philippjfr/xarray_interactive (in development): easily inject Panel widgets at any point in an xarray pipeline of method calls, to make arbitrarily interactive apps using only Xarray syntax (no need to learn or look up how to do things in HoloViews, and now able to use any Xarray methods, not just the ones that are handled by HoloViews).

2020-04-22 1600 GMT (9am PDT, 12pm EDT)        

Name / Institution / @GitHub handle

• Joe Hamman / NCAR&CarbonPlan / @jhamman
• Jim Bednar / Anaconda / @jbednar
• Tom Augspurger / Anaconda / @TomAugspurger
• Kevin Donkers / UK Met Office / @kaedonkers
• Rich Signell / USGS / @rsignell-usgs
• Nick Mortimer/ CSIRO O&A/@nickmortimer
• Haiying Xu / NCAR / @halehawk
• June Wu / @green-juniper
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Jeff Sadler/ USGS/@jsadler2
• Sebastian Alvis / UW / @salvis2
• Valentin Pesendorfer / UN WFP / @valpesendorfer

60 Second Updates:

• Joe:  playing with GKE node auto-provisioning
• JimB: NASA and (2019) ERDC projects completed; happy to send final reports to anyone who cares to skim to see what we did. Together they cover most non-Pangeo work from our group from March 2019 to March 2020. Still haven’t released Datashader with Dask-ified quadmeshes.
• Rich: will meet with Jeff to try to get the ML pangeo up on the USGS cloud
• Kevin: UK Met office working on RAMP, assisting with COVID response, providing met data; looking for help and collaboration

•  Medium blogpost announcement (best place for information)
  https://medium.com/informatics-lab/met-office-and-partners-offer-data-and-compute-platform-for-covid-19-researchers-83848ac55f5f
•  Discourse announcement
  https://discourse.pangeo.io/t/uk-met-office-covid-19-response/569
•  For updates, please subscribe to Google Groups mailing list
  https://groups.google.com/a/informaticslab.co.uk/forum/#!forum/covid19
• Email
  covid19@informaticslab.co.uk