Pangeo Weekly Meeting Minutes

https://columbiauniversity.zoom.us/j/6902819781

2019-10-09 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle

• Joe Hamman / NCAR/ @jhamman
• Ryan Abernathey / LDEO / @rabernat
• Julia Signell / Anaconda / @jsignell
•  Tom Augspurger / Anaconda / @TomAugspurger
• Jim Crist / Anaconda / @jcrist
• Naomi Henderson/ LDEO/ @naomi-henderson
• Jebb Stewart /NOAA/ @jebbstewart
• Matt Long/NCAR/@matt-long
• Julia Kent/NCAR/@jukent
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Zac Flamig / AWS / @zflamig
• Anderson Banihirwe/NCAR/andersy005
• Martin Durant / Anaconda / @martindurant (arrived late)

60 Second Updates:

•  Joe

• Releases (xarray, dask-jobqueue, dask-kubernetes???) and new Pangeo Stacks Images (https://github.com/pangeo-data/pangeo-stacks/pull/84)
• dask -gateway on pangeo chart: https://github.com/pangeo-data/helm-chart/pull/102 
• Pangeo-cloud-federation: reducing resource limits (https://github.com/pangeo-data/pangeo-cloud-federation/pull/442)

• Ryan: pushing hard on CMIP6 hackathon; putting out fires with google cloud; getting ready to spin up Moore proposal hiring
• Julia: releases across HoloViz - datashader, panel, holoviews done. hvplot and geoviews up today or tomorrow.
• Tom

• New optimizations: https://github.com/dask/dask/pull/5453,  https://github.com/dask/dask/pull/5451
• Picking up dask-gateway deployment things.

• Jim C: Released dask-gateway 0.5.0, working on getting dask-gateway up on the Pangeo cluster.
• Jebb: lurking, looking into code for problem related to image generation to talk with Jim B.
• Matt Long: new to Pangeo telecon, leading CMIP6 hackathon, working on intake / intake-esm for CMIP data
• Julia Kent - sci-parks (sp?) proposal, get two interns to help NCL users figure out how to produce pretty figures in python
• Charles BL: Working on CMIP cataloging problem
• Zac F: just looking!
• Anderson:

Agenda Items

• Lots of CMIP6 hackathon stuff

• New work from @andersy005 on ESM-Collection-Spec: https://github.com/NCAR/esm-collection-spec – this is our best attempt at adapting a STAC-style catalog to climate model data, and we are excited about it. Born of necessity.
• Charles is working on a pure javascript browser for the ESM Collection catalogs, demo here: https://charlesbluca.github.io
• @jhamman is prepping a new pangeo-stacks image here: https://github.com/pangeo-data/pangeo-stacks/pull/84/files 2

• Discussion on how to configure ocean.pangeo.io to prevent users from launching huge dask clusters (see Some usage analysis of ocean.pangeo.io for discussion)

• Improvements to dask dashboard scalability - open dashboard issues?
• Need dask gateway!
• https://github.com/dask/distributed/issues/2979 has some discussion
• Zac - replace jupyterhub? - https://github.com/uc-cdis/hatchery 

2019-10-02 2000 GMT (1pm PDT, 4pm EDT)

Name / Institution / @GitHub handle

• Joe Hamman / NCAR/ @jhamman
• Rich Signell/ USGS / @rsignell-usgs
• Ryan Abernathey / LDEO / @rabernat
• Jim Bednar / Anaconda / @jbedna
• Xu Chen / Florida State University / @chenfsu
• Matthew Rocklin / NVidia / @mrocklin
• Tom Augspurger / Anaconda / @TomAugspurger
• Jim Crist / Anaconda / @jcrist
• Jebb Stewart /NOAA/ @jebbstewart
• Julia Signell / Anaconda / @jsignell
• Zac Flamig / AWS / @zflamig
• Dax Soule/ CUNY/ @daxsoule
• Jeff Sadler/USGS/@jsadler2
• Rob Fatland/UW/@robfatland

60 Second Updates:

• Joe - at BIDS this week kicking off Pangeo meets Earth project, working on GPUs on Pangeo, making fun progress on xsd (downscaling with xarray)

• https://github.com/pangeo-data/pangeo-cloud-federation/issues/425 (maybe solved already?)

• Rich - giving Pangeo tutorial at IOOS code sprint next week in Ann Arbor, working with Zac Flamig from AWS open data on NWM data.  He is trying parquet, I’m doing zarr.
• Ryan - Pushing hard on CMIP6 in the cloud / CMIP6 hackathon (https://github.com/intake/intake/issues/417) / https://discourse.pangeo.io
• Jim B: Datashader now has GPU support (about to be merged, not yet released): https://github.com/pyviz/datashader/pull/793 (20-40X speedup over single-core Pandas, 5-10X over 12-core Dask, for points and lines; not yet implemented for regular or irregular grids.)
• Matthew - Dask documentation and marketing efforts
• Xu - Using Xmitgcm to analyzing MITgcm outputs. Wondering if I could contribute some functions to Xmitgcm package. Just install pangeo on FSU HPC and try to play with it.
• Tom - Working on limited form of backpressure. Learned about Dask’s blockwise today. Will help with memory issues.
• Jim C: Working on dask-gateway usability and docs, pushing for new release later this week.
• Julia - hvplot pandas entrypoint (merged! - not released yet…) - cleaning up xrviz for latest holoviz.
• Zac - converted 2017 NWM archive to parquet for experimentation. Handing off to Rich
• Jeff - working on ML-notebook. Pangeo intro to Brazil’s National Water Agency tomorrow

• https://github.com/pangeo-data/pangeo-stacks/pull/83

• Dax - working on 3D data visualization, mayavi, etc.
• Rob - first weekly hack at UW Data Science Studio, working on POETs, will be looking for reviewers, cloud bank
• Jebb - New to Pangeo

Agenda items:

• NOAA's Jebb Stewart plans/challenges using Pangeo

• Already using dask, zarr, xarray, etc. Working on visualization and information systems, tools that work with large datasets, interactive, minimizing moving data around.
• Using AWS, some troubles with AWS lambda and the size of the libraries.
• Doing conversion to zarr, uploading to S3 for themselves. Not public though.
• Pain points: not really getting /processing data. The hardest part is generating the actual images. Making 50-100 static images within ~1 second is hard (maybe matplotlib?). For one model, generating up to ~100 timesteps. Jebb to open  an issue documenting the issues and asking for advice, and Jim B volunteered to talk through where the bottleneck might be.

• Changes to weekly call (maybe move to monthly frequency) and focus more on working group meetings? (Joe)

• Rich: people might miss meeting
• Matt: planning meeting vs. hangout / checkin meetings
• Joe: https://github.com/pangeo-data/pangeo/pull/722

• GPUs on Pangeo cloud hubs (Joe)

• https://github.com/pangeo-data/pangeo-cloud-federation/issues/425
• Shared GPUs

• Intake ESM discussion

Short term coordination:

2019-09-25 1600 GMT (9am PDT, 12pm EDT)

Name / Institution / @GitHub handle

• Tom Augspurger / Anaconda / @TomAugspurger
• Zac Flamig / AWS / @zflamig
• Joe Hamman / NCAR / @jhamman
• Martin Durant / Anaconda / @martindurant
• Julia Kent / NCAR / @jukent

60 Second Updates:

• Tom

• Deployed dask-gateway on staging.hub.pangeo.io, next thing to try is multi-region setups
• Next up is dask-distributed memory issues

• Zac:

• Introducing himself. From AWS open data team (weather & climate lead).

• Joe:

• Tried out new spec-cluster dask-jobqueue PR (just got merged, thanks Matt)
• Talking with folks about NWM ML applications
• CESM-LENS binder running on AWS: https://github.com/NCAR/cesm-lens-aws

• Martin:

• Not too much. Just checking in.
• Caching work is still ongoing. About a week out.

• Julia K:

• Ran tutorial with NCAR scientists. Identified common problems with xarray in break out sessions. WRF datasets weren’t compatible. Developing workflow to get metadata from sidecar file.

• Rich:

• Met with head of NCEP to give pangeo demo.
• Made HIR demo, forecast data from zarr. But it was broken (dependency issues). Sorting out environments is still painful.

Agenda items:

• Zarr format discussion

• Customers are wondering about best practices for cloud storage.
• What’s the best way to convert things? What are the best parameters for Zarr (compression, chunking, etc.)
• The best practices aren’t written down anywhere right now.
• NetCDF is adding a backend. HDF5 also has similar parameters (chunking, compression) but people tend not to think about them as much.
• Zarr datasets tend to be larger. No need to break things up as much.
• David’s blogposts: https://medium.com/pangeo/continuously-extending-zarr-datasets-c54fbad3967d, https://medium.com/@david.brochart/transposing-chunks-in-zarr-datasets-f906a23b411f
• Rich: tend to have ~100Mb chunks. There’s some overhead to connecting to the blob storage. Balance that against having too large objects in a single task.
• Need to think about how chunking aligns with compute / access pattern.
• Joe: For CESM: made roughly square chunks, which is OK for all use cases, but not optimized for a specific use case.
• Martin: the spec is relatively simple, well-defined and the metadata human-readable (json)
• For very large datasets, the chunk size needs to grow, just to keep the number of chunks down. Especially when using Dask, the distributed scheduler tends to struggle with 100s of thousands of tasks.
• There’s a pangeo data working group would ideally write these best practices.

• Joe to make sure the WG posts their schedule/notes

• From xarray writer, you get nice compression. (https://github.com/zarr-developers/numcodecs), https://www.youtube.com/watch?v=qyJXBlrdzBs 

• WRF data format

• Problems loading WRF netCDF data.
• https://www.unidata.ucar.edu/blogs/developer/en/entry/wrf_goes_cf_two

• Saw an example using AWS batch converting a bunch of TIFFs into cloud-optimized geotiffs. Would like to make an example using pangeo / dask. Originally stored on S3. Answer questions about where to stop using pangeo.

• https://medium.com/pangeo/why-and-how-we-use-pangeo-at-cnes-74553c7fb19b

Short term coordination:

• dask-gateway
• Intake-stac issues: https://github.com/intake/intake/issues/410

2019-09-18 2000 GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://whereby.com/pangeo

Hello and welcome to the team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Charles Blackmon-Luca / LDEO / @charlesbluca
• Tom Augpsurger / Anaconda / @TomAugspurger
• Xu Chen / Florida State University / @chenfsu
• Julia Signell/ Anaconda / @jsignell
• Martin Durant / Anaconda / @martindurant
• Naomi Henderson/ LDEO/ @naomi-henderson
• Jim Crist / Anaconda / @jcrist
• Rich Signell / USGS /

60 Second Updates:

• Charles: set up binder for TRACMIP data; writing medium blog post about TRACMIP data ingestion; experimenting with qgrid
• Tom A: intake-stac performance issue; dask workers out of memory issue
• Xu Chen: experimenting with xmitgcm / xarray to process ROMS & MITgcm data; working on setting up Pangeo on local hpc
• Julia S: working on intake stuff with martin, demo later; fsspec whole file caching
• Martin D: big improvements to intake caching
• Naomi H: trying to get cmip6 data into cloud; trying to understand spread of ocean model grids
• Jim C: About to push a dask-gateway release (https://gateway.dask.org/). Wanting to get a demo instance of dask-gateway up and running soon. Will open issue/pr in
  https://github.com/pangeo-data/pangeo-cloud-federation/
• Rich Signell / USGS / @rsignell-usgs:  going to show this wind-shear demo from UKMET to DaNa Carlis and others at NOAA NWS tomorrow (in addition to standard Pangeo talk) and push for Pangeo to be a key part of EPIC.  Meeting with Scott Henderson on Friday to bring USGS AWS Pangeo deployment in line with the NASA Access AWS Pangeo deployment.

Agenda items:

• CMIP6 Catalog on the Cloud

• Cloud function to generate catalog: https://gist.github.com/rabernat/24164c7d5155678e259b99de0ddc2c6e
• Sure, there is an example notebook in a pangeo binder: https://binder.pangeo.io/v2/gh/naomi-henderson/CMIP6-qgrid/master 
• https://github.com/pangeo-data/pangeo-datastore/pull/44 - really big catalog!
• Martin: really easy to modify intake to accommodate other catalog formats
• Intake integration discussion: https://github.com/intake/intake/issues/417 

• Fsspec caching https://github.com/intake/filesystem_spec/pull/132  [ docs ]
• Xarray 0.13.0

• Fast path for dataset combining:
  https://github.com/pydata/xarray/issues/1823 

• Discourse forum: mm

Short term coordination:

• Action item: open an issue in intake to discuss how to integrate
  https://github.com/intake/intake
• Coordinate in https://github.com/pangeo-data/pangeo-cloud-federation/ to get a demo dask-gateway server up and running

2019-09-11 2000 GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://whereby.com/pangeo
Hello and welcome to the team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Martin Durant / Anaconda / @martindurant
• Jim Bednar / Anaconda / @jbednar
• Joe Hamman / NCAR / @jhamman
• Xu Chen / Florida State University / @chenfsu
• Tom Augspurger / Anaconda / @TomAugspurger
• Tim Crone / LDEO / @tjcrone

60 Second Updates:

• Jim Bednar: Julia is working on new Intake examples ("using the GUI", "how to make a data package", "how to spin up an intake server", "how to consume an intake server", "how to write datafile and catalogs") and improving hvPlot (#263 Adding derived datatime for xarray, #286 Derived datetime for tabular data) to ready it for a release.
• Martin: back!
• Joe: working on climate downscaling use cases using scikit-learn + xarray
  Created new website for pangeo stacks
  (https://pangeo-data.github.io/pangeo-stacks/images.html)

• See also: https://blog.jupyter.org/jupyter-meets-the-earth-1b0eb33c83f

• Xu Chen: postdoc at FSU, working on submesoscale air-sea interaction, oil spill modeling. Wants to use Xarray for data analysis
• Tom A: nothing concrete; David Brochard working on issues related to rechunking
• Ryan A: wrote NSF report, presented prefect use case to the ML working group
  (https://gist.github.com/rabernat/df7ec5fbea982d48e0092927c4b1f474)
• Tim Crone: Pangeo on Azure; ooi.pangeo.io is up and running; convinced EarthCube to fund Tim to work on Pangeo; microsoft really interested in Pangeo

Agenda items:

• Discourse Launch
  https://discourse.pangeo.io
  https://discourse.pangeo.io/categories 

• Tim willing to try it out and possibly help moderate
• Bokeh discourse: https://discourse.bokeh.org (for comparison and inspiration; is already full of discussion)
• https://www.discourse.org/github  -> use to migrate existing issues in github.com/pangeo-data/pangeo to discourse to seed the discussion
• How should we launch?

• Twitter?
• Email blast?

• Joe: Jupyter works hard to shuffle issues between Github and Discourse
• Tim: How will this relate to https://medium.com/pangeo?

• Xu use case

• MITgcm + ROMS

• http://xmitgcm.readthedocs.io
• http://xgcm.readthedocs.org

• http://xarray.pydata.org/en/scipy19-docs/examples/ROMS_ocean_model.html
• Data live on FSU HPC cluster

• http://xarray.pydata.org/en/scipy19-docs/examples/ROMS_ocean_model.html

• Microsoft “Pangeo Product”

• Azure wants to get more involved; make Pangeo work better on their system (node provisioning speed, etc.), playing catch-up
• What would a “1-click” Pangeo look like?
• Dask-gateway
• Jim: what’s their monetizing plan? Make it easier and quicker to spin up resources
• Joe: they could spend 6 months answering “what is Pangeo on Azure?”, figure out the missing pieces of their Kubernetes system, replicate Joe’s blog posts. FIRST bring their tech stack up to the other providers.
• Jacob Tomlinson working on dask-cloudprovider (https://github.com/dask/dask-cloudprovider)
• What about managed JupyterHub deployments?
  https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-jupyterhub.html
• https://docs.dask.org/en/stable/spark.html
• Think about how they can better support the datasets

• Learn about Zarr
• Catalog / discover datasets

• Call scheduled for Sept. 27
• Partnerships need to be coordinated through the Steering Council
• Let’s have members of the Steering Council on the Sept. 27th call

• Pangeo stacks

• Mostly based on repo2docker
• Has cool “onbuild” system to extend an existing docker image with r2d files
• Do we pin every version? https://pangeo-data.github.io/pangeo-stacks/images.html
• Joe: we should move away from this. Science users don’t work with dockerfiles, they prefer conda environments.
• But there are paths forward for going beyond that.

• Distribute a local conda environments to all dask workers

• What about publishing jupyterhub images on docker hub?

• Most cloud providers mirror dockerhub

• Can we expose the environment.yml file from the docker images

Short term coordination:

• Tim will write a blog post about OOI Pangeo
• Catalog environment files from docker stacks

2019-09-04 2000 GMT (1pm PDT, 4pm EDT)

Zoom or Appear today?

Video Conference Link: https://whereby.com/pangeo
Hello and welcome to the team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• James Munroe / MUN / @jmunroe
• Thomas Moore / CSIRO
• Julia Signell / Anaconda / @jsignell
• Tom Augspurger / Anaconda / @TomAugspurger
• Rich Signell / USGS / @rsignell-usgs
• Rob Fatland / UW / @robfatland
• Amanda Tan / UW / @amanda-tanm

60 Second Updates:

• Thomas Moore - preparing to head off on research voyage.  No updates, see you in 4 weeks!
• Julia Signell - working on reproducible projects/environments, dev site for pangeo #704
• Tom Augspurger - Filesystem things last week. Talking about coordinating distributed writes to cloud storage: https://github.com/dask/dask/issues/5361 (should help with netcdf -> zarr)
• Rich Signell.  Heading to DC on Sep 20 for face-to-face meeting with DaNa Corliss, likely project leader for the new NOAA/NWS EPIC project (Earth Prediction Innovation Center) https://owaq.noaa.gov/Programs/EPIC whose goal is to modernize the way weather prediction is done in the US.  They want to move to github, use community models, and the cloud.  I will promote Pangeo as a collaboration/analysis/visualization platform for EPIC.   Want to do a demo using the Rolling HRRR forecast stored in  Zarr, contrasting that to Grib files behind a THREDDS server.
• James Munroe - continuing to move CSIRO to use PangeoHub/AWS for real work; looking at connections between Pangeo and IOOS: https://www.glos.us/code-sprint/
• Ryan Abernathey:

• Presented about Pangeo to NOAA NESDIS cloud technology group:
  https://speakerdeck.com/rabernat/cloud-native-climate-data-with-pangeo-noaa-nesdis-cloud-technical-discussion
• http://earth-env-data-science.github.io - textbook for xarray / basic python
• Playing around with Prefect: https://gist.github.com/rabernat/9f3bb4a35b336c385ae592e01e43b297

• Rob Fatland: In the wake of the pangeo meeting I owe a playback on POETs

• POETs is the pangeo outreach education training working group
• pangeo-data/education-material is the organizing repo

• Amanda: based on experience at OHW - lots of questions about datasets… how to put datasets in the cloud? How to bring my own data to Pangeo

• http://pangeo.io/setup_guides/index.html#setting-up-pangeo-on-hpc-systems

Agenda items:

• CMIP6 Hackathon

Short term coordination:

• .

2019-08-28 1600 GMT (9am PDT, 12pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello and welcome to the team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Tom Augspurger / Anaconda / @TomAugspurger
• Jim Bednar / Anaconda / @jbednar
• Fabien Maussion / University of Innsbruck / @fmaussion

60 second updates

• Ryan: Writing summarizing meeting & sprints. Many good ideas. Authoritative pangeo gallery, but running examples is expensive. Use the pangeo binder to render the notebooks from within the CI job. Pangeo-forge
  https://github.com/pangeo-forge/pangeo-smithy. Solves new users getting data “into pangeo”.  Take legacy dataset, convert to cloud-friendly, and upload, all from CI (using a pangeo deployment for compute).
• Tom Augspurger: taking over fsspec / s3fs / gcsfs maintenance; working on cloud writing optimizations
• Jim: Finishing up Columbia Pangeo grant on 31 August; last topic: https://github.com/pyviz/datashader/pull/779  (no longer 500x faster QuadMesh rendering in Datashader; now more like 40X, partly due to discovery of a Numba race condition; will merge now but later may implement with Dask instead).  Should add examples to the Pangeo site about exploring large raster datasets that are already in Pangeo (ask Ryan if not obvious which ones).
• Fabien:

• Has some money from Uni Innsbruck to hire a student to help writing a “Jupyter Book” about xarray for climate analysis (based on my lecture notes + other things I can gather online.
• From Ryan: Ryan’s notes: https://rabernat.github.io/research_computing_2018/tag/xarray.html
• Writing an abstract for a short course at EGU (https://github.com/pangeo-data/pangeo/issues/685)

2019-08-14 1600 GMT (9am PDT, 12pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello and welcome to the team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Harman Deep Singh /  GSOC Student  / @hdsingh
• Joe Hamman / NCAR / @jhamman
• Martin Durant / Anaconda / @martindurant
• Tom Augspurger / Anaconda / @TomAugspurger
• Lindsey Heagy / UC Berkeley /@lheagy
• Riley Brady / CU Boulder and INSTAAR / @bradyrx
• Jim Bednar / Anaconda / @jbednar
• James Munroe / MUN /@jmunroe

60 Second Updates:

• Ryan: Mostly triaging after vacation, friction with new releases and cloud clusters, CMIP6 hackathon preparations
• Rich:  USGS “Pangeo” postdocs Jeff Sadler and Kirstie Haynie will be attending the pangeo users meeting,  USGS Community for Data Integration is forming a Pangeo group, and Pangeo will be used as part of the architecture for a new cloud-based sensor framework with first use case involving volcano hazards in Alaska. Also model tool development added to Filipe Fernandes workplan for IOOS (involving xgcm/xarray/sgrid/vertical coordinates)
• Harman: xrviz at GSOC
• Joe: Mostly getting ready for summer meeting
• Martin: Lots of bug fixes, docs updates and releases, blog post of fsspec (coming soon) (older dask site blog post: https://blog.dask.org/2019/07/23/extracting-fsspec-from-dask )
• Lindsey: Working with NERSC getting dask / jobqueue running on their system, next up looking at data registry in jupyterlab
• Riley: PhD student at CU (longer update below)
• Jim Bednar: Wrapping up 2019 Columbia/Anaconda contract ending 8/31, including work on xrviz (demonstrated by Harman below), massively faster quadmesh regridding/viz (still in heavy development, but soon to be merged!), and some Dask/fsspec/zarr work from Martin Durant.
• James M: Working on understanding jupyterlab extensions, problems around 1.0 release

Agenda items:

• XrViz demo (xarray visualization GUI), Harman Deep Singh, ESIP Google Summer of Code student.  (10 min)
• Demo of climpred from Riley Brady

• https://github.com/bradyrx/climpred
• https://climpred.readthedocs.io/en/latest/
• Link to slides https://www.slideshare.net/RileyXBrady/pangeo-climpred-presentation
• Comment from James Munroe:

• Similar to doppyo package (https://github.com/csiro-dcfp/doppyo)
• Nice work dealing with time (init + lead)

• Comment from Ryan: similar to weather forecasting needs?
• Get forecast data in the cloud

• Summer Meeting next week: http://pangeo.io/meetings/2019_summer-meeting.html

• Can we get a list of participants on the website?

• Niall Robinson posted paper:

• https://arxiv.org/abs/1908.03356

Short term coordination:

.

2019-08-07 2000 GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello and welcome to the team meeting!
Please sign in below so we know who was here: ········

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Tom Augspurger / Anaconda / @TomAugspurger
• Martin Durant / Anaconda / @martindurant
• Lewis McGibbney / NASA JPL / @lewismc
• Mike Gangl / NASA JPL PO.DAAC / @mgangl
• Joe Hamman / NCAR / @jhamman
• Rob Fatland / UW / @robfatland
• Rafael Guedes / Oceanum / @rafa-guedes
• James Munroe / MUN / @jmunroe
• Amanda Tan / UW / @amanda-tan

60 Second Updates

• Ryan A: on vacation last few weeks, getting excited about upcoming events (Pangeo meeting, OceanHackWeek
• Tom A.: Working on diagnosing workers dying and better scheduling of memory bound tasks. https://github.com/dask/distributed/pull/2940
• Martin D.: Dask maintenance and integration of fsspec with intake/dask. In progress: viz interface to intake catalogs. Going on family leave ASAP.
• Lewis M.: talking with folks about getting the Pangeo stack working with PO.DAAC. New to Pangeo, works on data services / data engineering PO.DAAC.
• Mike G: Cloud lead at PO.DAAC.
• Joe H.:  Getting ready for Pangeo Community Meeting, working on science use case.
• Rob: Key Reset (AWS) + planning / prep for pangeo workshop Edu Working Group

• Part A: 30 min plenary intro/invite incl 3 panel questions with room follow-up
• Part B: Breakout with deliverable: Participants, Roadmap, Owners of, Test Plan

• Rafael G.: Interested in discussing community hosted cloud datasets
• James M.: Working on NetCDF -> zarr conversion. Settled on the batching processing first on HPC.
• Amanda T.: Sprint proposal on documenting setup of Pangeo cloud resources

Agenda items:

• Intro from NASA JPL / PO.DAAC Team

• PO.DAAC = NASA Physical Oceanography Data Center - sea ice, sea surface temperature / salinity / height, etc.
• Mike & Lewis are involved in migration of PO.DAAC services to AWS
• Data challenges related to SWOT mission

• Bridge netCDF/HDF world and GIS world

• Have lots of capabilities in PO.DAAC platform; but so far activities have not been aligned with Pangeo perspective
• What other analysis capabilities could Pangeo stack provide for PO.DAAC scientists?

• Ryan’s answer: ability to process WHOLE dataset  efficiently (rather than just search / extract a specific granule)

• Key success indicators would be adoption of datasets in the cloud.
• https://github.com/pangeo-data/pangeo/issues/686
• JH: NASA is currently funding some of us to explore integrations with Pangeo: https://earthdata.nasa.gov/esds/competitive-programs/access/eos-data-cloud
• Lewis: define “success criteria”
• Analogy between Apache and Pangeo
• Lewis: appreciates that Pangeo conversations happen in the open on GitHub

• Presentation of climpred from Brady Riley next week
• Discussion forum: https://github.com/pangeo-data/pangeo/issues/677

• Some have suggested that github is not the right place for science-user discussion, perhaps Discourse would be better
• Others have emphasized the value of github as a global namespace, connections to other projects

• Any AGU plan?

• Joe submitted an abstract for a Pangeo talk
• Scott/Joe/Amanda have another workshop

• Education working group (starting Wed Aug 21 x 30 minutes)

• Why we have an Ed WG
• Framework: pangeo as a three-pillar by five-level structure

• Technology, community building, domain publication
• Organizations, domain scientists, tech/tools, data, research problems

• Topics for Thursday Ed WG “building session”

• Who is involved, what can you own?
• "program management" process needed
• Ed WG role in testing pangeo in advance of the demo / class
• Ed WG role in creating, growing, sustaining domain science communities
• Paper completion as definite end-point example
• Ed WG role in writing the next proposal(s) supporting the pangeo program

• Ryan: Let’s write a book! “Cloud-native climate science” I) User Guide II) Admin Guide

• Also website is getting pretty stale; so good tactics

2019-07-31 1600 GMT (9am PDT, 12pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello and welcome to the team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Fabien Maussion / University of Innsbruck / @fmaussion
• Martin Durant / Anaconda / @martindurant
• Rich Signell / USGS / @rsignell-usgs
• Lindsey Heagy / UC Berkeley / @lheagy
• Tom Augspurger / Anaconda / @TomAugspurger
• Julia Kent/ NCAR/ @jukent

60 Second Updates:

• Fabien:

• Presented JupyterHub and Pangeo (briefly) at IUGG: https://fabienmaussion.info/2019/07/24/IUGG2019/ 
• Roadmap for oggm.pangeo.io: https://docs.oggm.org/en/latest/oeps/oep--0002-cloud.html (timeline unknown, will seek for help). First step before pangeo community can help: we have to refactor our parallel workflow to use dask.distributed.
• Discussion: it’s nice to have pangeo deployments examples where xarray is not a central tool (i.e. beyond atmo/climate/gridded datasets)

• Martin

• Filesystem-spec & fsspec-based dask are being released todayish
• Intake using fsspec 2.x. GUI has a file browser now (including remote)
• Plugin model now based on entrypoints. Will want to update the main packages providing drivers (but package scan will be maintained for a long time)

• Rich:

• Said goodbye to ESIP AWS kops cluster, pangeo.esipfed.org, hello to cloud-federation esip.pangeo.io
• Reconnected with Jon MacCarthy (LANL) about seismic data handling

• Some think that pangeo is xarray- or geo/atmos/ocean-specific, don’t realize they don’t *have* to use xarray
• Also okay for data to not fit cf-conventions -- seismology should come up with their own conventions for NetCDF4/HDF5/Zarr that work for them
• Seismology also struggling with sparse data formats / operations

• Lindsey

• Interested in sparse arrays within xarray. There’s an open PR:  https://github.com/pydata/xarray/pull/3117

• Julia:

• Working on documentation for intake-esm. How new users can use a new dataset with intake-esm.
• Analysis for pangeo-approved bias correction between modeled and measured data

Agenda items:

• Approaches for sharing Pangeo Jupyter Notebooks: Dan Allen's blog post, JupyterHub Share-Link (Rich)

• Can this jupyterhub extension be installed on the pangeo deployments?
• Rich will open a GitHub issue to discuss with people more familiar with the jupyterhub deployment.

• Pangeo at EGU2020? Deadline Sept. 5th: https://egu2020.eu/guidelines/short_courses_guidelines.html

• Idea to have a double bloc (bloc=1h45): (1) xarray intro, (2) pangeo. Fabien can take over xarray intro - question to pangeo if folks plan to attend EGU. Will open an issue on github to engage community.
• Scott did a demo (link?). Questions about the format: maybe 1h45 is too short.  

• Issues with loading xarray datasets with 2D coordinates where dimensions are named lat and lon. Need a way to preprocess the dataset before xarray tries to interpret metadata.

Short term coordination:

• Rich will open an issue on JupyterHub share link

2019-07-24 2000 GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://appear.in/pangeo
Hello and welcome to the team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Rich Signell / USGS / @rsignell-usgs
• Scott Henderson / UW / @scottyhq
• Martin Durant / Anaconda / @martindurant
• Matthew Rocklin / NVIDIA / @mrockli
• Julia Signell / Anaconda / @jsignell (jsignell@anaconda.com)
• Thomas Moore / CSIRO / @Thomas-Moore-Creative
• Robert Woodcock / CSIRO (ODC, CEOS WGISS)  / @woodcockr
• Jim Bednar / Anaconda/ @jbednar

60 Second Updates:

• Rich:

• Working on esip.pangeo.io .  Staging working well but production not yet up. Scott is doing a bit of information transfer to Rich.
• ESIP meeting last week.  Presented xrviz GSoC project.

• Scott:

• Also at ESIP meeting.  Used AWS binder to run a workshop.  Previous examples that traditionally ran on GCP worked well, which was nice to see.
• Gave a presentation at WHOI

• Rich: Scott asked “how many people have heard of Pangeo” and 30/30 responded positively, despite the fact that we didn’t know most of them.  Hooray for buzz!

• Preparing the Pangeo community meeting

• Running into a room cap at 52, waitlist up to 70

• Martin:

• Xrviz
• Similar dataframe application.  This is a bit simpler at the moment.
• Working on fsspec work personally.  

• Rebased s3fs/gcs/dask on top of it; Intake coming
• Some small pains, but things seem decent
• Zarr and xarray should directly use functions in fsspec now, rather than ad-hoc or having to create stores (although that will always be offered as a possibility)
• Greatly expanded testing in fsspec itself, and of course extra testing is now happening in the other repos too

• Matt:

• Deployment work.  Jacob is working on Dask+ECS integration, which folks here may find interesting.  https://github.com/jacobtomlinson/dask-cloud
• Triggering a large amount of internal Dask cleanup

• Julia

• Working on intake-stac (https://github.com/pangeo-data/intake-stac/pull/19) and various hvplot bug fixes.

• Thomas

• Putting in support proposal to make Pangeo-style deployments on Australia’s Pawsey Supercomputing centre easier for general users.
• Curious about Pangeo meeting.  Is there a call for talks?
• RE: Zarr/NetCDF questions from Rich.  Perhaps we can get a report back from James Munroe - Thomas will follow-up.
  Rich: Anderson has a nice talk at scipy where they did a bit of this.

• JimB

• Also at ESIP meeting; presented HoloViz and discussed collection of ML-workflow examples with the ESIP ML cluster
• SciPy was fun too!

• Robert (CSIRO)

• Keen to collaborate over coming months on EO data in Zarr format prototype.

Agenda items:

• Update on dask issues with  netcdf to zarr conversion (memory/backpressure)
• Meeting updates

• NOAA + USGS meeting: two directors got together for high level discussions.  Pangeo was a topic and there is talk of putting together a cross domain team.  Fingers crossed.
• ESIP:

• There was a strong focus on having cloud friendly data that’s directly exposed.  
• This includes both data and metadata (like STAC).  
• Some conversation around Zarr.  
• HDF group was there as well talking about their efforts to make HDF cloud friendly.  
• Informal Meeting about NetCDF on the cloud conversation.  Including people from HDF, OpenDAP, …
• HDF group was talking about how previously there was a limitation on number of concurrent accesses to a single object on S3
• Acknowledgement that we don’t want services

• SciPy:

• Conversation with JupyterHub/BinderHub folks on encapsulating environments.

• Lots of talks using Zarr

• UniData said that they’ll also support a Zarr backend for NetCDF-Java
• There was some interest in seeing a large satellite imagery dataset in Zarr format from NASA data on the cloud.  ISat might be a good example.
• There was also a nice cloud friendly remote sensing data
• Robert (CSIRO, OpenDataCube) would be interested in running such an experiment.  We have a home-grown version of this, and wouldn’t mind replacing ours with Zarr.  Might also want to present this at an upcoming meeting in October.

• Scott’s interested in helping
• Element84 might also be interested in this for ISat-2 data
• Robert’s thought is that they’ll just use OpenDataCube to ingest data into Zarr format and then just use OpenDataCube.

• Lots of Australians on today’s call?  Should we move things later?

• It’s currently pre-commute time, which is nice, so either this time, or hours later.
• The two folks on the call were happy with this time, we’re probably not hearing from others from whom this is too early though.

• Robert: I’m curious about intake+stac.  We’d like to have someone present about STAC at an upcoming meeting in October (remote is fine).  

• Julia: I know a bit about integrating it with Intake, but probably not a deep STAC expert
• Scott: You should maybe chat with Matt Hansen, who is more deeply involved in the core STAC project (is this the right name?)
• Robert: we’re also interested in Intake+STAC

Short term coordination:

2019-07-17 1600 GMT (9am PDT, 12pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello and welcome to the team meeting!

Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Kevin Paul / NCAR / @kmpaul
• Martin Durant / Anaconda / @martindurant
• Anderson Banihirwe / NCAR / @andersy005
• Julia Kent / NCAR / @jukent
• Matthew Rocklin / NVIDIA / @mrocklin
• Jacob Tomlinson / NVIDIA / @jacobtomlinson
• Tom Augspurger / Anaconda / @TomAugspurger

60 Second Updates:

• Kevin:

• Kinda quiet

• Martin:

• Continued work on fsspec; PR for dask basically complete; requires release of fsspec

• Anderson:

• Transfer of CESM LENS dataset to S3; 1 TB on S3 already; almost 100 TB left
• SciPy talk went very well (Thanks, Matt!); received positive comments; Feedback suggests people would like to know more about JHub, but things are mostly private, trying to make public
• https://www.youtube.com/watch?v=vhawO8fgD64 

• Julia:

• Dealing with calendaring; Help wanted

• Matt:

• Jacob:

• New cluster manager for different cloud systems; supersimple startup of dask cluster requiring only your credentials;
• https://github.com/jacobtomlinson/dask-cloud 

• Tom:

• Nothing new

Agenda items:

• Calendaring?

Short term coordination:

[Special SciPy 2019 BOF edition] 2019-07-11 1800 GMT (11am PDT, 2pm EDT)  

71 people in the room!

Joe Hamman kicks off BOF with 5 minute intro of slides.

One minute intro from folks in the room:

Institutions represented:

Capital One Center for Machine Learning, Brookhaven, USACE, Argonne, Northeastern Kostas, NCAR, NASA Goddard, GOMRI, Chan-Zuckerberg Initative, MBARI, Oceanweather, Shell Oil, Quantum Spatial , EDF Renewable Energy, LDGO, Microsoft, IOOS, Texas Water Development Board, UCSB, UC Boulder, U Congo ,  ?? Marine, Monash University, Descarte Labs,  NASA Ames, NVIDIA, RENCI

Fields of interest:

Machine Learning, Cell Biology, Oceanography, Meteorology, Physics, Homeland Security, Geospatial Analysis, Hydrology, Remote Sensing, Microscopy,Biomedical imaging , Heathcare, Precision Agriculture, Groundwater Modeling, Astronomical Data, Hydrology

Ryan give a live demo of Pangeo on the Cloud on ocean.pangeo.io

Q: Do you need to run on Cloud, what do you do on HPC?

A: See Anderson’s talk later

Q: Do you need to use Zarr

A: No, Zarr works well on cloud, but you can use anything that xarray can read

Q: How do custom environments work?

A: Different JH use different default environments

Q: How does metering/cost/tracking for users work?

A:  Anyone can access data with S3, also working with NASA EarthDataLogin.

Q: What about GCM data?

A: Pangeo folks expect that providers will see the value of moving their data to the Cloud, and do it themselves.  

Q: You can we can just try this, how does that work?

A: You can use the pangeo binder right now with your own repo.  The pangeo binder is like mybinder.org but can launch dask clusters,  and has a bit more memory

Q: Seems like we heard about a lot of different fields that want Pangeo, what should we do?

A: You can set up your own using the Pangeo deployment guides. Some pangeo secrets are

2019-07-10 2000 GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello and welcome to the team meeting!

Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Kevin Paul / NCAR / @kmpaul
• Julia Kent / NCAR / @jukent
• James Munroe / MUN / @jmunroe
• Martin Durant / Anaconda / @martindurant
• Alex Leith / Geoscience Australia / @alexgleith

60 Second Updates:

• Kevin:

• Testing Z5 against MPI-IO solutions (NetCDF and HDF5)

• Julia:

• Blog post on C3DIS conference (expect something in the near future)
• Working on R to Python translation for bias correction
• intake-esm documentation

• James:

• Struggling with converting NetCDF files to Zarr
• AWS Pangeo Hub work
• Pulling down NetCDF; loading into Xarray; to Zarr
• From Martin:

• Pass around a function that creates the part of the Dask array

• Martin:

• Master branch of S3FS and GCSFS depend on fsspec
• Dask with also depend on fsspec soon (1600 lines gone!)
• DFViz alpha on Conda forge
• Dask is having a sprint session after SciPy (Sat & Sun)

• Alex:

• Leads team working on Open DataCube
• Working with ODC for about 1 year
• Saw Pangeo JHub env and was impressed
• Used Dask with success with ODC data
• ODC: A Python library (with postgress DB) to index land data files and load them into data structures (including Dask arrays); moving toward moving analysis to the data; Uses its own index of the data
• Martin:

• Sounds like this is crying out for an intake interface

Agenda items:

Short term coordination:

2019-07-03 1600 GMT (9am PDT, 12pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello and welcome to the team meeting!

Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Anderson Banihirwe / NCAR / @andersy005
• Kevin Paul / NCAR / @kmpaul
• Fabien Maussion / University of Innsbruck / @fmaussion (first time attendee)
• Lindsey Heagy / UC Berkeley / @lheagy
• Tom Augspurger / Anaconda / @TomAugspurger
• Julia Kent / NCAR / @jukent
• Nick Mortimer / CSIRO / @nickmortimer
• Paul Branson / CSIRO-UWA / @pbranson
• Martin Durant / Anaconda / @martindurant
• James Munroe / MUN / @jmunroe
• Jim Bednar / Anaconda / @jbednar

60 Second Updates:

• Anderson:  

• CESM LENS data for AWS in progress; ~50GB of data on AWS by end of week
• Q: How are you verifying that the data actually made it up to AWS?
• A: Checksums on files

• Fabien:

• Vanilla jupyterhub working with the OGGM model. You can try it out at https://hub.oggm.org (docs: https://docs.oggm.org/en/latest/cloud.html). Nothing special here (no pangeo yet), just the model working in a out-of-the box zero2jupyterhub. I’m thinking about the next steps involving pangeo now.
• App we developed with Pyviz a while ago: https://glaciers.pyviz.demo.anaconda.com/app

• Lindsey:

• AGU sessions - Lightning talks on Open Source Software
• https://agu.confex.com/agu/fm19/prelim.cgi/Session/78079

• Tom

• Investigating Netcdf -> zarr workflow (looking at Rich’s example). Want to write a best practices document
• TODO: Update hub dependencies
  https://github.com/pangeo-data/pangeo-binder/issues/59
• ML working group meet. Data loading is still a problem.  Want to write some benchmarks and best practices.

• Julia:

• Xarray PR complete and merged! (Yay!)
• Working on a bias correction example; translation from R to Python/Xarray/Dask
• Should we have a newcomers meeting?

• Nick:

• NetCDF--Zarr workflows
• ARGO float profile data into single Zarr store, using new Zarr appending feature
• Loading into holoviews and datashader looks promising
• Much more work to apply ML

• Paul:

• Converting 40yr hindcast data (hourly output in monthly files) to multiple Zarrs
• Using Zip Zarr files to reduce inode count

• Martin:

• Work on fsspec in Dask
• ZipFile will be a new option, along with FTP, SSH, etc.
• Common behavior across local FS and cloud storage
• Should be done in days

• James:

• More NetCDF → cloud
• Struggling with “partially completed conversions”, so checking validity is important
• Q: Should we have a newcomers meeting? Like JupyterHub monthly meetings
• Q: How do we publicize these?  Maybe just “verbally”

• Jim:

• PyViz.org and HoloViz.org split done; for background see http://blog.pyviz.org/pyviz-holoviz.html 
• Preparing brand-new SciPy HoloViz tutorial for Monday, based on USGS earthquake data, to appear in its first form tonight at http://holoviz.org/tutorial 

Agenda items:

• Fabien: questions about OGGM and Pangeo

• Is a Pangeo deployment Z2JHub + Dask?  Basically.
• Now has a Hub that is working and running the OGGM model
• Big datasets are not the issues for them; just want to simply parallelize with dask (i.e., replace multiprocessing with dask)
• Can be done with dask.delayed: https://docs.dask.org/en/latest/
• https://examples.dask.org/applications/embarrassingly-parallel.html
• Data output: encourage users to write to locations they have write access to

• Monthly Newcomer Meetings:

• Make this visible on the main homepage
• Maybe need a different platform than appear.in?
• Need to create a GitHub issue to start a discussion on this to get larger buyin in Pangeo
• Use 2 different times for rotating meetings to allow for differing time zones
• JupyterHub / BinderHub Team meeting issue for July:
  https://github.com/jupyterhub/team-compass/issues/175 

• Why the split between PyViz and HoloViz?

• See http://blog.pyviz.org/pyviz-holoviz.html

Short term coordination:

• Kevin will create a GitHub issue to drill down on technical details of setting up a monthly (quarterly?) newcomers meeting

2019-06-26 2000 GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello and welcome to the team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Rich Signell / USGS / @rsignell-usgs
• Anderson Banihirwe / NCAR / @andersy005
• Kevin Paul / NCAR / @kmpaul
• Julia Kent/ NCAR / @jukent
• Martin Durant / Anaconda / @martindurant
• Tom Augspurger / Anaconda / @TomAugspurger
• Jim Bednar / Anaconda / @jbednar
• James Munroe / MUN / @jmunroe

60 Second Updates:

• Rich Signell: Will NOT present “Pangeo for EarthMap” proposal to USGS Director Jim Reilly and NOAA Atmosphere and Ocean Undersecretary Tim Gallaudet on July 19.
• Kevin Paul & Anderson Banahirwe:

• Working on zarr-ification of CESM LENS data for AWS public dataset program
• https://github.com/NCAR/lens-on-aws/blob/master/notebooks/zarrification.ipynb
• Using intake-esm, may be avoiding an issue with open_mfdataset that Rich is seeing.

• Julia Kent:

• Submitted first pull request to Xarray!
• Starting to work with R also

• Joe Hamman

• Still not working :)
• Registration is closed for summer meeting in Seattle (57 registered as of today), confirmation email went out today

• Martin Durant

• Intake: param editor in the GUI and “upload” functionality
• Dfviz and fsspec moved to Intake org and becoming more complete
• Fsspec gains auto-caching file-system layer

• Tom Augspurger

• Fallout from Dask 2.x release

• Jim Bednar

• Working out plans for remaining 2019 Anaconda/Columbia funding (7 developer weeks).  Proposal:

• 1 week xrviz assistance
• 1-2 weeks finishing QuadMesh support for Datashader
• 2 weeks Dask QuadMesh rendering
• 2-3 weeks improving/extending pangeo-example-notebooks.

• For after August, other geo-specific regridding?
• As there was support from those present, moving forward with this plan unless requested otherwise.

• James Munroe

• pangeo-csiro hubploy deployment now running on AWS
• Also working on zarr-ifications of model data.  Also hitting some roadblocks.  Using s3fs to read and push to zarr all on the cloud

• Thomas Moore

• Top: Getting CIO support for CSIRO (5000 people!) commitment to Pangeo
• Middle:

• Internal effort to secure parallel storage for HPC deployment
• Trialing AWS workflow (see James Munroe’s efforts)

• Bottom:

• Supporting new post-docs moving to their workflow to Pangeo
• Dougie Squire flying the Pangeo flag at IMSC 2019
• (Still) trying to fix Pangeo HPC deployment on Pawsey Supercomputing Centre resources (working with CSIRO’s Paul Branson).

• Rob Fatland

• Icesat2 hack week, david sheen had great notebook, icesat2 data part of CMR, 60 students, AWS credits burn was only about $60/day.

Agenda items:

• Pangeo/OpenDataCube coordination progress (Rich)

• https://github.com/pangeo-data/pangeo/issues/96

• Zarr for forecast data from the National Water Model (Rich)

• Met Office solution?:
• https://medium.com/informatics-lab/creating-a-data-format-for-high-momentum-datasets-a394fa48b671
•  https://medium.com/pangeo/continuously-extending-zarr-datasets-c54fbad367d
  James: we are interested also.  
  Not sure where things stand with this.  Need to investigate.  

• CMIP6 Hackathon in fall 2019, to be split at two locations: Columbia and NCAR Short term coordination:

• https://cmip6hack.github.io/#/

2019-06-19 1600 GMT (9am PDT, 12pm EDT)

Conference Link: https://appear.in/pangeo

Hello and welcome to the Pangeo team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Kevin Paul / NCAR / @kmpaul
• Julia Kent / NCAR / @jukent
• Lindsey Heagy / UCBerkeley / @lheagy
• James Munroe / MUN / @jmunroe
• Julia Signell / Anaconda / @jsignell
• Matthew Rocklin / NVIDIA / @mrocklin
• Tom Augspurger / Anaconda / @TomAugspurger
• Martin Durant / Anaconda / martindurant
• Jim Bednar / Anaconda / @jbednar
• Chiara Lepore / LDEO / @chiaral

60 Second Updates:

• Ryan: NASA ECCO data portal
• Rich Signell

• Pangeo Talk last week at AWS Public Sector Summit was well-received.   Live demo on AWS binder worked perfectly.  Got interest from FEMA, EPA, STScI (Space Telescope Science Initiative)
• ESIP Lab Proposals due June 21 (<4 pages, <$7000), OpenDataCube/Pangeo demo would be cool
• Pre-proposal submitted “Pangeo for EarthMap”.  If selected, will present before the USGS Director Jim Reilly and Assistant Secretary of Commerce for Oceans and Atmosphere Tim Gallaudet on July 19.

• Matthew Rocklin:

• Dask is going through a largish release.  We’re breaking a few things and would appreciate spot checking existing workloads. (Xarray test suite should be fine in a day).
• pip install git+https://github.com/dask/{dask,distributed}@master
• Other announcements:

• Jim Crist pushed out a Kubernetes backend for Dask Gateway
• NetCDF + Zarr post: https://www.unidata.ucar.edu/blogs/news/entry/netcdf-and-native-cloud-storage

• Kevin Paul:

• Have generated scaling data on Cheyenne for computation operations and C-based I/O operations (via IOR).  Data is available for everyone to explore.  TL;DR: Good scaling in computation operations and surprisingly little dependency on chunk shape (“temporal” vs “spatial” vs “auto”), and (our use of) Z5 has very large latency but scales beautifully.

• Plan to scale out further on Cheyenne, if we can get the resources.
• Plan to test on Wrangler in the next month.
• Plan to compare Python-based I/O operations to C-based I/O.
• Would like to coordinate with others who are doing similar things on other machines and compare “apples to apples.”

• Nick Mortimer (CSIRO)

• Had James and Julia Kent visit which was effective at evangelism
• Visiting NCAR working on Intake ideas.  Maybe putting together a catalog for Argo float data

• Julia Kent (NCAR)

• Visited NSF EarthCube conference
• Getting hands dirty with Xarray

• Lindsey Heagy:

• Lots of workshops recently (Dashboarding and HPC): quite excited about voila as a dashboarding approach for Jupyter: https://blog.jupyter.org/and-voil%C3%A0-f6a2c08a4a93 
• Scipy BoFs: Pangeo BoF: Thursday 1:00-1:55 pm, Room 101

• James Munroe:

• AWS deployment for CSIRO
• CKAN / Intake / catalogs, etc.

• Julia Signell:

• Working on self-contained pyviz examples: examples.pyviz.org

• Tom Augspurger:

• Dask-kubernetes compatibility for distributed master
• Array scheduling / diagnostics

• Martin Durant

• Working on filesystem-spec / s3fs / gcsfs / dask integration

• Jim Bednar:

• PyViz.org is being repurposed to be a non-partisan site with overviews about viz; previous content migrated to holoviz.org.

• Chiara Lepore:

• Convective environments calculation code
• Started on ERA-interim
• More interested in ERA-5: how to get it?

Agenda items:

• Blog post on Petabytes of NASA Ocean Model data (Ryan)
• Coordination with OpenDataCube?  (Rich)

• Huge overlap between projects
• Matt: ODC has functionality we don’t provide (analysis ready data)

• Zarr creation: xr.open_mfdataset and memory usage issues (Rich)
• Self-contained domain-specific examples using anaconda-project (Julia)
• (if time) quick demo of dfviz, fsspec browser

Short term coordination:

• Ryan opens a dask issue about lazy array construction performance
• Try to have more science presentations and future checkin meetings
• James or Rich reach out to someone from ODC to come talk to us

2019-06-12  (4pm EDT)

Hello and welcome! If you are joining the team video meeting, sign in below so we know who was here:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Martin Durant / Anaconda / @martindurant
• Julia Signell / Anaconda / @jsignell
• Tom Augspurger / Anaconda / @TomAugspurger
• Rafael Guedes / Oceanum / @rafa-guedes

60 Second Updates:

•  Ryan Abernathey

•  https://www.bnl.gov/nysds19/files/pdf/agenda.pdf - gave a talk today
• https://data.nas.nasa.gov/ecco/ - working on new xarray data accessor
• Xrviz https://github.com/intake/xrviz

• Martin: working on intake, helping GSOC student
• Julia: Working on pyviz examples website, could eventually become a generic “examples” gallery
• Tom: PR for memory backpressure issue in dask; working on dask-kubernetes & async
• Rafael: worked for metocean solutions (forecasting of waves and currents), just started a new project called Oceanum (www.oceanum.science). https://github.com/metocean/wavespectra

Agenda Items:

• Ongoing conferences (NYSDS, PASC, EarthCube, etc.)

• https://pasc19.pasc-conference.org/program/minisymposia/

• Pyviz examples - reproducible and self-contained project storage

• https://examples.pyviz.org 
• Each project is in its own directory, at least one notebook + environment file
• https://github.com/pyviz-topics/examples
• anaconda-project run or just use environment file standalone
• But how does it work with binder
• https://github.com/pangeo-data/pangeo-stacks/
• Uses https://github.com/pyviz/nbsite 

• Fsspec in Intake example (Martin D)
• Dfviz example (Martin D)

•  https://github.com/intake/xrviz

2019-06-05  (9am PDT, 12pm EDT)

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / rabernat
• James Munroe / MUN / jmunroe
• Kevin Donkers / UK Met Office / kaedonkers
• Martin Durant / Anaconda / martindurant
• Tom Augspurger / Anaconda / TomAugspurger
• Jim Crist / Anaconda / jcrist
• Amanda Tan / UW / amanda-tan
• Ariel Rokem / UW / arokem
• Jim Bednar / Anaconda / jbednar

60 Second Updates:

• James Munroe: regridding / resampling research
• Kevin Donkers: just returned from CSIRO hackathon in Hobart; Peter K and Jacob were at Pangeo French
• Martin Durant: worked on filesystem-spec; refactored s3fs in a  branch around filesystem-spec; moving towards overall refactor of dask-related filesystems, could be performance benefits
• Tom Augspurger: ML working group meeting; re-dedication to writing a blog post about issues related to data loading, summarizing big picture; working on array scheduling issues; going to work on dask-kubernetes async (for dask labextension)
• Jim Crist: working on dask-gateway (https://github.com/jcrist/dask-gateway), for managing clusters across many users
• Amanda Tan: Pangeo AWS cluster stress testing for 70-user hackathon; replicating GCP datasets in S3
• Ariel Rokem: neuroscientist, created PanNeuro
• Jim Bednar: Been busy with formal Panel release (see Medium post) and Python Visualization eBook

Agenda Items:

• PanNeuro update

• Collab between 5 different experimental and theory groups (NYU, Chicago, Berkeley, UC Irvine). Part of Brain Initiative ($1B).
• Multichannel timeseries from human and animal subjects
• Access to shared datasets
• Challenges spinning up users (used to Matlab)
• Users want batch submission

• https://github.com/SchedMD/slurm-gcp?
• https://jupyterhub.readthedocs.io/en/stable/reference/services.html
• https://papermill.readthedocs.io/en/latest/
• https://www.prefect.io/

• Presentation: https://arokem.github.io/2019-BRAINI-PanNeuro-slides/
• Cloud Native Science workshop / whitepaper

• Update and summary from education working group

• Large discussion about learning objectives
• Need to come to consensus about target audience
• What are the fundamental things that will drive users to actually use Pangeo?

• Data!

• Let’s not develop all brand new content
• SHARE NOTES HERE

• Changes coming to intake caching

• Existing chaching mechanism has proven complicated / hard-to-use
• Refactor in progress

Short Term Coordination

• Met Office tests dask-gateway

2019-05-29 1600 GMT (1pm PDT, 4pm EDT)

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

Name / Institution / @GitHub handle

• Joe Hamman / NCAR / jhamman
• Tom Augspurger / Anaconda / TomAugspurger
• Ryan Abernathey / LDEO / rabernat
• Jim Bednar / Anaconda / jbednar
• Rich Signell / USGS / rsignell-usgs
• Julia Kent/ NCAR/ jukent
• Martin Durant / Anaconda / martindurant
• Thomas Moore / CSIRO / Thomas-Moore-Creative

60 Second Updates:

• Joe: last week of work for the next ~6 weeks; wrapping this up, working on Pangeo proposal with biomed folks
• Tom: working on dask scheduling under high memory load
• Ryan: working on new uploads for LLC4320 dataset (huge quadmesh); also proposal with Joe; gave Pangeo presentation at OOI last week
• Jim Bednar: Julia Signell working on infrastructure for maintaining a large suite of self-contained, portable examples. Jim Crist working on dask-gateway for HPC usage.
• Rich Signell: developing new national water model example (20700 netcdf files; ~3TB); still working on parallel hdf5 on the cloud
• Julia Kent: just came back from Hobart; hosting python-basics hackathon next week, then Earthcube conference; Friday thesis defense (!!!)

• https://ncar-hackathons.github.io/

• Martin Durant: excited to summarize HDF discussion! Intake-examples repo updated; mentoring Google SOC student on xarray visualizer using Panel; rewriting intake caching mechanism; fsspec is ready to be factored out of dask
• Thomas Moore: communicating with leadership; untangling his boondoggle of computing resources; working on deploying Pangeo on local HPCs; doing internal training; FY ends on June 30, trying to secure support for cloud going forward; coming to meeting in Seattle

Agenda Items:

• Scheduling blog post draft (Joe): https://hackmd.io/AhxR64RjQnmBPCSAQhjV6w
• Dask-kubernetes 0.8 released, rebuild your pangeo binders! (Joe)

• Should route dask workers to proper node pools

• Discussing steps forward for CMIP6 on the cloud (Ryan)

• https://pangeo-data.github.io/pangeo-datastore/master/cmip6.html
• NCAR is moving forward with CESM-LENS
• Are there funding agencies willing to support these efforts?
• We need a “white paper” (@ryan)
• AWS vs. Google

• Intake-STAC 0.1 released, check it out.

• https://github.com/pangeo-data/intake-stac/

2019-05-22 1600 GMT (9am PDT, 12pm EDT)

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

Name / Institution / @GitHub handle

• Amanda Tan/UW eScience/amanda-tan
• Jim Bednar /Anaconda/ jbednar
• Tom Augspurger / Anaconda / TomAugspurger
• James Munroe / MUN / @jmunroe
• Rob Fatland / UW eScience / robfatland
• Martin Durant / Anaconda / martindurant
• Aimee Barciauskas / Development Seed / @abarciauskas-bgse
• Sean Harkins / Development Seed / @sharkinsspatial

60 Second Updates:

• Amanda: ESIP Summer meeting and AWS public sector summit, talking about Pangeo work, setting up clusters
• Jim Bednar: Joris getting started on cythonizing GeoPandas and making it a Pandas ExtensionArray. Panel gearing up for a new release/announcement; will be a talk at SciPy.
• Tom: Looking into dask scheduling issue: https://github.com/dask/distributed/issues/2602
• James Munroe: Working on AWS deployments
• Rob Fatland: EDU WG call pushed to week of June 3-7

• Doodle poll: https://doodle.com/poll/a7xd8ciap375wdsi
• Repo ‘pangeo’ has some notes / enthusiasm
• Repo ‘education-material’ is more focused, includes some draft structure

• ...and links to materials like Damien’s atmos-ocean SWC

• Martin: Release of gcsfs, intake; examples docs and repo for intake being updated
• Aimee: Working on scaling dask / kubernetes for transforming MUR SST netcdf to zarr, working on presentations to NASA MSFC on Pangeo with AWS

Agenda Items:

1. Rob: Working Group Update

1. Exemplary notebooks exist, build more inclusive notebooks
2. Agenda for first WG meeting: identify where Pangeo is needed; don’t reinvent
3. Joe: can point to Zero to Pangeo, Zero to Jupyterhub

2. Pangeo meeting in Seattle

1. Jim Bednar will be at ESIP, with a PyViz overview talk
2. Reach out to Chelle Gentemann

3. Sean’s issues: eksctl vs kops

2019-05-15 2000 GMT (1pm PDT, 4pm EDT)

Due to expected attendance < 15, back to appear.in.

Conference link:  https://appear.in/pangeo

Welcome to the Team Meeting.

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Thomas Moore / CSIRO / @Thomas-Moore-Creative
• James Munroe / MUN / @jmunroe
• Joe Hamman / NCAR / @jhamman
• Matthew Rocklin / NVIDIA / @mrocklin
• Tom Augspurger / Anaconda / @TomAugspurger
• Rob Fatland / UW / @rob
• Amanda Tan / UW/ @amanda-tan
• Sean Harkins / Development Seed / @sharkinsspatial
• Martin Durant / Anaconda / @martindurant
• Tim Crone / LDEO / @tjcrone
• Aimee Barciauskas / Development Seed / @abarciauskas-bgse

60 Second Updates:

• Rich: benchmarking work + USGS Pangeo EKS
• Thomas Moore: just wrapped C3DIS conference, including Pangeo session, hackathons in Canberra + Hobart
• James Munroe: plugin for intake ESM for CSIRO/CAFE datasets
• Joe Hamman: first blog post about Pangeo on kubernetes published, second on the way; intake-stac driver nearing initial release, blocking on intake master release; work on dask-kubernetes

• Blog: https://medium.com/pangeo/pangeo-cloud-costs-part1-f89842da411d
• Intake-stac: https://github.com/pangeo-data/intake-stac
• Dask-kubernetes affinity: https://github.com/dask/dask-kubernetes/pull/147

• Matthew Rocklin:

• playing with how dask internally handles cluster objects
  https://github.com/dask/distributed/pull/2675  <<---- Joe, look here!
• new dask array HTML repr
  https://github.com/dask/dask/pull/4794
  

• Tom A: working on dask issues for large array computations
• Rob Fatland:

• Binder tutorial pre-scale up: works. Auto-collapse: Too fast. More tests planned
• AWS burn rate update (web page on existing Lambda)
• Education working group still spinning up

• Amanda Tan:

• Working on presentation for AWS public sector summit

• Sean Harkins:

• Working on pilot program for NASA DAACs to do AWS-based Pangeo deployment
• Cost modeling

• Martin Durant

Working on intake dependencies and import time

• New release of gcsfs

• Tim Crone - Hi! I’m Tim Crone
• Aimee Barciauskas - working on NASA MAAP program, interested in leveraging Pangeo for backend infrastructure, working on supporting scientist to convert dataset from netCDF to zarr

Agenda items:

• Announcement from @tjcrone about new OOI cloud initiatives

• Azure AI for Earth award last year $10K credits, Tim deployed Pangeo there
• Dax Soule: another AI for Earth award ($15K)
• OOI: major NSF-sponsored program
• Negotiated with Azure for storage for OOI data: 500 TB of blob storage for OOI data for 3 years, $30K compute credits per year for 3 years
• Azure people seem excited about Pangeo
• Spinning up team (includes Rich Signell)
• James Munroe of CIOOS (Canadian OOI): how can they learn from OOI experience?
• $30M has been spent on OOI cyberinfrastructure!
• Bringing Azure deployments into cloud federation?

• Recap of the recent Australian Pangeo activities (@jmunroe - are you back in Canada or still down under?)

• Thomas Moore is an oceanographer in 20-member team, decadal forecasting; advocating with CSIRO for Pangeo-style approach
• C3DIS conference

• every keynote name-dropped Pangeo
• 1-day tutorial
• Exposure to scientists and IT side

• Spinning up Pangeo deployments on AWS and National HPC
• Strong demand in CSIRO (5000+ people) - will want JupyterHubs and Binders
• Users all want it
• Lots of questions about storage formats (zarr vs. netCDF, etc.)
• Interest from IMOS group: all their data is in an Amazon S3 bucket
• CSIRO CIO became aware of Pangeo
• How did Binder deployment go?

• James was able to spin up a Pangeo binderhub on AWS using EKS CTL with some hacking
• Couldn’t use Amazon container registry

• https://github.com/pangeo-data/pangeo/issues/628

• Matt - what would it take to track costs back to individual users?

• Dev Seed is interested in investigating this

• Some discussion about the path forward for CMIP6

• What is the initial goal for CMIP6 data?
• https://github.com/pangeo-data/pangeo/issues/579
• Upload netCDF? Convert to Zarr in the cloud?
• Who is going to do the work?
• James: Australians think they need it locally, $ is being spent to
• NCI Australia, Claire Trenham, might be interested
• We can approach Open Data Programs (at AWS/GCP/ect) now

• Update on benchmarking (Rich)

• HDF5 cloud reading is slow because it is open a file over and over again
• How to make it faster?
• Rich wants input from people (like Martin Durant)

Short term coordination:

• Blog post from Thomas / James about Australian
• Connect about Azure / AWS cloud stuff

2019-05-08 1600 GMT (9am PDT, 12pm EDT)

Video Conference Link:  Zoom

Join Zoom Meeting

https://columbiauniversity.zoom.us/j/903370578

One tap mobile

+16468769923,,903370578# US (New York)
+16699006833,,903370578# US (San Jose)

Dial by your location

+1 646 876 9923 US (New York)
+1 669 900 6833 US (San Jose)

Meeting ID: 903 370 578

Find your local number: https://zoom.us/u/abcuAxbTMF

Welcome to the Team Meeting.

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Rich Signell / USGS / @rsignell-usgs
• Joe Hamman / NCAR / @jhamman
• Matt Rocklin / NVIDIA / @mrocklin
• Lindsey Heagy / Berkeley / @lheagy
• Martin Durant / Anaconda / @

60 Second Updates:

• James Munroe: There was strong representation for Pangeo at C3DIS.  That Pangeo session (5 talks) were very well attended and several conference keynote speakers highlighted the work being done with the Pangeo community.  There is lots of interest for our day long Pangeo training session tomorrow and the Hobart hackathon next week.
• Rich Signell:  Harmandeep Singh’s proposed GSoC work on a pyviz-based model viewer was officially selected,  one of only two ESIP projects selected by Google!  Maybe next year Pangeo could apply to be a GSoC org?   Ran into some issues on benchmarking (report later).  Also a bit of progress deploying Pangeo on USGS AWS “Cloud Hosting Solutions” with EKS (we now have an EKS kube cluster and I’m trying to follow the cloud federation doc).
• Matthew Rocklin: JupyterHub and Dask on Hadoop (video) Jim Crist at Anaconda has made it pretty easy to deploy a Pangeo-like stack on any Yarn cluster (Yarn is like Kubernetes or PBS/SLURM, but common on Hadoop deployments).  Anyone have an institutional Hadoop/Spark cluster and want to try deploying JupyterHub and Xarray on it?
• Joe Hamman: Scaling/scheduling on kubernetes, blog post coming. Blog series on the pangeo-model in progress -- will be asking for input. Intake-stac works!  
• Lindsey Heagy: Chatted with PIMS folks - interested in expanding deployments of pangeo-style custom JupyterHubs (targeted at different research domains)
• Martin Durant: Intake 0.5.0 released with new Panel-based GUI, links nicely with generic Xarray viewer project with GSoC student (co with Rich S). netCDF with s3fs works now.
• Ryan: Continuing to do datastore stuff (https://pangeo-data.github.io/pangeo-datastore/),  Did some cost analysis of GCP clusters - https://gist.github.com/rabernat/21f97865129a97ca561f8448cd88b19a
  Concerned about sustainability

Agenda items:

• Google cloud billing breakdown and cost optimizations that @jhamman has been working on

• Billing binder/repo here: https://github.com/pangeo-data/gcp-billing
• Summary here: https://github.com/pangeo-data/pangeo-cloud-federation/issues/264#issuecomment-490378624
• Need eyes on: https://github.com/dask/dask-kubernetes/pull/145 and https://github.com/dask/dask-kubernetes/pull/147
• Writing a technical blog post about how to implement this in practice.   

• Gauging interest in an Xarray-related sprint at SciPy
• Update from @rsignell-usgs on benchmarking efforts (Rich)
• Using Pangeo components for back-end service processing (Rich)

• How to set up a back-end service?
• Matt: start small

Short term coordination:

• Ryan create webpage for talks  / pangeo
• Rich create youtube channel for Pangeo (add existing talks to channel)
• Ryan will submit xarray sprint proposal
• Rich will open an issue about web service use case

2019-05-01 2000 GMT (1pm PDT, 4pm EDT)
Instead of appear.in, this week we are using  Zoom Video + Audio Meeting from any device:
https://columbiauniversity.zoom.us/j/535776170

One tap mobile:
+16468769923,,535776170# US (New York)
+16699006833,,535776170# US (San Jose)

Dial by your location:
+1 646 876 9923 US (New York)
+1 669 900 6833 US (San Jose)

Meeting ID: 535 776 170

Find your local number: https://zoom.us/u/abcuAxbTMF

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat
• Joe Hamman / NCAR / @jhamman
• Kevin Paul / NCAR / @kmpaul
• Julia Kent / NCAR / @jukent
• James Munroe / MUN / @jmunroe

60 Second Updates:

• Ryan: finishing up data catalog website
• Joe: lots of progress (w @scottyhq) on shareable home directories and other binder stuff
• Kevin: trying to revive datastore WG
• Martin Durant: Intake 0.5 released with new
• Julia: working on slides for EarthCube
• Rich: GSOC intern accepted to ESIP, Felipe Fernandez wants to help with conda stuff
• James: BinderHub running on AWS EKS. Prepping tutorials for C3DIS

Agenda items:

• Demo of new Intake-STAC package https://mybinder.org/v2/gh/pangeo-data/intake-stac/master?filepath=examples?urlpath=lab (Joe)
• Demo of new pangeo cloud data catalog website (Ryan)
  https://pangeo-data.github.io/pangeo-datastore/
• Mounting NFS in dask-worker-pods: https://github.com/pangeo-data/pangeo-cloud-federation/issues/234, https://github.com/pangeo-data/pangeo-cloud-federation/pull/255/files
• Summer Meeting Update

Short term coordination:

• .

2019-04-24 1600 GMT (9am PDT, 12pm EDT)

Video Conference Link:  https://appear.in/pangeo

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

Name / Institution / GitHub handle

• Rich Signell / USGS / @rsignell-usgs
• Ryan Abernathey / LDEO / @rabernat
• Joe Hamman / NCAR / @jhamman
• James Munroe / MUN / @jmunroe
• Chiara Lepore /LDEO/@chiaral
• Jim Bednar / Anaconda / @jbednar
• Lindsey Heagy / UC Berkeley / @lheagy
• Sean Harkins / Development Seed / @sharkinsspatial
• Kevin Paul / NCAR / @kmpaul
• Julia Kent / NCAR / @jukent
• Anderson Banihirwe / NCAR / @andersy005
• Kevin Donkers / UK Met Office / @kaedonkers

60 Second Updates:

• Ryan : working on machine learning
• Rich Signell: Unfortunately not attending pangeo dev meeting or ocean hack week, a simple benchmark file reading example , Pangeo at USGS aiming for EKS as opposed to OpenShift. “Ncview”-like dashboard proceeding nicely, now with time series, try the dashboard on pangeo’s binder.   Pangeo+PyViz-based paper on coastal ocean modeling just published
• Joe: Working on hydro.pangeo.io, nasa.pangeo.io, pangeo-stacks, etc. Update on Intake-STAC plugin. Summer meeting
• Chiara Lepore: Working on the convective environments code
• James Munroe: Trying to deploy PangeoHub/Binderhub on AWS for in preparation of C3DIS and with related training materials.
• Lindsey: EOS article
• Jim Bednar: Was busy filing final NASA grant report; completed and now starting new grants and projects. Our recent PyViz work with ERDC was summarized nicely by Dharhas Pothina.
• Sean Harkins:  Just getting involved with the Pangeo ecosystem.  We are currently working with NASA DAACs and would like to make some long term recommendations to them on storage formats.  We would like to contribute where possible to the format benchmarking project and we do core development on the STAC spec and the sat-api reference implementation of the API spec portion of STAC and would like to assist in this area wherever possible.
• Kevin Paul: Working on pangeo-data/benchmarking; developing a strategy for strong and weak scaling study on Cheyenne, Casper, and Wrangler; Developing a set of canonical xarray operations (global mean, climatology, anomaly, etc.) for the study; Thinking of framework based on dask-mpi;
  https://github.com/pangeo-data/benchmarking/blob/master/compute_study.md
• Kevin Donkers: “Hero Demo” - turnkey infrastructure implementation of Pangeo on Azure, Intake catalog of live Met Office forecast model outputs (stored on AWS Earth, also see https://github.com/MetOffice/aws-earth-examples), loading model output as hypotheticube in Pangeo Jupyter Lab session, visualise using PyViz ecosystem, annotate visualisation with touchscreen device by deploying plots to Bokeh server from Pangeo, 3D visualisation of forecast data; Talk at C3DIS conference in Canberra, Australia; Workshop at Pangeo Hackathon at CSIRO Hobart, Australia.

Agenda items

• Rich: Intake ⇔ ISO records useful?
• Rich: Worth using the storage format benchmarking test suite?  (answer from Ryan: NO! Too complex)
• Summer Meeting

• http://pangeo.io/meetings/2019_summer-meeting.html
• Should we merge this PR before finalizing URL:
  https://github.com/pangeo-data/pangeo/pull/599 
  DONE

• Intake-STAC

• https://nbviewer.jupyter.org/github/jonahjoughin/STAC-Intake/blob/master/stac_intake_example.ipynb

• Longer report from Kevin (UK Met)

• Efforts focused on “hero demo” which shows off all the magic capabilities of Pangeo
• Annoying task: get access to forecast data from the past week, currently involves lots of manual toil
• Trying to make it easier to access S3 data storage using intake catalog

• Intake hypothetic (link?)
• Iris hypothetic (link?)
• NCML: https://github.com/pydata/xarray/issues/2697

Short term coordination:

• Kevin or Kevin will ping the cloud datastore WG list for a new meeting
• Rich is going to ESIP Summer meeting.  Session ideas due Friday.  Any other Pangean attendees?
• Chiara + Kevin forecast data in the cloud

2019-04-17 8pm GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

• name / institution / GitHub handle
• Ryan Abernathey / Columbia U. / rabernat
• Jim Bednar / Anaconda / jbednar
• Lindsey Heagy / UC Berkeley / lheagy
• Matthew Rocklin / NVIDIA / mrocklin
• Julia Kent / NCAR / jukent
• Anderson B / NCAR / andersy005
• Joe Hamman / NCAR / jhamman

60 Second Updates:

• Jim Bednar: Submitted full version of http://earthml.pyviz.org, a good resource for learning about: geoscience visualization using PyViz, Intake for image data catalogs, and xarray image preprocessing for ML pipelines.  Will have PyViz tutorial at SciPy.
• Lindsey: picking up the EOS article
• Matt: Pass!
• Julia Kent - preparing to go to the C3DIS (?), preparing presentation and demo on ESMlab, intake ESM, wants to get involved in xarray docs
• Anderson - working on NCML file format
• Rich Signell - struggled trying to rechunk a dataset, working on HSDS vs Zarr comparison
• Joe Hamman - docker stuff, pangeo proposal
• Ryan - https://xhistogram.readthedocs.io/en/latest/ 

Agenda items

• Summer meeting details (21-23 August 2019):

• http://pangeo.io/summer-meeting.html
• Registration: https://docs.google.com/forms/d/e/1FAIpQLSemu3o7Noo-B620tvIhIVrDCJHiBjYaTRin3RlBd-PN_CnVFw/viewform
• Matt & Rich: liked the format of ~25
• Matt: could have one day for the core group, another day for a broader group
• Do we include other disciplines (e.g. Pan Neuro, Astro, CZI)
  Other groups (e.g. cloud providers, companies, funders)
• Need to update the website with past meetings
• Scott Henderson is lead on coordination

• Blog / twitter discussions around Xarray:

• https://medium.com/pangeo/supporting-new-xarray-contributors-6c42b12b0811
• https://medium.com/pangeo/thoughts-on-the-state-of-xarray-within-the-broader-scientific-python-ecosystem-5cee3c59cd2b
• https://twitter.com/mrocklin/status/1117792874001772544

• ESMlab (https://github.com/NCAR/esmlab)? - it seems like a case study in the fragmentation of the xarray extension ecosystem

• ESMLab aims:

• To abstract xarray’s boilerplate code required for some `time` coordinate dependent operations such as weighted means, computing weights, etc
• To provide easy-to-use functionality to users (w/ limited xarray and/or dask experience) who are transitioning from NCL to Python.
• To preserve metadata within the netCDF files, potentially (domain) model specific metadata that don’t get preserved during xarray’s operations.

• The original code came from Matthew Long’s research/work (Scientist/Oceanographer @NCAR)
• Planning to eventually push some of the functionality that is general enough upstream in xarray.
• http://matthewrocklin.com/blog/work/2018/01/27/write-dumb-code
• https://mpas-analysis.readthedocs.io/en/master/
• https://cfpython.bitbucket.io/docs/latest/

• Recent progress from @yuvipanda on https://github.com/pangeo-data/pangeo-stacks/

Short term coordination:

2019-04-10 (9am PDT, 12pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

• name / institution / GitHub handle
• Rich Signell / USGS / @rsignell-usgs
• Martin Durant / Anaconda / martindurant
• Lindsey Heagy / UC Berkeley / @lheagy
• John MacCarthy / LANL /
• Joe Hamman / NCAR / @jhamman
• Ryan Abernathey / LDEO / @rabernat

60 Second Updates:

• Rich:  working with John Ready (HDFgroup) on comparing:

• Parallel byte-range requests to HDF5/NetCDF4 file as single object (using HSDS, which is now open source)
• Parallel Dask  access to same dataset in Zarr

Will run with dataset with same chunking, compression and number of CPUs):

Struggling a bit with best tool for rechunking (nco, nccopy, python w/dask)

GSoC proposals closed yesterday, have  a great candidate for project to build ncView-like tool of future with pyviz/panel app

• John MacCarthy

• Seismologist at LANL, starting to work on larger problems that are making use of Pangeo tools

• Lindsey:

• Chat with folks a LBL / NERSC yesterday - motivating some work of large data in JupyterLab. I will be getting into Xarray
• AGU session submissions
• Scipy BoFs call

• Martin:

• anacondaCON
• Intake release
• Dask PR to interface to TileDb: https://github.com/dask/dask/pull/4679

• Joe:

• Lots of work on pangeo-cloud-federation
• Working on two proposals so busy on that front

• Jim Bednar:

• New versions of most PyViz projects released for AnacondaCON now that Bokeh 1.1 released; many announcements to go out soon.  conda install -c pyviz -c bokeh … to get the new versions.

Agenda items

• Add items before meeting
• Presentation from Jonathan MacCarthy about seismic data processing
• Update on AWS deployments (https://github.com/pangeo-data/ACCESS_K8S)  

• using eks-ctl

• Crowd sourcing a list of EO datasets for cloud storage
  (https://github.com/pangeo-data/pangeo/issues/588)

2019-04-3 8pm GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://appear.in/pangeo

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

• name / institution / GitHub handle
• Ryan Abernathey / LDEO / rabernat
• Joe Hamman / NCAR / jhamman
• Lindsey Heagy / UC Berkeley / lheagy
• Julia Signell/ Anaconda/ jsignell
• Julia Wagemann / ECMWF , University of Marburg
• Rob Fatland / UW / robfatland
• Amanda Tan / UW / amanda-tan
• Matt Rocklin / NVIDIA

60 Second Updates:

• Joe: Working on AWS deployment, tuning binder, big security update on binder and jupyterhub
• Lindsey Heagy: Simpeg example for EOS article, AGU session calls approaching

• Notes here
• Wants collaborators for open source workshop

• Julia S: Pyviz release happening today (including panel for dashboards, layouts refactor in Bokeh), Army funding for pyviz
• Julia W: submitted a proposal to Earth on AWS for cloud credits for ECMWF project,
  https://github.com/pangeo-data/benchmarking (Julia + Kevin + James Munroe)
• Rob F: Working on education repo
  https://github.com/pangeo-data/education-material , will soon send invite to join working group
• Amanda Tan: Working with Scott on deploying icesat2 pangeo
• Matt Rocklin: Jim Crist ran into a scaling problem on binder, Chan Zuckerberg institute

Agenda items:

• Keeping notebooks fresh        

• Ocean group is getting ready to publish notebooks. Concerned that notebooks created today will not work in two months
• Lindsey: has done some work on this with her thesis and SimPEG
  https://github.com/opengeophysics/testipynb https://github.com/simpeg-research/heagy-2018-aem 

Just checks notebooks for errors (doesn’t verify content)

• Julia S: pyviz uses https://github.com/pyviz/nbsmoke which does smoke testing and linking  and works for bokeh/holoviews output.
• Examples.dask.org solution

• https://github.com/dask/dask-examples/blob/master/.travis.yml
• https://readthedocs.org/projects/nbsphinx/

• Pangeo website currently uses:
  https://sphinx-nbexamples.readthedocs.io/en/latest/ 
• Matt: two problems: a) verifying notebooks b) big data scalability
• Ryan: using a single docker image for both binder and hub https://github.com/pangeo-data/pangeo-stacks 
• Lindsey: Where to continue discussion?
  https://discourse.jupyter.org 

• Data store progress:

• Experimenting with CI on cloud datasets using intake
• https://github.com/pangeo-data/pangeo-datastore 
• Required metadata:
  https://github.com/pangeo-data/pangeo-datastore/issues/19 

• NASA IceSat2 Workshop Hackathon

• New satellite just launched
• eScience institute hosting a workshop in june
• Pushing ball forward on AWS Kubernetes stuff

Short term coordination:

• Lindsey will start a discourse issue about testing of notebooks + post link on gitter

2019-03-27 16:00 GMT (9am PT, 12pm EST)

Video Conference Link: https://appear.in/pangeo

Welcome to the Team Meeting.

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

• name / institution / GitHub handle
• Ryan Abernathey / LDEO / rabernat
• Joe Hamman / NCAR / jhamman
• Lindsey Heagy / UC Berkeley / lheagy
• Martin Durant / Anaconda / martindurant
• Naomi Henderson/ LDEO/naomi-henderson
• Rich Signell/ USGS/ rsignell-usgs
• Kevin Paul / NCAR / kmpaul
• Claire Trenham / CSIRO / hot007
• Julia Wagemann / ECMWF, University of Marburg / jwagemann
• Stephan Siemen / ECMWF / StephanSiemen
• James A. Bednar / Anaconda/ jbednar
• Julia Kent / NCAR / ?

60 Second Updates:

• name: update
• Joe: at NASA ESDSWG last week.  
• Lindsey:

• AGU - OSSI: https://github.com/agu-ossi/best-practices/issues
• Google Cloud HPC day last week:
  https://hackmd.io/0zCZin0VSeumZ3DLSOStLQ?view

• Martin: helping with Intake and storage
• Naomi: CMIP6 catalog issues
• Ryan: CMIP6 Binder, Google HPC Day NYC
• Rich: (1) at UNIDATA last week, talking about use case of comparing efficient access to netCDF in object storage (pre load metadata + parallel byte range requests) vs. Zarr.  (2) Would like to have a discussion with Pangeo community about what to do when the free research credits run out.  On premise computing is highly subsidized for project use.   Should Cloud use be subsidized also?   What strategies are institutions using?
• Kevin:  preparing for upcoming trip to CSIRO/C3DIS; new https://github.com/pangeo-data/education-materials repository
• Claire: Quiet today (calling from Aus … 3am). Will be at CSIRO hackathon.
• Stephan: ECMWF is working on getting involved with Pangeo. Already using dask/xarray. Julia has more info…
• Julia W.: Working with ECMWF to try Pangeo. Applying to AWS for credits.
• James: Anaconda has new funding from USACE and NASA to work on geopandas, dask, pyviz, and related things over the next 12 months (until March 2020):

• improving Bokeh/HoloViews tools for collecting user inputs (annotations, ML training labelling, ROI selection, etc.),
• adding additional native types to Datashader (e.g. polygons and area),
• adding zonal statistics, Euclidean distance calculations, and viewshed calculations to Datashader,
• integrating some limited 3D viz capabilities into Panel and/or HoloViews,
• supporting static HTML/JS export from Panel and other Panel improvements,
• speeding up GeoPandas with Cython (Joris Van den Bossche),
• adding GeoPandas-like capability to Pandas and Dask using Pandas extension arrays,
• supporting GPU dataframes in Datashader,
• speeding up Conda, and
• developing and providing optimized workflows for ML and simulation using climate data.

• Julia K.: Getting ready for CSIRO/C3DIS visit where I’ll be presenting NCAR+Pangeo use.

Agenda items
Let's collect all potential agenda items here before the start of the meeting. We will then attempt to create a coherent agenda that fits in the 60m meeting slot. If there are similar items try and group them already.

• Demo of new CMIP6 examples binder and discussion of the potential role of Pangeo within CMIP / ESGF:
  https://github.com/pangeo-data/pangeo_cmip6_examples 
• Discussion from Julia Wagerman (@jwagemann) about possible applications of Pangeo at ECMWF

• Julia is testing different big data technologies for processing earth data (e.g Dask, Spark, etc.)
• Stephan is interested in contributing more broadly to Pangeo on behalf of ECMWF
• European Weather Cloud
• Jim Bednar: Anaconda knows how to do these sort of optimizations and intercomparisons (e.g. spark vs dask), and we do have funding for some limited such comparisons, but it’s a bit of a quagmire and we would only jump into it for very specific purposes with well-defined goals, to keep it manageable. E.g. if there is an existing relatively straightforward dask example and people would like to see that one tried with spark, we could do that.
• Rasdaman is not actually used operationally at ECMWF

• Update on summer meeting (@jhamman)

• https://github.com/pangeo-data/pangeo/issues/573
• Seattle, WA, Aug. 21-23, EScience Institute, save the date
• Followed by Ocean Hack Week

• Crowdsourcing help to fill in the new awesome-open-climate-science list:
  https://github.com/pangeo-data/awesome-open-climate-science (@rabernat)
• Strategies for getting institutions on board with Cloud (@rsignell)

• NSF Cloud Access Call:
  https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=505591 
• How much money is appropriate for cloud computing in a grant?
• ECMWF is building a private cloud with elasticity to public cloud
• NASA managed to make this transition

2019-03-20 8pm GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://appear.in/pangeo

Welcome to the Team Meeting.

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

• name / institution / GitHub handle
• Ryan Abernathey / LDEO / rabernat
• Naomi Henderson/ LDEO/ naomi-henderson
• Julia Signell/Anaconda/jsignell
• Martin Durant / Anaconda / martindurant
• Brendan Collins / Anaconda / brendancol
• Rob Fatland / UW / robfatland
• Amanda Tan / UW / amanda-tan
• John Kirkham / NVIDIA / jakirkham
• Matthew Rocklin / NVIDIA / mrocklin

60 Second Updates:

• name: update
• Ryan: getting CMIP6 stuff ready for Barcelona workshop next week
• Naomi: producing CMIP6 zarr data
• Julia: working on a GUI for exploring intake catalogs, demo last week
• Martin: had some time off :) - trying to work a bit with streamz library
• Brendan: new, listening in, going to be working on Pangeo in the future, GIS analyst background (nature conservancy); reading GDAL source code, polygon rasterization, adding statistics functions to datashader
• Rob: painfully learning the basics of xarray and dask, combining various ocean data sources, https://github.com/pangeo-data/pangeo_ocean_examples/issues/15 
• Amanda: NASA Access proposal - launch AWS kubernetes clusters via hubploy, lots of challenges with AWS EKS, spot instance and autoscaling
• John K: conda-forge, dask, zarr; recently moved to nvidia
• Matt R: general dask work, see recent blog post on Dask + GPUs, Jim Crist working on a dask hub  / dask gateway

Agenda items
Let's collect all potential agenda items here before the start of the meeting. We will then attempt to create a coherent agenda that fits in the 60m meeting slot. If there are similar items try and group them already.

• Recent progress on unifying Pangeo cloud datastore (Ryan)

• https://github.com/pangeo-data/pangeo-datastore
• Master catalog + sub catalogs
• Running CI on catalog

•  Education Working Group

• Rob F: working with middle school students. Questions arise: how big is the audience? What skills can we assume? How are we going to approach building material?
• When should we teach packaging?
• How to overcome existing biases / expectations about how software should work?
• Rob’s pull request with pedagogical philosophy:
  https://github.com/pangeo-data/education-material/pull/1 
• Why don’t people use python? Their advisor (at least at Harvard)
• https://github.com/pangeo-data/pangeo/issues/575
• https://github.com/pangeo-data/education-material
• Rob & Amanda (UW) are ready to lead the education
• Brandon interested in educational materials around ESRI/GDAL style situations
• For profit companies might be interested in co-developing materials

• Anaconda has pulled back a little bit
• QuanSight might be a good partner?

• Intake GUI demo - if enough people haven’t seen it. Looking for feedback on look and flow (Julia 3 min):

• PR: https://github.com/intake/intake/pull/286
• Plan: https://github.com/intake/intake/issues/225

2019-03-13 4pm GMT (9am PDT, 12pm EDT)

Video Conference Link: https://appear.in/pangeo

Welcome to the Team Meeting.

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

• name / institution / GitHub handle
• Joe Hamman / NCAR / jhamman
• Tom Augspurger / Anaconda / TomAugspurger
• Martin Durant / Anaconda / martindurant
• Rich Signell/ USGS/ rsignell-usgs
• John Readey/ The HDF Group/ jreadey
• Jacob Tomlinson / Met Office / jacobtomlinson
• Matthew Rocklin / NVIDIA / mrocklin
• Julia Signell / Anaconda / jsignell
• Jim Bednar / Anaconda / jbednar

60 Second Updates:

• name: update
• Joe: Redeployed pangeo-binder
• Tom: Working on UCX, verified basic operations work on Cheyenne. Starting to go through ML workflow notebooks
• Martin: GUIs for Intake coming along
• Rich (chief evangelist): Met with USGS National Water Model people, talking to Rhys Ryan about using Pangeo
• John: HDF cloud data service (conversation topic below)
• Jacob: Rebuilding k8s cluster using EKS, adding GPU nodes, Theo has been working on rolling zarrs
• Matt:

• HPC blogpost: https://github.com/dask/dask-blog/pull/6
• NumPy + GPU blogpost: https://github.com/dask/dask-blog/pull/15
• John Kirkham now at NVIDIA.  Gathering requirements for image processing.  Let us know if you’re interested in chatting.

• Julia S.: GUI for Intake, navigator, plotting capability coming soon
• Jim: Working with Julia S. and Philipp Rudiger (who is answering Pangeo questions/issues with PyViz tools)

Reports and celebrations

Things people should know about.

• Pangeo/hubploy update - Joe

• https://github.com/pangeo-data/pangeo-cloud-federation
• Three hubs running now (dev, ocean, nasa)
• AWS-EKS support was added to hubploy
• UK Met Office is working on a EKS setup now, will post workflow/config

• Pangeo-binder update - Joe

• Updated chart (now up to date with mybinder.org)
• Still working on https, node pool optimizations, and …

• Trying to simplify environments btw hub and binder (Ryan)

• Refactored ocean.pangeo.io and pangeo_ocean_examples to use the same underlying docker image
• Docker images use https://github.com/pangeo-data/pangeo-stacks/
• We should do this more, to make sure all example repos are runnable in all pangeo environments
• Can we use a docker image on HPC?

Agenda items
Let's collect all potential agenda items here before the start of the meeting. We will then attempt to create a coherent agenda that fits in the 60m meeting slot. If there are similar items try and group them already.

• AWS EKS - Why is it so hard to use and is there any hope? Should we just expect to use KOPS for the foreseeable future? - Joe
• HDF5 as a backend to Zarr? - the idea: read metadata from HDF5, write to Zarr metadata, then issue byte-ra        nge requests to get chunks instead of reading Zarr objects.   - Rich Signell/John Readey (10 min?)

• New HDF5 release provides info on chunk locations.  See: H5Dget_chunk_info here: https://forum.hdfgroup.org/t/hdf5-1-10-5-release-newsletter-168-the-hdf-group/5478
• Should enable Zarr to just store pointers to chunk locations which are accessed via range GETs
• I’m using these technique in the HDF Data service: https://github.com/HDFGroup/hsds/blob/master/docs/design/single_object/SingleObject.md
• There’s a Docker image you can use to dump chunk info into json: https://hub.docker.com/r/hdfgroup/store_info.py

• Intake GUI demo . Related github issues:

• https://github.com/intake/intake/pull/286
• https://github.com/intake/intake/issues/225 - Julia (2min)

• Improving our weekly meetings -Joe

• Frequency of these meetings. Can we change to twice a month?
• Meeting format: https://github.com/pangeo-data/pangeo/issues/570

• Summer Meeting

• Save the date: 8/21-23

2019-03-06

In attendance:

• Ryan Abernathey / LDEO
• Joe Hamman / NCAR
• Scott Henderson / UW
• Rich Signell / USGS
• Kevin Paul / NCAR
• Rob Fatland / UW
• Tom Augspurger / Anaconda
• Kevin Tyle / U Albany
• Lindsey Heagy
• Julia / NCAR
• Anderson / NCAR
• Jiawei Zhuang / Harvard
• Amanda Tan / UW

Quick Reports:

• Joe working on cloud clusters, AWS Kubernetes cluster

• staging.nasa.pangeo.io is up

• Scott Henderson also working on AWS cluster, xarray PR for opening HDF5 file-like objects (possible remote access on cloud storage)

• https://github.com/pydata/xarray/pull/2782

• Rich: GSOC intern likely to work on pyviz based universal visualization tool, meeting with USGS brass to talk about Pangeo and HPC on the Cloud as a reliable solution for USGS Hazards research.  Spoke with Kevin Jorissen from Amazon about HPC on AWS.  He has a slide showing 90% scaling at 1000 cores with WRF
• Rob: ocean sciences AWS pangeo, documentation and outreach, offered to collect email addresses
• Tom A: Infiband configuration conda package, going through ML working group examples
• Kevin Tyle: XSEDE allocation renewed, zarrified datasets in jetstream, Rich is involved
• Lindsey H: submitted Pangeo NSF grant w/ Joe, Fernando, Laurel; standards for finite volume meshes in geophysics
• Julia: working on tracking down floating-point differences between IDL vs xarray / dask version of OHC (https://github.com/NCAR/notebook-gallery)
• Anderson: working on Pangeo binder issues regarding grafana + prometheus
• Amanda: Pangeo EC RCN - NSF PM indicated the Pangeo concept was too broad.

Status Check on Larger Issues:

• Proposals

• UW EarthCube RCN (due March 15)

• Program officers discouraged from a general pangeo RCN
• Potential more focussed topics: Water, remote sensing
• Pangeo outreach proposal - might be better suited for the “Data capabilities” call

• Berkeley EarthCube Data Capabilities

• Title: “Jupyter meets the Earth: Enabling discovery in geoscience through interactive computing at scale”

• Clusters

• Consolidated many hubs into one repo, which is deployed automatically via hubploy: https://github.com/pangeo-data/pangeo-cloud-federation/
• Consolidated kubernetes clusters -> save money
• Pretty new templates: http://staging.ocean.pangeo.io 
• Authentication with Globus / ORCID

• ML working group

• Second meeting was held yesterday (post link to minutes)
• Reviewed different workflows to identify common challenges
• Problem getting separated into two steps: transform / preprocess + model training
• How to improve preprocessing step
• Identifying bottlenecks in xarray (isel) / dask (distributed shuffle)
• https://github.com/pangeo-data/pangeo/issues/567
• https://paper.dropbox.com/doc/Meeting-notes-ML-WG-Meeting-2--AY1SnAjn58vayYTG8mNXcN7uAQ-xOE0uGGLjsnJAyw5At51x

• Grid working group

• Minutes: https://docs.google.com/document/d/19PTpG0EDZiW-vULQ1EOjvGTbB-S3_0D5wI2INyFroB4/edit 
• Draft blog post (do not publish):
  https://medium.com/pangeo/working-with-gridded-data-in-python-6f46e098907d 

• Lustre on cloud

• https://jiaweizhuang.github.io/blog/fsx-experiments/
• Funding from Amazon to develop MPI HPC on AWS

• Running FV3 up to 1000-2000 cores
• I/O limited

• AWS just launched a Lustre parallel file service
• Ran a classic pangeo / netcdf HPC deployment in Amazon HPC service
• Perhaps easier way to deploy Pangeo in cloud
• Big advantage: just works out of box with HDF data
• Can access S3 data from Lustre filesystem via AWS magic
• Joe: alternative is to have models write directly to object store (Unidata netCDF / zarr integration)

•  ESGF CMIP6 data to cloud: https://github.com/pangeo-data/esgf2xarray

• Coordinate with intake-esm: https://github.com/NCAR/intake-esm  

• Summer meeting

• Looking at Aug 26-28 in Seattle (UW eScience)

Coordination Items:

• Jiawei sends some examples to Julia of IDL -> xarray translations

2019-02-27

In attendance:        

• Jim Bednar / Anaconda
• Raphael Dussin / LDEO
• Julia Kent / NCAR
• Anderson / NCAR
• Lindsay Heagy / UC Berkeley
• Rich Signell / USGS
• Matt Rocklin / NVidia
• Tom Augspurger / Anaconda
• Kevin Paul /NCAR
• Joe Hamman / NCAR
• Jacob Tomlinson / Met Office

Quick Reports:

• Jim Bednar:

• Julia Signell working on an ncview-style data viewer project using Panel, hvPlot, and Intake.
• Playing with Pandas extension arrays to use ragged arrays, in service of datashader rendering polylines of arbitrary lengths.  Let me know if someone else wants to use ragged arrays in other libraries.

• Raphael:

• Adding ECCO/mld notebook to pangeo ocean examples

• Anderson:

• Benchmarks for Zarr + HDF5

• Julia:

• Working to migrate two workloads around ocean heat content + temperature and salinity to use Xarray.
• Waiting on feedback from original authors

• Lindsey (need to leave at half-way through today):

• lots of grant writing - submitted initial draft to UC Berkeley, should have full draft by end of the week

• Rich: still playing with hvplot+panel dashboards. Q: Where best to ask Panel/hvplot questions? A: (from Jim) gitter for simple usage questions, panel/hvplot github for possible bugs/feature requests, stackoverflow for usage questions that need more extensive code examples than is feasible on gitter. Plan to explore performance of zarr access vs. custom hdf5 access with Jim Gallagher from OPeNDAP.
• Tom Augspurger: UCX (infiniband) comms for dask. Making conda packages and benchmarking soon.
• Matt:

• JupyterLab + Bokeh dashboard example with Ian:
  https://github.com/ian-r-rose/jupyterlab-bokeh-server/
• Drafting potential work items around Xarray (please help!):
  https://docs.google.com/document/d/1Qk8fN4NvXc2OkaiKW4bbnX4pgoV3uy0tZUiqtO2xp_E/edit?usp=sharing 
• Other Dask/GPU things

• Jacob:

• Cron jobs within Kubernetes, getting into Papermill

• Making a CLI that looks like crontab (krontab), but generates the excessively long yaml file necessary to manage cron jobs in Kubernetes
• https://github.com/jacobtomlinson/krontab

• Theo is looking at appending and dropping chunks with Zarr datasets (maintaining rolling archives of forecast data)

• Kevin:

• Not much!

• Joe:

• https://github.com/pangeo-data/pangeo-cloud-federation
• Pangeo helm chart now depends on JupyterHub 0.8 chart

Longer Discussions

• Dask-Kubernetes async branch.  Please test!
  https://github.com/pangeo-data/pangeo/issues/562 

Short-Term Coordination:

2019-02-20

In attendance:        

• Matt Ueckerman / PODPAC
• Mark Shapiro / PODPAC
• Tom Augspurger / Anaconda
• Raphael Dussin / LDEO
• Chiara Lepore / LDEO
• Rich Signell / USGS
• Jim Bednar / Anaconda
• Joe Hamman / NCAR
• Amanda Tan / UW
• Scott Henderson / UW
• Wu-Jung Lee / UW

Quick Reports:

• Joe Hamman: slowly making progress on cloud deployments
• Tom A: infiniband-based communications for Dask
• Raphael: working on migrating user spaces on jupyter clusters
• Chiara: about to start working on ERA-5 data in cloud context
• Rich S: playing with hvplot + panel, making a demo dashboard for the HRRR model
• Jim Bednar: 3D capability of plotly, integrate into pyviz, Julia Signell working on “ncview-like” application, working with Grey Nearing (U. Alabama) as part of NASA project, who is moving from Matlab to python tools.

• Philipp Rudiger has developed initial quadmesh rendering for datashader, with 100X speedup over treating it as a trimesh
• Datashader now has various geo tools for xarray rasters, e.g. hillshading, slope, aspect, NDVI, bump mapping

• Amanda is putting AWS clusters into hubploy, also Pangeo RCN proposal
• Scott Henderson gave ESIP presentation
• Wu-Jung: thinking about how to incorporate Pangeo + OceanHackWeek

Longer Discussions

• Podpac Introduction

• (please post link to slides)
• https://github.com/creare-com/podpac
• Automated pipeline for geospatial data wrangling
• Ingesting NASA data on AWS / OpenDAP
• NASA-funded SBIR
• Want to avoid duplication, maximize value added by PODPAC
• (please post link to demo notebooks)
• Can send calculations through AWS lambda
• Using xarray under the hood
• “Wrapping” cloud based datasets

• Cluster Refactor (Joe + Ryan + many others)

• Working on consolidation of continuous deployment of pangeo cloud-based clusters using hubploy
• Yuvi showed up and helped move it forward
• Discuss auth:
  https://github.com/pangeo-data/pangeo-cloud-federation/issues/60 

• Evidence: user metrics or five powerful user stories?
• Don’t want any manual steps
• Google probably fine

• Pangeo publication policy

• What do we cite?
• https://github.com/jupyter/governance/blob/master/papers.md

• Pangeo Summer Meeting?

• Combine with UW OceanHackWeek?
• SciPy? BoF? SciPy sprints?

• Working Groups

• Add to calendar / website
• Add regridding working group?

Short-Term Coordination:

• Reach out to Jim Bednar if interested in 3D plotting in pyviz
• Collect pangeo talk videos on pangeo website
• PODPAC folks weigh in on regridding discussion:
  https://github.com/pangeo-data/pangeo/issues/356 

2019-02-13

https://columbiauniversity.zoom.us/j/666242279

In attendance:

• Name / Institution / @github
• Lindsey / UC Berkeley / @lheagy
• Joe Hamman / NCAR / @jhamman
• Rich Signell / USGS / @rsignell-usgs
• Ryan Abernathy / Lamont / @rabernat
• Kevin Paul / NCAR / @kmpaul
• Amanda Tan / UW eScience /@amanda-tan
• Tom Augspurger / Anaconda / @TomAugspurger
• Philipp Rudiger / Anaconda / @philippjfr
• Jim Bednar / Andaconda / @jbednar

Quick Reports

• Jim Bednar: Looking for ways anaconda can contribute
• Kevin Paul: Cheyenne JupyterHub progress, met with AWS people
• Joe Hamman: testing of Cheyenne JupyterHub, 40 students used recent NCAR workshop
• Lindsey H: working on Berkeley Jupyter EarthCube Proposal
• Amanda Tan: working on EarthCube RCN Proposal
• Phillip R: works on holoviews / geoviews
• Tom A: working on dask stuff, finishing up project enabling distributed communication over infiniband (using UCX)
• Rich S: Pangeo demo using National Water Model data (in zarr format) at national water center from weather service (Fernando Salas), USGS director Jim Reilly was very engaged.  Used the new pangeo.esipfed.org cluster on the pangeo-access AWS account.   ESIP Google Summer of Code opportunity (hopefully) to build a next-generation GUI for visualizing big gridded data in Python , hopefully fitting within the GUI framework envisioned by Martin Durant
• Wu-Jung Lee: lead for Ocean Hack Week (August), wants to engage Pangeo for this; also interested in apply Pangeo for ocean acoustics
• Raphael: worked on consolidating pangeo Google cloud JupyterHubs
• Ryan: cluster refactor, web site updates, data catalogs
• Julia: notebook gallery of Pangeo examples, IDL to Python translation
• Anderson: how to make use of Cheyenne JupyterHub

Status check on larger issues

• Coordinating Pangeo scipy submissions

• Joe: resubmit xarray tutorial proposal
• Matt Rocklin wants to see a Dask JobQueue talk: Joe wants Anderson to lead this
• Rich: Darhas will likely give an EarthSim update (Jim will check)
• Jim Bednar: pyviz tutorial
• Ryan: xgcm
• BoF: a month away

• Update on Pangeo NSF EarthCube proposals from @lheagy and @amanda-tan

• Jupyter + Pangeo EarthCube proposal. One-pager overview - https://docs.google.com/document/d/1T9roA6QbYaRsLggu5oDZEpSYJ0ko6zlFkBClzzYb2YA/edit#

• Feedback and ideas welcome! Comments on the google doc, or ping me on gitter (@lheagy)
• Internal due date Feb. 25

• Pangeo RCN (Earthcube RCN). Bullet point objectives:

https://docs.google.com/document/d/1Of__qhrAPES9mhnnby9SDRsKY2qV_mFrgLhr1CGgciI/edit?usp=sharing 

• Progress report from me, @jhamman and @raphaeldussin about ongoing consolidation of pangeo cloud jupyterhubs

• Consolidating 8 existing google cloud jupyterhubs into one kubernetes cluster / hubploy repo
• Current status: broken system which is in a better state than the previous unbroken system
• Next week adding AWS clusters
• How to get unstuck: find Yuvi
• Raphael volunteers to look at error

• Anaconda Support

• Earthml.pyviz.org - online examples of machine learning examples with pyviz

• Let Jim know if you have any ideas
• Check in on pyviz channel

• Datashader for generic geographical regridding?

Short term coordination

• Organize a call on regridding / datashader with NCL team, ESMF (Ben K) and other interested parties

2019-02-06

In attendance:

• Name / Institution / @github
• Lindsey Heagy / UC Berkeley / lheagy
• Joe Hamman / NCAR / jhamman
• Alando Ballantyne / Radiant Earth / @alando46
• Rich Signell / USGS / @rsignell-usgs
• Ryan Abernathy / Lamont / @rabernat

Quick Reports

• Lindsey

• Conversations around EarthCube grant:

• Three high level themes: Dashboarding, Scaling patterns, Data catalogs
• Three use cases: hydrogeology, inverse problems (ML & geophysics), climate

• Ryan

• Pangeo/NetCDF/STAC meeting was today
• ESGF and Pangeo working group kicking off

• Joe

• Lots of xarray work: major release coming soon(!)
• GCP budget: burning way too fast

• Rich

• Re-spun-up ESIP pangeo cluster on NASA ACCESS account, have autoscaling working
• USGS Director (Jim Reilly) and NOAA Director (Tim Gallaudet) are BFF and both are pushing for more focus on big data and cloud.  They want to show USGS/NOAA collaboration, scheming a USGS/NOAA use case with Pangeo, with National Water Model output and NLDAS-2 (7TB).  Would be great to get some more NOAA folks in Pangeo, perhaps from NWS. y

• Julia

• Starting to work on making legacy IDL code work with Pangeo (Ocean Heat Content), getting used to workflow on Cheyenne
• https://jupyterhub.ucar.edu/ch
• Thinking about a blog post: https://medium.com/pangeo

• Alando:

• Radiant is moving towards deploying a Pangeo to process sentinel / modis, running up against gaps in Pangeo cluster test suite
• Researching different types of ML workflows, how to build on xarray / dask workflows

• Raphael

• Regridding for MITgcm, based on xesmf

Status check on larger issues

• GCP Spending / budget

• $200 per day (2X what we should be)
• Biggest chunk is compute
• Ryan: release pangeo into the wild - get users, this is an avenue to demonstrate value to NSF / Google and go back to ask for more funding

• Testing of Clusters

• Joe: CI for Helm chart - short term dev
• Alando: Where to document the pangeo testing setups
• http://pangeo.io/setup_guides/index.html#setting-up-pangeo-on-cloud-systems

• Discussion of catalog issues

• Rich thinks we should use ISO metadata records

• Federation of pangeo examples / binders / galleries

• Maintain a list of repos where examples live

Short-term Coordination:

• Rich chats with Lindsey about EarthCube proposal
• Alando and Joe will coordinate helm / hubploy testing
• NCAR people (Kevin) draft a blog post about Cheyenne

2019-01-30

In attendance:

Quick Reports

• Matt:

• Asyncifying Dask-Kubernetes with Yuvi
  https://github.com/dask/dask-kubernetes/pull/119
  In service of https://github.com/dask/dask-labextension/pull/36 
• Dask on MPI blogpost draft: https://github.com/dask/dask-blog/pull/9/files

• Raphael:

• Busy with other work recently
• Ocean.pangeo.io up and running

• Lindsey:

• Updates on the Jupyter + Pangeo EarthCube proposal

• In addition to dashboarding, rich views on data mentioned last time…

• Connections to HPC facilities
• Interfacing to legacy software, non-notebook apps (like paraview)

• Kevin:

• Pangeo NSF CyberTraining grant deadline to be extended, but don’t know how long; Waiting for feedback from NSF
• Struggling with dask-mpi issue (https://github.com/dask/dask-mpi/issues/18)

• Rich:

• Back from US Government furlough
• Pangeo credits running out soon at USGS, planning to move pangeo.esip.org to dev.pangeo.io type deployment on AWS.
  Would like to move to AWS because USGS has credits there
• Want to explore STAC metadata (refer to previous meeting’s notes), relative to other metadata standards like OpenSearch, JSON-LD (used by Google’s Dataset Search), and ISO 19115

• Sam:

• New!
• PhD student at Indiana university (almost done)
• Joined Planet this december
• Curious about Pangeo generally, how it works, what the development cycle, etc..
• Can universities host their own pangeo?

• Jacob:

• Looking at sparse LIDAR data and trying to scale it out with Datashader
• UK Met has been publishing a bunch of datasets recently that aren’t particularly cloud-friendly.  Looking at how to modify these.  Same work as recently
• Also interested in dashboarding.  Looking at Voila from Quantstack
• AWS professional certified

• Joe:

• Hubploy update

• https://github.com/pangeo-data/pangeo/issues/535

• Cleaning up GCP

• https://github.com/pangeo-data/pangeo/issues/533

• Helm chart update

• https://github.com/pangeo-data/helm-chart/pulls
• https://github.com/pangeo-data/pangeo-stacks

• Uses repo2docker

• CMIP6
• JupyterHub on Cheyenne

• http://juypyterhub.ucar.edu/ coming this week

• Julia:

• Still on ramping
• Translating IDL and NCL visualization capabilities to use Xarray/Pydata/Pangeo tooling
• NCL group now ready to start working on Python code

Status check on larger issues

• Hubploy meeting summary

• https://github.com/pangeo-data/pangeo/issues/535
• Git/CI based service to manage deployments on kubernetes
• Needed to be able to wrap other charts
• Yuvi opened a bunch of different issues on things that Derek had done for Pangeo that should possibly be upstreamed.
• Rich: still GCP only?  AWS still a painpoint.  

• Jacob, any effort here?
  Jacob: it’s on my list, but pretty far down.  UK Met has our own system.
• Joe: Short term priority for NASA-UW work.  Element-84 or Amanda at UW will likely work on this.
• Derek/Anaconda is likely out for this category of work

• JupyterHub on Cheyenne

• Short term priority of CTSM tutorial, which is necessary by Monday
• Worked on by Anderson and High End Services group here at NCAR
• Pangeo kernel available
• Log in with traditional credentials (Yubikey/PIN)
• Splash page asks for type of session

• General announcement that conda-forge/defaults binary package compatibility has occurred.  https://github.com/conda-forge/conda-forge.github.io/issues/616 

Short term coordination

• Matt to look at dask-mpi issues

2019-01-23

In attendance:

Quick Reports

• Joe:

• Meeting with Yuvi and folks on Hubploy.  

• Notes here:
  https://docs.google.com/document/d/1GejmF82L7yNMem1sv5-ZsA25KMuUHgvSe5E736LJRSg/edit?usp=sharing

• Pangeo Jhub and helm-chart cleanups

• Clearing out old issues and PRs
• Updating documentation, helm-chart, and docker images

• Matt:

• Asyncifying Dask-Kubernetes with Yuvi
  https://github.com/dask/dask-kubernetes/pull/119
  In service of https://github.com/dask/dask-labextension/pull/36 
• https://blog.dask.org/2019/01/23/dask-1.1.0 

• Lindsey:

• Work on Jupyter + Pangeo NSF grant

• Rich views on data
• Data catalogs
• Dashboarding
• Planning to write a one-page summary

• Scoping for the Pangeo education grant with Kevin et al

• Julia: Getting spun up, investigating other HPC jupyterhubs
• Anderson:

• Working on JHub on Cheyenne; possibly deployed by Feb workshop
• Intake plugin for CESM in the works
• PBS CI tool, pulled out from Dask-jobqueue into stand-alone project

• Kevin:

• NSF CyberTraining proposal is starting to form; Still uncertain about QuanSight involvement, but they are potentially interested

Status check on larger issues

• Pangeo-stacks: https://github.com/pangeo-data/pangeo-stacks (nothing here yet)

• Motivation: get docker images out of pangeo-helm chart
• Use jupyter-stacks-cookiecutter or build something with repo2docker
• Pain points for jupyter-stacks

• Hierarchical nature of docker challenging - have to buy into everything in the base image
• Whereas repo2docker appendices can specify each piece

• Trajectory?

• Have a set of base images and then have an appendix where environment can be specified on the fly

• Educational resources conversation

• Some tension between domain specific and reusable components
• Being generic allows us to target many groups, while being specific allows us to succeed with higher probability with a starter group.
• Concept/domain first approach provides an opportunity for horizontal scaling, which is easy for new people to come on board

• SciPy talks/tutorial deadline coming up
  https://www.scipy2019.scipy.org/participate 

• Xarray community is doing a tutorial
• Joe planning to submit a Birds of a Feather session (BoF)
• Folks will be around at the sprints

Short term coordination

2019-01-16

In attendance: Joe H., Ryan A, Kevin P, Lindsey H., Rachel P, Anderson B., Julia K, Rob F

Quick Reports

• Joe:

• Pangeo + Zarr conversations
• Talking to Amazon about moving public datasets into cloud-optimized format (ERA-5 is happening)

• Ryan

• Zarr call with ~16 people from climate and bioengineering worlds. Conversation around development and sustainability
• LENS data is approved for AWS public dataset program

• Kevin

• NSF Cyber Training Proposal coordination (due Feb 6)
• Exploring AWS Globus connection

• Lindsey

• New here and recently joined the Jupyter Team
• Looking for connection points and opportunities to contribute

• Rachel

• At AMS last week - Python/Machine Learning
• This week - visiting NCAR
• Coming up: converting radar data for use on Pangeo

• Anderson

• Jupyterhub on Cheyenne is in testing

• Julia

• New to NCAR/Pangeo project -- working through admin details
• Going through Pangeo tutorials and thinking about Pangeo publications

• Rob (Just listening;

• Mentions ‘long tail / lightweight’ oriented Zero To API idea

• When this model makes sense: Access to CDs-in-shoebox datasets

• Pangeo: Following Damien, Scott, STAC, etc etc in pangeo-data/cmr repo

• With the idea of contributing to outreach/adoption of the pangeo stack

Status check on larger issues

• Pangeo education materials

• Ryan/Columbia is submitting a proposal focused on datascience/ml education, idea is using Software Carpentry
• https://github.com/pangeo-data/pangeo/issues/411#issuecomment-452575875 

• Jupyterhub migration

• https://github.com/pangeo-data/dev.pangeo.io-deploy/issues/32 

• Lindsey: Geophysics inversion problems

• Diverse datasets and formats
• Computationally intensive inversion calculation
• Suggest putting together a repo / binder to demo workflow

• Steering council had its first meeting

• Working groups discussion, currently have data storage and machine learning groups. Discussed template for proposing new working groups
• Is the weekly check in still working? Considering monthly Town Hall Meetings, with more frequent meetings handled by the working groups

Short term coordination

• Schedule jupyterhub sprint
• LENS dataset creation - discuss chunking and other decisions

2019-01-09

In attendance:  Joe H., Jim B. Matt R., Noah B., Raphael D., Ryan A., Rich S., Alando, Ariel R.

Quick Reports

• Joe:         

• Started playing with Zarr.  
• Spoke with AWS about converting their datasets to Zarr format.  In particular ERA5 looks promising.

• Matt:

• NVIDIA, playing around with GPU stuff, recent blog post using Cupy and dask together.  http://matthewrocklin.com/blog/work/2019/01/03/dask-array-gpus-first-steps 

• Raphael:

• Working on ocean.pangeo.io .  Running into authentication errors.

• Ryan:

• Playing with STAC (spatial temporal asset catalogs)
• Playing with PyDAP
• Added new repository with thoughts, will discuss more in agenda items

• Jim:  

• Meeting with Joe and Ryan tomorrow to explore possible synergies between Pangeo and EarthML (Anaconda+NASA) and EarthSim (Anaconda+US Army Corps of Engineers).

• Noah:

• Working on large high resolution dataset that we’d like to share
• Working on calling Python machine learning models from within Fortran models

• Rich:

• Furloughed!
• Wrote rec letter to Azure to store all of Ocean Observatory (OOI) data

• Alando:

• Radiant Earth is looking to use Pangeo as the basis for a ML engine
• Working on getting RE to having a pangeo deployment running for users

• Ariel:

• Dask/Xarray for neuro time-series (and image) data.
• Setting up neuro.pangeo on GCP.

Status check on larger issues

• Pangeo.pydata.org is closing down

• https://github.com/pangeo-data/pangeo/issues/524
• Is the replacement really ready?
• Bring all Pangeo deployments under one roof: https://github.com/pangeo-data/dev.pangeo.io-deploy/issues/32

• Data repositories

• https://github.com/pangeo-data/pangeo-datastore/issues/1
• What about ISO Metadata? https://www.fgdc.gov/metadata/iso-standards
• Pycsw: http://pycsw.org/
• CSW: https://www.opengeospatial.org/standards/cat
• https://medium.com/radiant-earth-insights/announcing-the-spatiotemporal-asset-catalog-stac-specification-1db58820b9cf 

• Machine learning working group

• Noah  is doing ML for climate model parameterization
• Data processing steps

• Coarse graining - straightforward but expensive
• Training models (issues with training models and batching)

• Reproducibility is easy to get wrong -- interest in workflow engines for ML
• Need for labelled arrays for DL:
  https://twitter.com/harvardnlp/status/1080911225427496966 
• Noah’s workflow:

• Snakemake for data prep
• Sacred for tracking model configs

• What are some common problems to tackle?

• Tools for Xarray -> deep learning library pipeline

• https://github.com/rabernat/xbatcher
• http://earthml.pyviz.org/

• CMIP6 working group?

Action Items:

• Create the Pangeo ML Working Group! Noah + Alando?

• Task: Organize first meeting to determine what to do.  
  I recommend raising an issue on github summarizing this conversation and adding a doodle poll.

• Sure! - noah

• Datastore Working Group: Daniel Rothenberg + Kevin Paul

2018-12-19

In attendance: Raphael, Anderson, Kevin, Joe, Matt, Jacob, Pete

Quick Reports

• Matt: Getting oriented at NVIDIA, working on some GPU dataframe work, but not particularly related
• Raphael: working on alternate ocean dev cluster (didn’t want to break live cluster).  Ran into issues with helm upgrade.
• Kevin:

• Cheyenne/JupyterHub update: hoping to get to “some of this” this week.
• Pulling dask-mpi out of dask/distributed into its own package
  Have a pull request up that enables “import dask_mpi.initialize” within a Python script.  Designed to keep client, scheduler, and workers all deployed uniformly and nicely at the same time.
  Currently depends on distributed master, will probably need a release of that before publishing
• Julia Kent starting on January 14th

• Joe: Was at AGU last week.  Gave a tutorial for 75 attendees for four hours.  Binder deployment worked nicely.  Used JupyterLab server extension.  Taught basic XArray and Dask, then did full fledged demos, which broke things.  Might have been that we’re over-provisioning notebook pods.  
• Anderson:  

• Off last week.
• Building intake plugin for CMIP5.  Generates database of files, including institution, frequency, model, …. Intake queries this database and provides an XArray dataset.
• Holding Kevin’s hand :)
• https://github.com/NCAR/intake-cmip5 

• Jacob:

• Poached Pete from AVD in UK Met
• Exposing UK Met data in a publicly accessible way
• Pumping out four live model feeds to AWS under a Creative Commons license

• https://registry.opendata.aws/uk-met-office/ 
• Global/UK ensemble/deterministic forecast with 24 hour delay
• Kept for 7 days
• 5e5 NetCDF files
• Looking at open ended Zarrs for this kind of workflow where data continuously arrives and leaves.  Tricky because we’re doing lots of concurrent writes with S3.

• Looking at point cloud lidar data with dask dataframe
  Running into issues with datashader
  https://github.com/pyviz/datashader/issues/668

• Pete:

• Spinning down from UK Met AVD

Status check on larger issues

• AGU Debrief

• Workshop, about 6 talks, lots of Pangeo logos on posters from people we didn’t know
• Had a dinner with around 20-25 people

• Pangeo.pydata.org

• We get several requests for it a week, which is a maintenance problem
• It’s outdated
• We should find ways to push these people to other mechanisms, like domain-specific deployments or binder
• Maybe we should remove it at the end of the year?

• Jupyter community proposal

• Joe and Scott Henderson proposed three-day-hack-week on JupyterLab+Pangeo needs

• NFS mount as a cost-saving (?) alternative to persistent drives

• NFS costs around 10x as much per byte, but usage is often less than 10%
• Can share the same filesystem across all clusters.  1TB across all deployments.
• Should decide one way or the other

• Pangeo deployment on AWS update?

• University of Washington has AWS credits in hand ($50k/year)
• Derek was working to get hubploy system up and working on AWS, removing GCP specific logic.  Currently a work in progress.
• Joe: does NCAR need credits?
  Kevin: doing some testing on uploading datasets.  Mostly looking for infrastructure rather than credits.
  Jacob: we’re doing some experimentation that would benefit us.  We’re limited to the London region.

Short term coordination

• Joe should send Twitter credentials for Jacob
• Jacob should tweet asking AGU attendees for posters
• Review: https://github.com/NCAR/intake-cmip5 
• Review: https://github.com/pyviz/datashader/issues/668
• Joe to give Jacob credentials to AWS credits
• No meeting next week

2018-12-05

In attendance: Matt, Niall, Rich, Raphael,  Anderson, Kevin, Joe

Quick Reports

• Matt:

• Starting new job this week.  
• Cleaned up high level graphs work in https://github.com/dask/dask/pull/4092
• Proposing dask-on-HPC blog post here:
  https://github.com/dask/dask-blog/issues/5

• Niall:

• Jacob has been working with subject matter experts with data shader and elastic clusters to rasterize large point clouds
• Continuing to work on fargate
• Working on first version of Nature-communications style document to submit
  Focuses on high level lessons learned

• Rich:

• Tried making an ncview application using Panel.  Takes an intake catalog, you select a dataset, select a variable, you get a map, …  Having some difficulty with interacting with widgets, decided to wait a few weeks to wait for improved Panel documentation.  Ran into issue how to uniquely identify “latitude”, “longitude” and “time”.   Found solution in metpy ds.metpy.parse_cf()

 Should this be in xarray or a stand-alone package?

• Giving a talk tomorrow at USGS HPC Group in Denver (these people serve users)
• Giving the same talk to Woods Hole Oceanographic Institution (WHOI) all hands the next day (these people serve users too)

• Raphael:

• Working on ocean.pangeo.io .  Struggling with authentication, currently things are open.  
• Also working on selecting between different kinds of nodes

• Anderson:

• Working on Zarr dataset transfer to S3
• Planning to transfer 300TB
• Looking at different compression algorithms and optimal chunk sizes
• JupyterHub on Cheyenne: hoping for end-of-year, but no definite progress

• Kevin:

• PASC 19: https://pasc19.pasc-conference.org/
  Could propose a two-hour slot with four 30m speakers
  HPC, Cloud technology of interest to HPC, reverse, Data formats, Data cataloging and discovery

• Joe:  

Status check on larger issues

• AGU:

• Pangeo dinner Tuesday night

• Tear down pangeo.pydata.org

Short term coordination

Rich works with Joe to improve best practices for dask-jobqueue.   (e.g. if folks want to speed up their calculations, what considerations are there based on the type of job and type of nodes on their HPC?)   Joe says his HPC jobs are often I/O bound (e.g reading from NetCDF), so doesn’t use too many threads per worker.  But if reading from something like zarr that is thread-safe, might go with more threads per worker.   Also usually looks at how many cpus/node are available and submits jobs with one node per job.

2018-11-28

In attendance: Joe, Ryan, Kevin, Anderson, Raphael, Derek, Matt

Quick Reports

• Derek

• Setup with pangeo AWS account, testing kubernetes clusters, looking into hubploy/CI working for Jhubs
• HTTPS on binder.pangeo.io!
• Taints and tolerations. Working prototype on atmos.pangeo.io, Joe brought up using repo2docker appendix:         https://github.com/dsludwig/hubploy/blob/master/hubploy/imagebuilder.py#L54-L72 

• Ryan

• Consolidated metadata in xarray (https://github.com/pydata/xarray/pull/2559)
• AGU tutorial coming up
• Next release of pangeo - doc updates and outreach

• Raphael

• ocean.pangeo.io is up
• dev.ocean.pangeo.io - problems with building images

• Kevin

• Self education on jupyterhub deployments
• Interviewing SE candidates this week
• NCAR-Cheyenne jupyterhub is moving along slowly. Currently in IT hands
• Need to check on CESM-LE
• Allison Baker’s papers on lossy compression for climate data:

• https://www.researchgate.net/publication/257791669_Assessing_the_Effects_of_Data_Compression_in_Simulations_Using_Physically_Motivated_Metrics
• https://scholar.colorado.edu/cgi/viewcontent.cgi?article=1010&context=atoc_facpapers 
• https://www.geosci-model-dev.net/9/4381/2016/gmd-9-4381-2016-assets.html 

• Anderson

• (what Kevin said)

• Joe

• AWS award in place

• Matt

• Funemployed, not officially working on Pangeo at this point
• Moving to Nvidia, starting on Monday
• Dask 1.0 release

Status check on larger issues

• Pangeo AWS Credits
• Pangeo communication channels
• Jupyterhub scheduling:

• https://discourse.jupyter.org/t/planning-placeholders-with-jupyterhub-helm-chart-0-8-tested-on-mybinder-org/213

• NFS Home Dirs on Jhubs

• Can we share these between clusters?
• What happens when the same user uses multiple clusters?

Short term coordination

• Raphael goes over ongoing challenges with ocean.pangeo.io deployment in gitter
• Kevin follows up with Gokhan about LENS
• ZFP compression: https://github.com/zarr-developers/numcodecs/issues/117 - Allison Baker? Haiying Xu?

2018-11-21

In attendance: Matt, Rich, Raphael, Joe, Ryan

Quick Reports

• Matt: high level graph fusion is in a release.  Doing more work in branches (won’t be merged until 1.0 is released).  
• Ryan:

• Working including Zarr-consolidated-metadata work into XArray.  Helps to avoid lots of queries to GCS when loading a large file.  This is in Zarr now, we need to add hooks to XArray to connect things up.
• Also developing demo for ESGF (Earth System Grid Federation).  Poking on XArray and OpenDAP (thredds server).  This seems to work decently well.

• Joe:

• Derek has been working on HTTPS for binder.pangeo.io .  Working in staging now.
• Binder appendix: A way to add some extra configuration on top of user’s repo2docker configuration.  This lets us specify dask-specific details ourselves without having the user handle this.  Could replace our current cookiecutter.
• Also starting to attend JupyterHub meetings

• Rich:

• Gave 40m presentation at WHOI: went through the options of doing computation on remote data.  (slide #21 of talk)
• Writing conference proceedings paper “analysis and visualization of coastal data on the cloud”.  Combines pangeo scaling with pyviz interaction.

• Raphael

• Working on ocean.pangeo.io .  Most building blocks working, but something is still missing.  CircleCI/DNS/Auth on github all working.  

• Nic:

• Good feedback from people on polar.pangeo.io
• Have some issues around getting metadata
• Ryan: tried using this for my class, but there wasn’t sufficient metadata for students to know how to manage coordinate systems

Status check on larger issues

• Changed the schedule of this meeting.  Every other week will be 4 hours later than this time.
• Think about authorship policy

• Writing papers is important for many of the academic contributors here
• Making sure that people who should be listed might be important
• Rich’s pending conference proceedings paper is a good case study for this

• A new earthcube solicitation is coming out

• March 05, 2019: EarthCube Science-Enabling Data Capabilities [link]
• March 14, 2019: EarthCube RCN [link]
• Seems focused on reusing existing tooling
• Rich: Maybe propose something for handling unstructured grids?
• Ryan: I’m not personally interested in leading another proposal.  It would be great to see other groups expand things out.
• Joe: it would be great to see several pangeo-related proposals go in.  This might also serve the EarthCube program well.
• Joe: This is a good opportunity to strengthen our relationship with Jupyter
• Brainstorming

• Virtual sensors suite: Have a bunch of ADCPs and want to sample my model output regardless of how they’re stored.  

• Raphael: ESMF package has some functionality for this now.  Logstream
• Ryan: technically I think that this requires improvements to XArray’s indexing.

• Analytic tools for unstructured meshes Phil Elson at UK Met is interested in this, but it would be good to make this multi-institution
• Cloud-based datastores.  We have made a good argument for this in a few domains, but there are several pending issues and questions.

• http://pangeo.io/data.html#guide-to-preparing-cloud-optimized-data
• Probably also good opportunities for coordination with Jupyter

• Dask JupyterLab Server extension
  Ian has a good implementation up, but currently only tested with LocalCluster
  
  Someone should try this with binder.pangeo.io to test it with KubeCluster

• Relatedly, Ian Rose is giving an ESIP tech dive on making JupyterLab widgets
  http://wiki.esipfed.org/index.php/Interoperability_and_Technology/Tech_Dive_Webinar_Series#13_Dec_2018:_.22Developing_JupyterLab_Extensions.22:_Ian_Rose_.28Berkeley.29

Short term coordination

• Ryan evaluate atop fusion
• Think about authorship policy
• Schedule coordination meeting for response to NSF Earthcube Solicitation
• Nic: provide feedback / pull request to http://pangeo.io/data.html, link to these docs from deployment setup guide.  
• Raphael and Nic will check in on deployment issues

2018-11-14

In attendance: Anderson, Matt, Rich, Jacob, Raphael, Kevin Tyle, Joe

Quick Reports

• Matt: Not much.  Rolling through some Dask issues.
• Anderson:

• JupyterHub on Cheyenne: We now have JHub running on a simple VM ( behind a firewall for now). Using Yubikey for auth.  Just has a VM, no access to the file system or the job scheduler
• Converting some NetCDF datasets to Zarr.  Looking for some guidance here:
  https://github.com/pangeo-data/storage-benchmarks/issues/47 

• Compute nodes on Cheyenne don’t have internet access, so need to route things through login node.  Would be nice not to have to store the full dataset in an intermediate state
• Joe says to use Geyser or Caldera, which do have access

• Rich:

• dev.pangeo.io : Struggled a bit with packaging.  Hard to get all of holoviews/geoviews/datashader to work well with conda-forge packages
• Still curious about aws dev.pangeo.io framework

• Matt: Derek is pretty busy this week with other projects
• Joe: Amanda at UW is likely to do this (probably with some help from Derek).  But waiting until the NASA ACCESS award spins up.

• Looking for best practices on how to transfer NetCDF data to AWS Zarr

• Jacob:

• Looking at AWS Fargate (pretty fast container management service).  Much finer-grained costs, but 5-10x greater costs.  Running into problems with pulling the docker image every time, takes 5min.  
• Started working on a dask-fargate package
• Hoping that AWS sorts out the timing issue at some point
• Exploring adding GPUs to our stack, now accessible as an option with JHub profiles

• Raphael:

• Working on ocean.pangeo.io deployment.  Got a bit of help from Derek.  Am almost there but am running into an issue with gcrypt, but hopefully things should be ok after that.
• Planning to write up more novice-friendly documentation afterwards

• Joe:

• Nic Wayland has been working on polar.pangeo.io
  https://github.com/pangeo-data/example.pangeo.io-deploy/issues/3
  https://medium.com/@nicway/polar-deployment-of-pangeo-96865774287c 
• Ryan and Joe giving a tutorial at AGU.  
  Feel free to encourage people to attend
• Talking about next steps for data catalogs

• Interested in adding an OpenDAP endpoint to Intake server

• Looking forward to major version change in Binder.  This broke several things.  We’ll have to do a large update at some point.

• Ryan: out today, giving talk at conference

Status check on larger issues

• Lots of interest in developing JupyterHub work, but not much manpower

• Training peers
• Apply pressure onto for profit companies like Anaconda
• Element84 and UW might be good here, but they’ll need help to get started
• Lots of other groups (NCAR, Unidata, SDSC …) are also playing in this space
• NSF Jetstream XSEDE support (Andrea Zonca) might be another resource.  JupyterHub on Kubernetes now works on XSEDE Jetstream.

• Intake

• How does Intake handle large and complex datasets like CMIP?

• Jacob: it’s easy to write your own driver for these sorts of things
• Anderson: Experimenting with intake catalog for CMIP5 data on NCAR’s glade (https://github.com/NCAR/cmip5-intake-datasets)

• Can we just use Xarray?

• Well, there is a lot of metadata, opening all of the files to capture metadata is quite expensive.
• Can we improve Xarray then?
• This is similar to what we just did with Zarr in the “consolidate metadata” PR
• UK Met ended up doing something similar with Iris where we had an abstract dataset with missing files.
• One of a few xarray fast path opener issues: https://github.com/pydata/xarray/issues/1823

• Meeting timing

• Joe: there are now lots of people on the US West coast who would like to join this meeting, but won’t because it’s at 8am.  Obviously this conflicts with the 4pm UK / 5pm France time.  Maybe shift every other week?
• Jacob: Every other week seems reasonable.  The meeting notes are pretty extensive.
• Matt: Is a recording valuable?

• Jacob: prefer the meeting notes

Short term coordination

• Rich: planning to contact Andrea Zonca in Jetstream support.  Will task him with deploying Pangeo and see how it goes.
• We should start connecting people who develop *.pangeo.io clusters

• https://github.com/pangeo-data/pangeo/issues/476

2018-11-07

In attendance: Joe, Ryan, Matt, Guillaume

Quick Reports

• Matt: visit to ECMWF, UK Met.  Not much else.
• Ryan: New interns at Columbia intern program.  

• Charles will be setting up atmos.pangeo.io focusing on TRACMIP (sp?).
• Raphael is hitting road blocks when setting up ocean.pangeo.io
• Developed new pangeo binder for ocean model data:
  https://github.com/rabernat/pangeo_ecco_examples 
• Working on xbatcher to batch up xarray datasets and feed them to Keras/Tensorflow models

• Joe: Mostly busy at UK Met + ECMWF

• Put together a dask+xarray example on dask-examples
• Had good conversation at UK Met about Xarray / Iris relationship
• Good conversation with Rachael Prudden about downscaling

• Guillaume:

• Evangelized pangeo to french ocean data pole people (ODATIS).  Also evangelized Dask to HPC community in France (CNRS, Inria) and remote sensing community (Airbus DS and Thales, working on Deep earning things).
• Also worked on dask-jobqueue to separate Dask Scheduler from the notebook/cluster object (https://github.com/dask/distributed/issues/2235).
• Also working with IFREMER on SWOT aval use case.

• Rich:

• working with dev.pangeo.io, working great.  Adding library to handle grib files.  Pretty happy with the dev/staging/production environment.  Would love the same for the AWS deployment.  
• Also working on holoviews interactions
• Have removed all pinning, have questions about this

• Jacob:

• Back to work after paternity leave.  Lots of meetings
• Focused on water resilience event in our building this week
  Deployed a pangeo stack for a hack event with them
• Wrote intake-geopandas plugin to help deal with shapefiles
• Operational side of the Met office is uploading around 500k NetCDF files a day.  Looking at how to make this useful from the Pangeo perspective.  
• Issue for open ended rolling Zarr dataset:
  https://github.com/zarr-developers/zarr/issues/323 
  Trying to handle datasets where data might arise soon or be deleted
• Looking at dask-kubernetes and separating the scheduler from the notebook

• Derek: busy with other work this week.  Happy to see dev.pangeo.io working well.
• Raphael: ocean.pangeo.io update

• https://github.com/pangeo-data/pangeo/issues/448 
• Have created cluster on GCP
• Need to specify secrets for circle-ci, metadata, ...

Status check on larger issues

• Dask Labextension server extension:
  https://github.com/dask/dask-labextension/pull/31
• Https on *.pangeo.io: problem with websockets in Demos.

• Https + websockets seems not to work on some restrictive networks?
  This increases the value of https on binder.pangeo.io
• Jacob: Might be resolvable by defining an ingress in configuration
  Could also use LetsEncrypt.  Some tradeoffs exist.
• Derek: Our pseudo-cookie-cutter implementation uses LetsEcrypt.  Seems ok.
  Haven’t had time yet to copy the jupyterhub solution

• Hvplot performance on pangeo-example-notebooks

• Ryan: Rich changed many of our example notebooks to use hvplot rather than matplotlib defaults.  This has some performance challenges.  Matplotlib takes about 1s while hvplot takes tens of seconds.
• Rich: could switch backend to matplotlib.  Creating a lot of JS to make an interactive plot does take time.  
• Rich: if you point me to an example I’m happy to take a look
  Ryan: open up sea surface height example

• Update on tutorials

• Scaling up well, still having a hard time scaling down quickly
• Solution: use taints

• Dask-labextension task stream sometimes hangs.  Seems to require reloading frame
• Zooming out a bit … what else should we be focusing on?

• Rich: cloud agnosticism - would be nice to have dev/staging/production on AWS and Azure.  UK Met, Element84, UW groups all planning on AWS systems.
• Ryan: there is a view that this group has drifted into scalability/cloud, rather than focusing on goals of the group of the meeting a few years ago.
  Rich: packaging
  Matt: what is the scope of this group?  There are costs of going too broadly
  Joe: We had three bullet points early one, one was scaling, two were other-focused

• Pangeo AGU tutorial (Ryan and Joe)
• To pin or not to pin?

• Matt says pin everything (especially packages with lots of churn, e.g. jupyterlab, dask)
• On the other hand, that makes it hard to upgrade

Short term coordination

• For Derek: Raphael needs help on ocean.pangeo.io .  Also can we set up atmos.pangeo.io

• Derek and Raphael should check in one-on-one

• Rich and Ryan might check out sea surface height
• Version pinning
• Split out https://github.com/pangeo-data/pangeo-example-notebooks into topical repositories

2018-10-24

In attendance: Matt, Anderson, Ryan, Raphael, Derek, Rich

Quick Reports

• Matt: Delivered tutorial with binder.pangeo.io.  Upstreamed changes to cookie-cutter and reported scaling issues.  Did the same thing with another tutorial at PyCon DE and had a slightly smoother time.  Dask 1.0 release prep.
• Anderson: Looking at issue “Use Case related to PyReshaper” https://github.com/pangeo-data/pangeo/issues/15. https://github.com/NCAR/xarray-pyreshaper 
  JupyterHub at NCAR work pending person to come back from vacation
  Lens dataset: still waiting on legal team, nothing back from them yet, but we can probably do some testing in the meantime on AWS
• Ryan: Working with ESGF https://github.com/rabernat/pangeo_esgf_demo
  Karan (google) is working with ESGF to set up a THREADDS server within GCP
  Would be nice to have someone go to the ESGF meeting
• Raphael: https://github.com/pangeo-data/example.pangeo.io-deploy
• Derek: Set up dev.pangeo.io deploy (+ staging.pangeo.io, hub.pangeo.io) + investigating scaling + taint & toleration + node selectors
• Rich: Tried out cookie cutter https://github.com/pangeo-data/cookiecutter-pangeo-binder/issues/6

• Tried out pyviz publishing with bokeh server application
• Got Unidata to engage more in Zarr discussion
• Interested in less fragile conda environments that use fewer channels
• Looked at publishing data on AWS, but was turned off.  It was a bit intense.  Looking to see if other groups are willing to take this on.

Status check on larger issues

• Publish data on AWS

• Originally this seemed really exciting, but it turns out that maintaining large datasets on the cloud is a bureaucratic process that we probably don’t want to be involved in.  Instead, let’s try to convince the original data owners to do this.  Lets build good demos to convince them to do that.
• Having a paper on the benefits here might also be helpful

• Binder.pangeo.io experiences with recent Dask tutorials
• ESGF OpenDAP discussion

• Wanted to convert a bunch of data from OpenDAP to Zarr.  Ran into problems historically, but now things seem to work with dev version of XArray.
• Ran into performance limitations with OpenDAP after around ten so workers
• PyDAP + scalable service

• PyDAP was dead for a while but we now have commit rights
• Having a student handle PyDAP <-> XArray connections would be a good project: https://github.com/pydap/pydap/issues/74 
• We would still need to build a scalable routing service to handle this in the cloud.  It’s not clear how expensive this would be.

• Who can go to the workshop before the AGU?
  Ryan will try to go down for the day if no one else can go.  If anyone else can do this and understands the full stack of issues we’d be happy to have them go instead.  Rich could be a backup.

• Extra pangeo deployments

• Astro.pangeo.io
• Yesterday set up repository for dev.pangeo.io, staging.pangeo.io, hub.pangeo.io
• Can we retire pangeo.pydata.org ?

• We first need to have some stable places to redirect people
  Ocean.pangeo.io is particularly important for Ryan and his colleagues
• We would like to delete all of the data, but probably need to alert current users

• Neuro.pangeo.io : Ariel Rokem had some feedback on the setup process.  Derek integrated and responded this feedback.  Not sure what current status is.
• Ryan: important feature is resource selection: https://github.com/pangeo-data/pangeo/issues/87 

• Want to be able to select resources for notebook, including low/medium/high memory nodes, or GPU nodes.
• Might be possible with profile list in recent JupyterHub

• Publishing panel visualization:
  https://github.com/minrk/binder-bokeh-server
  

Short term coordination

• Raphael and Derek communicate about where to find state-of-the-art instructions for creating a new pangeo cluster (raphael opens github issue)
• Rich gives dev.pangeo.io a try
• Ryan gives Derek access to billing account
• Rich tries to create a panel binder on pangeo binderhub

2018-10-17

In attendance: Ryan, Raphael, Joe, Kevin, Matt, Nick Wayland, Anderson, Rich Signell

Quick Reports

• Matt: Array expression fusion is in master, probably won’t be a release for a little while.  Would appreciate folks updating to dask:master and checking things out.
• Derek: generating new deployments with CI (astro.pangeo.io)
• Joe: AWS proposal for credits is approved. $49,000 / year to UW eScience institute.
• Ryan:a New NSF funded project that involves Pangeo (Poseidon) at Johns Hopkins on their own infrastructure.
• Rich: Met with Amanda Tan (UW) and transferred knowledge (gained mostly from Jacob Tomlinson) on getting Pangeo deployed on AWS.  Switched to spot instances on the esipfed pangeo cluster and have had a good experience so far.  Good experiences demoing Panel for pyviz  interaction.
• Anderson: Yesterday met with security team (defer to Kevin).  Looking at NetCDF->Zarr conversion and built up some good experience.  Looking at container technology for HPC.
• Kevin: Can’t run docker itself on HPC, (considering uDocker, shifter, singularity).  Considering just letting people use their own stack in user space.  A little bit blocked; hard to get admin rights for anyone outside of their group.  
• Nic: Transferred over polar.pangeo.io from personal account to pangeo/gce account.  Placing some data into intake catalog.

Status check on larger issues

• Questions from Nic on polar.pangeo.io

• Datasets are updated daily, is this challenging from a cost or technological basis?

• Ryan: append workloads are tricky with XArray + Zarr today.  Zarr supports append, but XArray doesn’t support that for Zarr.  We could also open up many zarr datasets, like with an `xarray.open_mfzarr`.  
• Matt: it sounds like these might be good tasks for new devs.
• Joe: https://github.com/pydata/xarray/issues/2022 , https://github.com/pydata/xarray/issues/1672 

• Taking a while to load in a zarr dataset on the cloud.  Lazy load is fast, but actually grabbing data will be slow.

• Matt: that sounds like an issue, lets move this to a github issue and track it down

• Split Pangeo into different domains

• Astro.pangeo.io (Simon Krughoff) - no one super excited yet
• Polar.pangeo.io (Nic Wayland) - two closed datasets, Lamont polar geophysics group (Nick Frearson) interested to
• Ocean.pangeo.io (Raphael) - what credentials are needed? High level for service account. Copy Nic Wayland’s. New github issue needed to track
• Hydroshare.pangeo.io - being used for a short course on hydrologic modeling, ping Fernando Perez on sharing / exporting from jupyterlab environment
• Neuro.pangeo.io - not set up yet, managed by UW people
• “aws.pangeo.io” (UW AWS Pangeo, Amanda Tan) - Blog post about setup process

• AWS engagement

• USGS needs to sign waivers
• Need guidance on producing zarr, zarr needs feedback from netCDF devs, Rich will encourage engagement between Unidata and zarr

• Zarr dataset production

• Want to produce a blog post about how to do a large-scale netCDF→ zarr conversion

• Talk about Panel / Binder / Cookie cutter

• IPywidgets vs. pyviz, Rich is going towards pyviz because of Army engineering development lab, Rich will report back on how to “publish” panels within pangeo cluster
• Cookie Cutter: https://github.com/pangeo-data/cookiecutter-pangeo-binder 

• Governance

• 5-6 people, reasonable size
• Nominations via github, seconding required, from members of community
• Advertise via twitter, mailing list, github

Short term coordination

• Rich will try Cookie Cutter for annotation example
• Kevin pings NCAR lawyers on NCAR
• Exchange info between Joe, Nic, and Raphael
• Blog post on AWS setup by Amanda
• New github issues / discussion related to

• Xarray appending
• Zarr performance on GCS (Nic)

• Rich encourages Unidata to interact with Zarr

2018-10-10

In attendance: Ryan Abernathey, Matt Rocklin, Derek Ludwig, Anderson B

Quick Reports

• Matt: atop fusion is in, should see lower overhead on dask array operations
• Ryan: teaching, met with NCAR about AWS hosting of large ensemble data.
• Derek: looked into HTTPS a bit, but don’t have access to the deployment
  Planning to launch some new deployments today or tomorrow
  Getting good feedback from Nic Wayland on the README
• Anderson: Still getting started, investigating deployment of JupyterHub on Cheyenne.  Blocked on meetings with internal security and networking teams.  Investigating repo2docker.

Status check on larger issues

• Split Pangeo into different domains:

• Polar.pangeo.io is up from Nic Wayland.  Currently under his own account but we’re moving it under our own.  Has generated some useful feedback on the process.
• Planning to generate an astro.pangeo.io afterwards.
• Q (ryan): Can we have all of these deployments in one kubernetes cluster?
  A (derek): Yup.
• Q (ryan): Raphael plans to jump into ocean.pangeo.io .  How hard is it to select different machine sizes for the notebook server?
  A (matt): maybe we just use larger notebook sizes for ocean.pangeo.io ?
• Ocean, astro, sandbox, neuro
• People will want to customize their JupyterHub landing page.  Can we have an index.html in the repository that gets put into the image (pre-populated with our css).
• Eventually we’ll also want some of these on AWS

• AWS engagement:

• Large ensemble:

• NCAR Legal needs to be ok with this process.  
• Also need someone internal to NCAR to format the data as Zarr.  Maybe Anderson ends up doing this?

• Note that Karan Bhatia (Google) is working to stand up a pangeo deployment within ESGF serving data over OpenDAP.  https://github.com/pangeo-data/pangeo/issues/420 

Short term coordination

• Derek needs deployment secrets from Joe for binder in order to do HTTPS

2018-10-03

In attendance: Raphael, Anderson, Guillaume, Rich, Derek, Joe, Matt

Quick Reports

• Matt: Array expression fusion in a usable state.  This reduces overhead for arrays with many chunks and many operations.  Its implementation is a little hacky today, but gives a good view for future large scale work in dask.
• Joe: Collecting tasks to do in pangeo binder.  

• Need security stuff (binder should soon get github authentication)
• Https
• Autoscaling (faster scaledown)
• Not a ton of time to work on these at the moment

• Derek: Put up documentation for CI template to deploy domain-specific pangeo cloud deployments.  Let Ariel know, but should probably disseminate more widely.
• Rich: Machine learning workshop on pangeo finished up.  Have a blogpost draft written up.  (had some frustration with Medium).  Looking for ways to accelerate tensorflow development.  (does dask have something here?)
• Guillaume: Built my own pangeo deployment on Google Cloud.  Also ran into HTTPS.  
• Anderson: Investigating JupyterHub on Cheyenne
• Raphael: Looking forward to being a guinea pig for a pangeo deployment.

Status check on larger issues

• Pangeo.pydata.org is no longer open access

• Monday morning we got an e-mail from Google stating that they detected some potentially nefarious activity coming from pangeo.pydata.org.
• First time this had happened
• Closed open access, now need to be part of a known github organization to get access.
• Also raised questions about how we manage organizations in the future.
• Resolved by changing the helm configuration.  JupyterHub provides github organizations as an option in their whitelist.  Orgs listed at github.com/pangeo-data/pangeo/ :: gce/secret-config.yaml file.  If there are other organizations that we should add, please speak up.

• Split pangeo.pydata.org for domain specialties

• Documentation is in place for new example deployment
• Ready for people to start playing with.  Will likely be some challenges, but Derek is pretty responsive.
• Remote-sensing.pangeo.io, is Scott Henderson or NASA interested ? Guillaume EB interested in being some guinea pig here

• Moving to AWS, where most of the data lives right now
• Waiting for Amazon to give us credits, looking to make this open to the public, credits should arrive later this month
• Could deploy on google ahead of time
• Rich @ USGS also deployed on AWS in the past and ran into problems that were eventually handled by Jacob @ UK Met.  Should we include these in docs?

• Guinea pigs

• Raphael for ocean.pangeo.io
• Guillaume
• Nic Wayand (seaice.pangeo.io) (already up and running, not connected to domain name yet)
• Bart Nijssen (hydroshare.pangeo.io) (already up and running)

• AWS engagement:

• Research credits: Joe is leading the charge for this.  Application is finished.  Administered by University of Washington, who has volunteered to run one deployment for the next couple years
• Datasets: Waiting on people to sign license agreements to publish key datasets.

• Rich: what is the next step for me?
  Joe: Fill out application for public dataset program, then they ask you to sign something.

Short term coordination

• Joe should send an e-mail from Joe Flasher to Rich Signell about AWS dataset licensing
• Derek should move docs to pangeo org and then publish
• Guillaume needs input on big data from space paper.  Joe will send some materials.
• HTTPs on pangeo.pydata.org: Derek says that he can probably do it
• Joe should dump information on some of his issues:

• Need security stuff (binder should soon get github authentication)
• Https
• Autoscaling (faster scaledown)

• Matt would like an example notebook that stresses task overhead for dask.array from xarray.  Rich might be able to help.

2018-09-26

In attendance: Ryan Abernathey, Joe Hamman, Matt Rocklin, Rich Signell, Derek Ludwig, Niall Robinson, Jacob Tomlinson

Quick reports

• Matt: Atop fusion. Reduce overhead in dask graphs.  
• Ryan: Converted tabular workflow to use dask.dataframe and datashader.  Busy with non-pangeo work.
• Rich: Also looking at tabular data.  Using Parquet and dask dataframe.   360m record CSV file on birds from ebird, write to parquet, read with dask dataframe using map_partitions.    Need a package from R to smooth centroids.  Can you call R in parallel from Dask?  (From matt: check out rpy2, but watch out, it doesn’t release the GIL.  You’ll want one-thread-per-process and short tasks.)
  Also Jacob helped us get auto-scaling working on our AWS cluster.  Currently being used internally for 40 person, 3-day class on deep neural networks. Working smoothly.  Thanks Jacob!
• Niall:

• Trying to get weather forecast data up and public from UK Met office.  
• Dashboarding: Fernando Perez is putting us in contact with dashboard people.  Sylvain Corlay.
  From Ryan: There appear to be two separate stacks.  
  1: Jupyter Widgets/bqplot/leaflet and 2: Bokeh/Holoviews/Param
• Dot-Mesh: startup based in UK focused on devops and robust production systems.  Started new startup doc-science focused on provenance etc..
  From Ryan: https://github.com/recipy/recipy 
  (http://blog.rtwilson.com/introducing-recipy-effortless-provenance-tracking-with-python/ )

• Jacob: About to drop off for a month, focusing on knowledge transfer to Theo.

• Continuous deployment on dev pangeo
• Data availability with intake.  Intake-iris plugin.  Dynamo plugin.  Looking at production data flows within UK Met.
  Ryan: can I use your intake catalog?  Jacob: Sure, but you need our S3 Fuse mounts currently.  Once we move to something like Zarr then this will be easier.

• Derek:

• Intake catalog available in our documentation, include code snippet for how to access that data
• CI/CD: maybe leave this for longer discussion in next session
• Iterating over pangeo data bucket and discovering what datasets we already have (pending)
  From Niall: I wonder if we can sweep up some existing data in AWS into an intake catalog
  Ryan and Rich: There are already some existing metadata standards.  We can create ISO metadata from OPeNDAP endpoints based on the attribute and variable data in the NetCDF-Data-Model datasets. We submitted an ESIP/NUMFOCUS proposal  to generate metadata form xarray datasets.  These then can allow discovery via Google dataset search and CSW search via data.gov, data.gov.uk, etc.

• Joe: Back from a week from vacation.  Pangeo binder is up and running.  Interested in feedback.  NASA Pangeo award should be kicking in any day now.  Helped someone new to stand up a hydro-share pangeo to learn a bit about novice users.
• Guillaume:

• Working on deploying a Pangeo infrastructure on GCE, with or without @dsludwig terraform script. All steps well understood and OK.
  I will probably need some help for correctly starting KubeCluster soon (it does not seem to work yet), but I will reach to some of you when I've dug a little more.

Status Check on larger issues

• Split pangeo.pydata.org to many smaller domains

• UK met office may want to do this internally
• Derek gives demo of current pos
• Connect Raphael Dussin with Derek to try deploying
• Need to define and communicate expectations to other groups

• Position Papers:

• data discovery/cataloging;
• Dashboarding
• AWS datasets

• Met with Amazon
• Converged on a set of datasets
• Now need to get people who manage these datasets involved.  
  For example NCAR CESM leadership needs to sign a license for redistribution
  CMIP will be very complicated.  Has a very restrictive license.
  Either work with them directly (google is doing this) or work with individual groups.
• License might be ok eventually for NCAR, but now need staff to move the data

Short term coordination

• Scott Henderson GeoTIFF example
• CI/CD

• guinea pigs (Joe and Raphael are both busy this week)
• Need basic docs

• Rich should look at https://github.com/dask/dask-examples/pull/25 and dd.map_partitions

Meeting Template:

YYYY-MM-DD 1600 GMT (9am PDT, 12pm EDT)
YYYY-MM-DD 2000 GMT (1pm PDT, 4pm EDT)

Video Conference Link: https://whereby.com/pangeo
Video Conference Link:  Zoom

Join Zoom Meeting:
https://columbiauniversity.zoom.us/j/535776170

One tap mobile:
+16468769923,,535776170# US (New York)
+16699006833,,535776170# US (San Jose)

Dial by your location:
+1 646 876 9923 US (New York)
+1 669 900 6833 US (San Jose)

Meeting ID: 535 776 170

Find your local number: https://zoom.us/u/abcuAxbTMF

Hello and welcome to the team meeting!
Please sign in below so we know who was here:

Name / Institution / @GitHub handle

• Ryan Abernathey / LDEO / @rabernat

60 Second Updates:

Agenda items:

Short term coordination:

• .