Pangeo Community Weekly Meeting
Join the meeting: NumFocus Zoom Link
Archived Pangeo Weekly check-in notes from 2018-2024!
Please provide feedback on the spring showcases!
2025-04-16 (9am PDT; 12pm EDT, 1600 UTC)
Attendees: Name / Institution / @GitHub:
- Max Jones / Development Seed / @maxrjones
- Rich Signell / Open Science Computing / @rsignell
- Joe Hamman / Earthmover / @jhamman
- Tom Augspurger / NVIDIA / @TomAugspurger
- Shane Loeffler / CarbonPlan / @shane98c
- Raphael Hagen / CarbonPlan / @norlandrhagen
- Kata Martin / CarbonPlan / @katamartin
- Valentin Genin / Thales Services Numeriques / @valentin-genin
- Katelyn FitzGerald / NSF NCAR / @kafitzgerald
- Enrico Milanese / Woods Hole Oceanographic Institution / @enrico-mi
- Justus Magin / LOPS / @keewis
- Sean Harkins / Development Seed / @sharkinsspatial
- Devin Rand / Berkeley Earth / @devinrand
- Colette Kelly/Woods Hole Oceanographic Institution / @ckelly314
- Joanna Millstein / Colorado School of Mines + CryoCloud / @jdmillstein
- Wietze Suijker / Space Intelligence / @wietzesuijker
- Martha Morrissey / Pachama / @martham93
- Martin Hungendorfer / Josephinum Research / @mhungen
- Damian Smyth / Marine Institute /@DSmythMI
- Ellen Park/Woods Hole Oceanographic Institution/@ellenrpark
- Kyle Barron / Development Seed / @kylebarron
- Tom Nicholas / Earthmover / @TomNicholas
- Tina Odaka / Ifremer-umr-lops/ @tinaok
- Deepak Cherian / Earthmover / @dcherian
- Matt Brown / UKCEH / @mattjbr123
Showcase Talk: Lightning talks!!!
- “A n00b tries to scale geospatial zonal statistics” by Deepak Cherian [see links to slides and more below]
- “Use-cases for parquet datasets in oceanography” by Enrico Milanese [see links to slides below]
- “Visualizing multidimensional climate data on the web - the OAE Efficiency Atlas” by Shane Loeffler slides
- “Organizing model predictions stored in Zarr with STAC…a journey + example” by Martha Morrissey slides
- “Speeding up Zarr data access with obstore” by Kyle Barron [link to slides below]
- “Rechunking using SLURM-like job arrays with Coiled” by Rich Signell slides
Discussion
60 Second Updates:
- Max J:
Please provide feedback on the spring showcases!
- Request for example datasets to drive GeoZarr finalization
- Deepak:
- Link dump for my talk:
- Slides: Learning about zonal stats
- https://xvec.readthedocs.io/en/stable/
- https://flox.readthedocs.io/en/latest/
- https://xarray.dev/blog/flox
- https://loreabad6.github.io/post/articles/post01_post.html
- https://isciences.github.io/exactextract/index.html
- https://flox.readthedocs.io/en/latest/user-stories/large-zonal-stats.html
- https://github.com/xarray-contrib/flox/issues/428
- Tina
- second session of the ESA EOPF Sentinel Zarr Samples webinar series on 24 April 2025 at 2:00 PM (CEST) by Anne! Register here : https://us02web.zoom.us/webinar/register/WN_uIqoYtx9Q4-hPI9dJU1Xlw#/registration
- Kyle
- Slides for my talk:
- Obstore: https://developmentseed.org/obstore/latest/
- Gist example: https://gist.github.com/kylebarron/b07560566a67fe7608613b63d5bca8ae
- Zarr-Obstore PR
- Joe
- Earthmover released Flux today
- Zarr3 is about to get extension data types
- If you care, take zarr-python#2874 for a spin
- Justus
- Stac-recipes (running xstac using pangeo-forge): https://github.com/keewis/stac-recipes
- More research into distributed indexing and interpolation algorithms
- Xarray api for the generalized regridding libraries:
- Looking into using zarr with signed urls (copernicus-marine, but works the same with planetary computer) – might be able to use obstore for this?
- Enrico
Agenda Items:
- TBD
2025-04-09 (9am PDT; 12pm EDT, 1600 UTC)
Attendees: Name / Institution / @GitHub:
- Aimee Barciauskas / Development Seed / @abarciauskas-bgse
- Rich Signell / Open Science Computing / @rsignell
- Tom Nicholas / Earthmover / @TomNicholas
- Sarah Zwiep / Floodbase/ @szwiep
- Chuck Daniels / Development Seed / @chuckwondo
- Justus Magin / CNRS / @keewis
- Tom Augspurger / NVIDIA /@TomAugspurger
- Max Grover / Argonne National Laboratory / @mgrover1
- Jiwoo Lee / Lawrence Livermore National Laboratory / @lee1043
- Hendrik Makait / Coiled / @hendrikmakait
- Enrico Milanese / Woods Hole Oceanographic Institution / @enrico-mi
- Jesse Anderson / Numbers and Strings / @jessjaco
- Matt Brown / UKCEH / @mattjbr123
- Aaron Kaplan / IRI, Columbia U. / @aaron-kaplan
- Martha Morrissey / Pachama / @martham93
- Tom Vo / Lawrence Livermore National Laboratory / @tomvothecoder
- Devin Rand / Berkeley Earth / @devinrand
- Chang Liu / Axiom - Tetra Tech / @cliu3
- Katelyn FitzGerald / NSF NCAR / @kafitzgerald
- Harsha Hampapura / NSF NCAR/ @hrhampapura
Showcase Talk (15-25 min): Dask Array: Scaling Up for Terabyte-Level Performance - Hendrik Makait
- Space to time rechunking use case: Automatic selection of P2P rechunking method released in dask 2024.8.2
- Downscaling use case
- Discussion
- Comment: a lot of these improvements are part of general trend, cubed as well, solving in similar ways, fundamental insight graph rechunking has to be special-cased, and then implemented efficiently. And prevent the explosion of tasks.
- Hendrik: General trend toward higher level APIs that can be optimized
- In dask we have array-expr: gives a higher level abstraction over the actual operation we need to do.
- Ticket on the conversion btw the 2 libraries: Cubed vs dask.array: Convergent evolution? · Issue #570 · cubed-dev/cubed · GitHub
- Bounded memory part of the rechunking - dask does it in constant memory vs bounded memory - how do you know how much memory you need to provision?
- General recommendations 8-16 GB is probably a good value, keeping chunk sizes sane (250MB tops)
- What’s more likely is you are using way too large a dataset
- Really we’re just reinventing query planners (database method)
- Meta observation: pangeo community asking the community for challenges and trying to solve them, that is very cool.
- “Making dask great again?” - Rich Signell
- Xarray dependencies?
60 Second Updates:
- Rich S:
- Bring your own image has been added to the European Pangeo DaskHub on EOSC (running on EGI). (I found out about bring your own image by participating in a NOAA NMFS HackHours meetup on a 2i2c cluster). We will be using this Hub in the GlobalCoast EU Project, which will implement Pangeo-style infrastructure and practices on the EGI Cloud. MinIO is used for s3-compatible object storage.
- Tom
- Working on benchmarking number of GET requests when opening data in S3 from various formats
- Justus
- More progress on grid-indexing / grid-weights:
- https://discourse.pangeo.io/t/distributed-computation-of-regridding-weights/5031
- Xarray api draft (https://github.com/keewis/grid-weights/issues/7)
- Looking into distributed nearest neighbours interpolation based on cells
- Max G.
- Working on improvements to NEXRAD reader in xradar - https://github.com/openradar/xradar
- Py-ART 2.0 is out - https://github.com/ARM-DOE/pyart
- Held a summer school in Tasmania a bit ago - materials are available
- Reviewing SciPy abstracts this week for the earth/geo/ocean/atmos/climate track 😀(many familiar faces… expect notifications soon)
- SciPy info! - https://www.scipy2025.scipy.org/
- Playing around with myst… great for quicklooks especially with github workflows!
- https://crocus-urban.github.io/geospatial-analysis/
- And short courses :) https://www2.meteo.uni-bonn.de/data/radar/prepep2025/
Agenda Items:
- TBD
2025-04-02 (9am PDT; 12pm EDT, 1600 UTC)
Attendees: Name / Institution / @GitHub:
- Justus Magin / CNRS / @keewis
- Martin Durant / Anaconda / @martindurant
- Raphael Hagen / CarbonPlan / @norlandrhagen
- Tom Nicholas / Earthmover / @TomNicholas
60 Second Updates:
- Tom
- Working on refactoring VirtualiZarr
- Use ManifestStore abstraction
- Justus
- HuggingFace spaces with STAC APIs have been flagged / disabled, new STAC APIs will be at https://gfts-stac.duckdns.org/ (for now), for example https://gfts-stac.duckdns.org/copernicus/api/v1
- Regridding post on the discourse summarizing the current status of my work on regridding / distributed indexing
- https://discourse.pangeo.io/t/distributed-computation-of-regridding-weights/5031
- Especially, I’m trying to find a good, flexible xarray API (see https://github.com/keewis/grid-weights/issues/7
- Other missing piece is writing sparse arrays to disk (especially zarr). Supported by `binsparse-python` (https://github.com/ivirshup/binsparse-python), but not as convenient as `xr.open_dataset`
- Will look into adding to https://developmentseed.org/warp-resample-profiling/
2025-03-26 (1pm PDT; 4pm EDT; 2000 UTC)
Attendees: Name / Institution / @GitHub:
- Max Jones / Development Seed / @maxrjones
- Justus Magin / CNRS / @keewis
- Martin Durant / Anaconda / @martindurant
60 Second Updates:
- Max J
- Working on VirtualiZarr TIFF support
- Justus
- Merged PR to get Pangeo-Forge to work with Zarr V3 (with Raphael)
- Deciding whether to support both or just V3 moving forward
- Martin
- Demo reading from S3 from pyscript via pyscript fs proxy server
2025-03-19 (9am PDT; 12pm EDT, 1600 UTC)
Attendees: Name / Institution / @GitHub:
- Max Jones / Development Seed / @maxrjones
- Rich Signell / Open Science Computing / @rsignell
- Katelyn FitzGerald / NSF NCAR / @kafitzgerald
- John Clyne / NSF NCAR / @clyne
- Justus Magin / CNRS / @keewis
- Arjun Verma / QuantStack / @arjxn-py
- Diane Fritz / NSIDC
- Matt Savoie / NSIDC / @flamingbear
- Martin Durant / Anaconda / @martindurant
- Raphael Hagen / CarbonPlan / @norlandrhagen
- Martin Renou / QuantStack / @martinRenou
- Matt Fisher / Schmidt Center for Data Science and Environment @ UC Berkeley / @mfisher87
- Andy Barrett / NSIDC / @andypbarrett
- Tom Nicholas / Earthmover / @TomNicholas
- Matt Brown / UKCEH / @mattjbr123
- Chang Liu / Axiom - Tetra Tech / @cliu3
- Lindsey Nield / Earthmover / @lindseynield
Showcase Talk (15-25 min): “Real-time collaboration and collaborative editing for GIS workflows with Jupyter and QGIS” - Martin Renou and Matt Fisher
- Discussion
- Connection with GeoBlacklight ?
- Built on OpenLayers
- Developing StoryMap-like capabilities is already on the roadmap
60 Second Updates:
- Max J
- April 2 is the next Pangeo showcase: on Dask performance updates
- April 9 lightning talks - sign up here
- John C.
- WCRP Digital Earths - Global Hackathon (NSF NCAR “node”), May 12 - 16
- Travel support applications due Friday, March 21
- Register by April 23, 2025 (other U.S. nodes may have different registration deadlines)
- Hackathon goals:
- Accelerate the analysis of Storm Resolving Models (global and regional)
- Socialize best practices in coding and data intensive applications (infrastructure and workflows)
- Technologies: Pangeo “stack”, HEALPix, Zarr, UXarray
- New Project Pythia Cookbook: Earth Observation Data Science Cookbook!
- Thanks to the Department of Geodesy and Geoinformation at the TU Wien (Austria)
- Rich Signell
- Participating in the GlobalCoast EU Project, which will implement Pangeo-style infrastructure and practices on the EGI Cloud. EGI Cloud is a cloud computing platform designed for the European research community. It's a federation of clouds which combines resources from various providers into a single, unified system. This allows researchers to access a wide range of computing power and storage options from a single point of entry. They are interested in expanding to a global federation. Would NCAR be interested? (Matt mentions that this sounds very much like the US National Research Platform)
- Tom N
- Working on making xarray.DataTree work with zarr-python v3
- Also (with Max J) refactoring VirtualiZarr to have a much more solid foundation, which should remove a bunch of weird data model inconsistencies and footguns
Agenda Items:
- TBD
2025-03-12 (1pm PDT; 4pm EDT; 2000 UTC)
Attendees: Name / Institution / @GitHub:
- Max Jones / Development Seed / @maxrjones
- Rich Signell / Open Science Computing / @rsignell
- Martin Durant / Anaconda / @martindurant
- Len Stran / @ljstrnadiii
- Tyler Erickson /@tylere
- Justus Magin / CNRS / @keewis
- Eni Awowale / NASA GES_DISC/ @eni-awowale
- Henry Rodman / Development Seed / @hrodmn
- Deepak Cherian / Earthmover
- Amelia Snyder / USGS / @amsnyder
- Tom Nicholas / Earthmover / @TomNicholas
- Bernhard Stadlbauer / Earthscale / @bstadlbauer
- Sarah Zwiep / Floodbase / @szwiep
- Manish Venumuddula / NSF NCAR / @manishvenu
- Matteo Simone / Planetek Italia
- Martha Morrissey / Pachama / @martham93
- Lindsey Nield / Earthmover / @lindseynield
- Frank Warmerdam / Planet / @warmerdam
- Sean Harkins / Development Seed / @sharkinsspatial
- Tina Odaka / UMR LOPS, IFREMER/ @tinaok
Showcase Talk (15-25 min): “Building Scalable Mosaic Workflows with Flyte/UnionAI, GDAL, and Xarray” - Len Strnad
- Discussion
60 Second Updates:
- Max J:
- Next week’s showcase will be on Jupyter-GIS and GeoJupyter!
- Put together some geozarr examples, looking for feedback - https://developmentseed.org/geozarr-examples/
- Working on a async-tiff based VirtualiZarr backend
- Tyler E
- Working on tutorial infrastructure based on Quarto (static site generation), pixi (package management), GitHub (CI, dev containers). Goal is to have each tutorial be runnable with its own code environment, and to use it for improving the CNG tutorials. https://github.com/tylere/test-quarto-pixi-ci
- Simplifying Xee’s (Earth Engine backend for Xarray) model of pixel grids. https://github.com/google/Xee/pull/225
- Martin
- Sketching out two related file access patterns:
- A panel in jupyter-lab that allows access to fsspec storage anywhere (jupyter-fsspec)
- Pyscript integration for fsspec/http to enable pydata in the browser ( Example )
- These two have a possible unification in a server component that it mostly a bytes proxy, but can also be a credentials stash
- Reading about spatial partitioning schemes, e.g., h3 and hilberts
Agenda Items:
- TBD
2025-03-05 (9am PST; 12pm EST, 1700 UTC)
Attendees: Name / Institution / @GitHub:
- Max Jones / Development Seed / @maxrjones
- Rich Signell / Open Science Computing / @rsignell
- Joe Hamman / Earthmover / @jhamman
- John Clyne / NCAR / @clyne
- Tom Augspurger / NVIDIA / @TomAugspurger
- Justus Magin / CNRS / @keewis
- Pier Lorenzo Marasco / SEIDOR / @pl-marasco
- Michael Sumner / Australian Antarctic Division / @mdsumner
- Nick Hodgskin / Utrecht University / @VeckoTheGecko
- Katelyn FitzGerald / NSF NCAR / @kafitzgerald
- Sarah Zwiep / Floodbase / @szwiep
- Martin Durant / Anaconda / @martindurant
- Anthony Aufdenkakmpe / LimnoTech / @aufdenkampe
- Raphael Hagen / CarbonPlan / @norlandrhagen
- Sean Harkins / Development Seed / @sharkinsspatial
- Riley Conroy / NCAR / @rpconroy
- Shane St Savage / Axiom Data Science / @srstsavage
- Ahmad Jiha / Axiom Data Science
- Chang Liu / Axiom - Tetra Tech / @cliu3
- Aimee Barciauskas / Development Seed / @abarciauskas-bgse
- Aaron Kaplan/IRI, Columbia University / @aaron-kaplan
- Davis Bennett / Independent / @d-v-b
- Harsha Hampapura / NCAR/ @hrhampapura
- Apurba Biswas / Statkraft / @apurba-biswas
- Yoni Nachmany / Zeus AI / @yoninachmany
- Pierre LOICQ / Murmuration sas / @pierreloicq
- Doug Schuster / NCAR / @dcschus
- Nicholas Delli Carpini / RPS Ocean Science / @ndellicarpini
- Kristen Thyng / Axiom Data Science / @kthyng
- Tom Nicholas / @TomNicholas
- Paige Martin / Australian Climate Simulator / @paigem
- Tina Odaka / UMR-LOPS Ifremer / @tinaok
- Jonathan Joyce / RPS Tetra Tech / @jonmjoyce
Showcase Talk (15-25 min): "Zarr Python 3 and beyond" - Joe Hamman
- Discussion
60 Second Updates:
- Max J:
- Showcase talk next week on “Building Scalable Mosaics with GDAL, Xarray, and Flyte” by Len Strand
- Last Pangeo showcase for the spring on April 9 will be lightning talks - I’ll share an announcement on the Discourse on Friday
- Registration open for CNG in April - https://2025-ut.cloudnativegeo.org/ and SatCamp in September - https://satcamp.xyz/
- Tom A:
- NCAR hackathon last week working on zarr / xarray / GPU things with some other pangeans. Blogpost forthcoming.
- Presenting at the Cloud Native Geospatial conference. Let me know if you’re using GPUs for geospatial workloads.
- Martin D:
- Akimbo gains polars-lazy backend. Polars-GPU not in scope right now.
- Davis B:
- Adding dtype extensibility to zarr v3
- John C.
- WCRP Digital Earths - Global Hackathon, May 12 - 16
- Global, distributed event (3 U.S. sites)
- Goals: Get experience with workflows for analysis of km-scale global climate model outputs
- HEALPix hierarchical, petascale data stored in Zarr
- Register for NCAR node by April 23, 2025 (other U.S. nodes may have different registration deadlines)
- Tina Odaka
- EOPF Zarr sample service started
- Justus
- Got distributed tree to work with Dask to search on polygons, for regridding based on polygons
Agenda Items:
- TBD
2025-02-26 (1pm PST; 4pm EST; 2100 UTC)
Attendees: Name / Institution / @GitHub:
- Justus Magin / CNRS / @keewis
- Martin Durant / Anaconda / @martindurant
- Raphael Hagen/ CarbonPlan / @norlandrhagen
- Max Jones / Development Seed / @maxrjones
- Matt Fisher / Schmidt DSE / @mfisher87
- Tom Nicholas / @TomNicholas
No Showcase Talk (next talk in Mar 5)
60 Second Updates:
- Justus:
- Progress on the distributed rtree (it works, but tends to grind to a halt for big grids)
- Proposals for scipy
- Martin week
- Implementing some geo stuff for akimbo
- Doing big complicated refactor for akimbo
- Performance drive on awkward and dask-awkward
- https://github.com/dask/dask/issues/11768
- Max
- NCAR hackathon on Xarray on GPUs this week
- Working on GeoZarr examples, starting with an explainer - https://link.excalidraw.com/l/762mjh4w1Pv/4C01qyi9rgS then building https://github.com/developmentseed/geozarr-examples
- Collaborating with Kyle on Obstore - Zarr and async-tiff
- Matt
- Organizing a GeoJupyter hackathon
- Tom
- Started at EarthMover
Agenda Items:
- Jeff’s I/O benchmark
2025-02-19 (9am PST; 12pm EST, 1700 UTC)
Attendees: Name / Institution / @GitHub:
- Rich Signell / Open Science Computing / @rsignell
- Justus Magin / CNRS / @keewis
- Anissa Zacharias / NSF NCAR / @anissa111
- Martin Durant / Anaconda / @martindurant
- Raphael Hagen/ CarbonPlan / @norlandrhagen
No Showcase Talk (next talk in Mar 5)
60 Second Updates:
- Rich S:
- Raphael
- Justus and I have been working on getting pangeo-forge working with Zarr v3
- Wondering if anyone has tips for zonal stats w/out getting into tiff/rioxarray land
- https://cloudnativegeo.org/blog/2025/02/geoparquet-2.0-going-native/
- Martin: ( plans )
- duckDB backend for akimbo (joins spark, ray, polars, cudf, dask, pandas)
Agenda Items:
- TBD
2025-02-12 (1pm PST; 4pm EST; 2100 UTC)
Attendees: Name / Institution / @GitHub:
- Rich Signell / Open Science Computing / @rsignell
- Justus Magin / CNRS / @keewis
- John Clyne / NCAR / clyne
- Tyler Erickson / VorGeo / @tylere
- Kevin Tyle / UAlbany / @ktyle
- Michael Sumner / Australian Antarctic Division / @mdsumner
- Martin Durant / Anaconda / @martindurant
- Orhan Eroglu / NCAR / @erogluorhan
- Raphael Hagen / CarbonPlan / norlandrhagen@gmail.com
- Julia Signell / Element 84 / @jsignell
- Tom Nicholas / @TomNicholas
- Harsha Hampapura / NCAR/ @hrhampapura
- Jonathan Joyce / RPS Tetra Tech / @jonmjoyce
- Ian Carroll / NASA GSFC & UMBC / @itcarroll
- Kristen Thyng / Axiom / @kthyng
- Chang Liu / Axiom - Tetra Tech / @cliu3
- Negin Sobhani / NCAR/ @negin513 /
- Sean Harkins / Development Seed / @sharkinsspatial
- Katelyn FitzGerald / NSF NCAR / @kafitzgerald
- Tadd Bindas / Penn State / @taddyb
- Lindsey Nield / Earthmover / @lindseynield
Showcase Talk (15-25 min): "FROST: Federated Registry Of Scientific Things" - Tom Nicholas
- Discussion
- Interest form
- Ryan Ab.: simple proof of concept: could we just chain together two separate datasets as a start?
60 Second Updates:
- Rich S:
- Pangeo Discord invite (expires in 7 days)
- Exploring source.coop
- John C.
- WCRP Digital Earths - Global Hackathon, May 12 - 16
- Global, distributed event (3 U.S. sites)
- Goals: Get experience with workflows for analysis of km-scale global climate model outputs
- HEALPix hierarchical grids stored in Zarr
Agenda Items:
- TBD
2025-02-05 (9am PST; 12pm EST, 1700 UTC)
Attendees: Name / Institution / @GitHub:
- Max Jones / Development Seed / @maxrjones
- Rich Signell / Open Science Computing / @rsignell
- Tom Nicholas / [C]Worthy / @TomNicholas
- Justus Magin / CNRS / @keewis
- Jon Marokhovsky / Clark CGA / @jmarokhovsky
- Martin Durant / Anaconda / @martindurant
- Julian Zeidler / German Aerospace Center (DLR) / @julian-zeidler
- Tom Moran / INCATech/ @tmodurr
- Valentin Genin / Thales Services Numeriques / @valentin-genin
- Tom Augspurger / NVIDIA / @TomAugspurger
- Julia Signell / Element 84 / @jsignell
- Wietze Suijker / Space Intelligence / @wietzesuijker
- Raphael Hagen / CarbonPlan / @norlandrhagen
- Pete Gadomski / Development Seed / @gadomski
- Jonathan Beezley / Spire Global / @jbeezley
- Doug Schuster / NCAR / @dcschus
- Riley Conroy / NCAR / @rpconroy
- Orhan Eroglu / NCAR / @erogluorhan
- Apurba Biswas/Statkraft/@apurba-biswas
- Martha Morrissey/Pachama/ @martham93
- Aaron Kaplan/IRI/@aaron-kaplan
- Will GIslason/Whitecap & UC Santa Cruz/@gislawill
- Jon Duckworth / Regrow Ag / @duckontheweb
- Pete Ingraham / Element 84 / @pingraham
- Anthony Aufdenkampe / LimnoTech / @aufdenkampe
- Harsha Hampapura / NCAR/ @hrhampapura
- Guido Stein / OSGeo US / @guidos
- Yves Moisan / Natural Resources Canada / @ymoisan
- Adam Cox / UIUC, OldInsuranceMaps.net / @mradamcox
- Sean Harkins / Development Seed / @sharkinsspatial
Showcase Talk (15-25 min): "High-performance Python STAC tooling, backed by Rust" - Pete Gadomski
- Slides: https://www.gadom.ski/2025-02-05-PangeoShowcase/
- Discussion
- Performance of geoparquet
- Putting Zarr in STAC
- Will virtualization replace STAC?
- Keep in mind small to middle practitioners who won’t have access to the latest technologies - disorganization
- STAC offers entrypoint discover for Zarr
- Zarr is broader than geospatial
- STAC expands metadata and defines metadata extensions really well
- Value in all that generated
60 Second Updates:
- Tom N:
- VirtualiZarr 1.3.0 released
- Now finally works simultaneously with released versions of Kerchunk, Icechunk, and Zarr-python v3!!! Means you can install all zarr-python v3 dependencies through conda in one environment without the funny business of using janky unmerged branches any more.
- Published blog post “Science needs a social network for sharing big data”
- Looking at feasibility of using Bluesky’s ATprotocol to publish updates to catalogs of version-controlled (initially Icechunk but idea is to be general) datasets via federated network
- (https://github.com/TomNicholas/FROST )
- Next Pangeo showcase (12th) will be a short talk + community discussion about this
- Martin ( week plan ):
- Akimbo’s spark and ray backends passing tests; considering if DuckDB is doable
- Any interest in fsspec-in-jlab file browser?
- “Fixed” dask-awkward is out; also releases of fsspec, kerchunk
- Tom A:
- Mentoring NREL/NCAR/NOAA hackathon on cupy / xarray / zarr GPU things.
- .
Agenda Items:
- Discussion about STAC <-> ZARR - https://hackmd.io/@maxrjones/B1jkBm-Fye/edit
2025-01-29 (1pm PST; 4pm EST; 2100 UTC)
Attendees: Name / Institution / @GitHub:
- Max Jones / Development Seed / @maxrjones
- Joe Hamman / Earthmover / @jhamman
- Rich Signell / Open Science Computing / @rsignell
- Alex Kerney / Gulf of Maine Research Institute / @abkfenris
- Justus Magin / CNRS / @keewis
- Ross Slater / University of Leeds / @rossawslater
- Tom Nicholas / [C]Worthy / @TomNicholas
- Wai Tik Chan/ Landscape and Geoinformatics (University of Tartu)/ @tik65536
- Matt Iannucci / Earthmover / @mpiannucci
- Martin Durant / Anaconda / @martindurant
- Kevin Tyle / UAlbany / @ktyle
- Tom Augspurger / NVIDIA / @TomAugspurger
- Yoni Nachmany / Zeus AI / @yoninachmany
- Micah Wengren / NOAA/US IOOS / @mwengren
- Sarah Jordan / LimnoTech / @sjordan29
- Raphael Hagen / CarbonPlan / @norlandrhagen
- Harsha Hampapura/NSF NCAR/ @hrhampapura
- Jonathan Joyce / RPS TetraTech / @jonmjoyce
- Patrick Tripp / RPS TetraTech / @patrick-tripp
- Chuck Daniels / Development Seed / @chuckwondo
- Scott Staniewicz / JPL / @scottstanie
- Miguel Jimenez / OPeNDAP / @Mikejmnez
- Florian Ziemen / DKRZ / @florianziemen
- Amelia Snyder / USGS / @amsnyder
- Chang Liu / Axiom Data Science - TetraTech / @cliu3
Showcase Talk (15-25 min): Pangeo Showcase: “Xpublish at Scale(s)” - Alex Kerney and Joe Hamman
- Discussion
- Xpublish delivering model output:https://github.com/asascience-open/xreds
- Live version of XREDS demo: https://nextgen-dev.ioos.us/xreds/
60 Second Updates:
- Max J:
- Still recruiting spring showcase speakers - reach out
- Pete Gadomski next week will present on STAC-rs
- Tom N:
- Big push right now to get VirtualiZarr working with zarr-python>=3.0.0
- Been thinking a lot about requirements for domain-agnostic catalogs…
- Coiled’s new integration with Prefect Serverless work pool looks cool
- https://docs.coiled.io/user_guide/prefect.html
- Interesting that this is Coiled but not Dask
- Justus:
- Plotting of multi-dimensional datasets with lonboard (using sliders)
- https://github.com/xarray-contrib/xdggs/pull/124
- Rich: (just some interesting links…)
- Panel 1.6 release (jan 25)
- The Petabyte FFT blog post by Alex Merose (jan 14)
- Martin (work doc)
- Kerchunk for zarr3 is slowly coming along (PR)
- Akimbo gains spark backend and will get ray.data soon
- Dask-awkward now fixed in main, dask-histogram still broken
- Fast optimize and task-specific options coming
- Joe:
- Zarr-Python 3 was released a few weeks ago - https://zarr.dev/blog/zarr-python-3-release/
- Continuing to work on Xarray/Zarr3/Icechunk front
Agenda Items:
- TBD
2025-01-22 (9am PST; 12pm EST, 1700 UTC)
Attendees: Name / Institution / @GitHub:
- Raphael Hagen / CarbonPlan / @norlandrhagen
- Max Jones / Development Seed / @maxjones
- Jesse Anderson / Numbers and Strings / @jessjaco
- Justus Magin / CNRS / @keewis
- Martin Durant / Anaconda / @martindurant
No Showcase Talk
60 Second Updates:
- Max
- Submitted an EGU abstract on open source work at Dev Seed
- Showcase presentations start next week!
- Benchmarking Zarr + Obstore - https://github.com/maxrjones/zarr-obstore-performance
- Planning work for the next few months
- VirtualiZarr DataTrees / COG support - https://github.com/NASA-IMPACT/veda-odd/issues/31
- Web-Optimized Zarr - https://github.com/NASA-IMPACT/veda-odd/issues/41
- Justus
- Progress on the regridding pipeline!🎉
- For finding pixels / cells to aggregate: https://github.com/keewis/grid-indexing/
- For actually computing the weights: https://github.com/keewis/grid-weights
- Martin ( public gdoc )
- Working on kerchunk (and fsspec tests) zarr v3 compat
- Dask broke dask-awkward/dask-histogram again
- Who can answer https://stackoverflow.com/questions/79367888/arrow-data-in-pyspark ?
- A fun intake workflow: “take this catalog and cache each entry in duck to produce a new catalog”
Agenda Items:
- TBD
2025-01-15 (1pm PST; 4pm EST; 2100 UTC)
Attendees: Name / Institution / @GitHub:
- Raphael Hagen / CarbonPlan / @norlandrhagen
- Max Jones / Development Seed / @maxrjones
- Martin Durant / Anaconda / @martindurant
- Aimee Barciauskas / Development Seed / @abarciauskas-bgse
- Ola Formo Kihle / UW/Kartverket / @ofk123
No Showcase Talk
60 Second Updates:
- Martin
- working on the Kerchunk <-> Zarr v3 update
- Listed out what he’s working on in general and each week
- Aimee working on virtualizarr icechunk implementation for MUR SST, trying to upgrade to icechunk 0.1.0a11/zarr-python 3, resolving API changes
- Max
- Planning to share https://github.com/maxrjones/why-virtualize-geotiff with the Pangeo Discourse to see if people would actually use virtualized GeoTIFFs
- Showcases start in two weeks! Reach out with any topics that you want to hear about or present on
- Tina shared that she can accept late abstracts for EGU - find her in the Pangeo Discord
- Ola
- Meeting with Tina and Anne
- Interested in connecting about VirtualiZarr+ in a couple weeks
Agenda Items:
- TBD
2025-01-08 (9am PST; 12pm EST, 1700 UTC)
Attendees: Name / Institution / @GitHub:
- Rich Signell / Open Science Computing / @rsignell
- Raphael Hagen / CarbonPlan / @norlandrhagen
- Justus Magin / CNRS / @keewis
- Tom Nicholas
No Showcase Talk
60 Second Updates:
- Rich:
- EGU Session reminder (abstracts due Jan 15)
- Upcoming Pangeo Showcase talks
- Justus:
- Working on a distributed rtree implementation:
- https://github.com/keewis/grid-indexing
- Result is a sparse boolean matrix
- Computes already, but the result is different from the in-memory index
- @justus have you seen this blog post? https://kylebarron.dev/literate-flatbush/
Justus: I’ve seen it because Rich pinged me on discord (but didn’t fully read it yet)
- Raphael:
- I’ve been exploring xpublish-wms
- Tom:
- Experimenting with virtualizarr.open_virtual_mfdataset
- Including using lithops for serverless parallelization of xarray.open_mfdataset
2025-01-08 No meeting - holiday break
See also the:
Archived Pangeo Weekly check-in notes from 2024
Archived Pangeo Weekly check-in notes from 2023
Archived Pangeo Weekly check-in notes from 2022
Archived Pangeo Weekly check-in notes from 2021
Archived Pangeo Weekly check-in notes from 2018-2020!
** TEMPLATE **
2025-XX-XX (1pm PDT; 4pm EDT; 2000 UTC)
2025-XX-XX (9am PDT; 12pm EDT, 1600 UTC)
Attendees: Name / Institution / @GitHub:
- Max Jones / Development Seed / @maxrjones
- Rich Signell / Open Science Computing / @rsignell
Showcase Talk (15-25 min):
- Discussion
60 Second Updates:
- Max J:
- Rich S:
Agenda Items:
- TBD