ESDS Forum

Links and Resources

Join the ESDS mailing list!

ESDS website

ESDS Zulip (messaging platform similar to Slack)

ESDS office hours

ESDS Forum Presentation Signup

Meeting Notes

September 15, 2025

Sign-In:

Name / Organization / @GitHub Handle:

? attendees

Agenda:

• ESDS announcements:

• Community updates (resources, events, suggestions, challenges, etc.):

• Presentation:

• Title:
• Presenter: Luis Lopez
• Slides:
• Notes:

August 18, 2025

Sign-In:

Name / Organization / @GitHub Handle:

• Katelyn FitzGerald / NSF NCAR | CISL / @kafitzgerald
• Harsha Hampapura / NSF NCAR | CISL / @hrhampapura
• Anissa Zacharias / NCAR / CISL / @anissa111
• Mya Sears / NCAR / EOL / @myasea8
• ana v espinoza / UCP | NSF Unidata / @ana-v-espinoza
• Brian Vanderwende / NCAR / HPCD / @vanderwb
• David Ahijevych / MMM
• Sean Arms / NSF Unidata / @lesserwhirls
• Tom Cram / NCAR CISL / @tcram
• Doug Schuster / NCAR / CISL / @dcschus
• Brian Bonnlander / NCAR - ISD / @bonnland
• Bob Dattore / NCAR CISL ISS / @rda-dattore
• Ethan Davis / NSF Unidata /@ethanrd
• Wayne Chuang / Columbia / @wkchuang

24 attendees

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• September 15th - NASA earthaccess

• Planning for Fall 2025 - reach out to Katelyn FitzGerald or Harsha Hampapura with ideas / input

• Community updates (resources, events, suggestions, challenges, etc.):

• ESDS Collaboratory @ Foothills Lab: Please sign up for the next ESDS collaboratory and let us know which topics you would like to discuss: https://docs.google.com/forms/d/e/1FAIpQLSdQ8WA1oOqzLjHHJSaOMF4tZWApUrAvxDnfHB8ZFNoBtG5qBA/viewform 
• Last ESDS collaboratory: https://ncar.github.io/esds/ 

• Presentation:

• Title: Data Commons Update
• Presenter: Doug Schuster, NSF NCAR / CISL
• Slides: Google Slides Link
• Links:

• CESM2 LENS Zarr Subset https://rda.ucar.edu/datasets/d010092/
• OSDF notebook examples: https://github.com/NCAR/osdf_examples 
• List of RDA Zarr datasets: https://rda.ucar.edu/gsearch/dataset-search/?q=zarr 
• OSDF Cookbook: https://projectpythia.org/osdf-cookbook/ 

• Description:
• Notes:

• Motivation for Data Commons

• NCAR’s data is scattered across different siloed data repositories
• We are not realizing the full potential of NCAR’s data

• Current Challenges

• Remote Users

• Legacy Download and Analyze Model → Time consuming and inefficient
• Data transfer and service qualities can vary widely based on the repository

• NCAR’s HPC users

• Limited to NCAR HPC account
• Only a subset of NCAR’s data accessible via Derecho/ Casper
• No good way to search for a dataset

• Solution ? Loosely connected data repos → GDEX Data Commons, with the following features

• Simple and consistent access to both novice and expert users
• Integrated for Insight: Data proximate compute, seamless tools for data exploration, AI/ ML factory
• Trust

• GDEX data commons:

• RDA already has many features of a Data Commons

• Scalable streaming access for remote users
• Multiple data delivery mechanisms
• Integrated with NCAR and other community computations services
• RDA is CoreTrust Seal certified
• RDA also has many popular datasets (see slides)

• So, rebrand RDA → GDEX

• Implementation Timeline (see slides)
• Vision to Impact:

• Trust and Reuse
• Analysis Ready Data
• Support AI/ML
• Operational resilience

• FY25

• Data curation tools
• Possible LLM integration
• Deploy on CIRRUS

• Enables CI/CD, ARGO CD
• Resilient: Can run both on NWSC and Mesa Lab

• Research and curate analysis-ready, AI-optimized datasets
• Develop and publish data ingestion examples- Pythia Cookbooks

• Integration with Open Science Data Federation (OSDF)

• https://rda.ucar.edu/resources/ndcc-osdf/ 
• Integration with the OSPool (https://osg-htc.org/services/ospool/)

• Zarr access to datasets: Currently 6 datasets. Rapidly growing list
• FY25 milestones:

•  CDG and legacy GDEX data migrated to either NCAR Zenodo or RDA (Aug 27th). CDG and legacy GDEX will be deprecated
• Sep 9, 2025:
• Rebrand RDA → GDEX
• Legacy links point to new GDEX
• RDA data help desk consolidation: datahelp@ucar.edu (More resilient)

• FY26: Goals

• More testing on CIRRUS
• UX research on data access
• LLM integration for data search, discovery and curation
• Curation of AI/ML datasets
• Test integration of EOL and HAO datasets
• Develop example workflows → Pythia Cookbook

• Do you have a dataset that might benefit the community ? Get in touch with Doug!

• Q&A (drop things here or in the chat if you’d like)

• Input, Feedback, Discussion, Q&A

July 21, 2025

Sign-In:

Name / Organization / @GitHub Handle:

•  David Ahijevych/MMM
•  Negin Sobhani / CISL / HPCD / @negin513
•  Orhan Eroglu / CISL / @erogluorhan
•  ana espinoza / NSF Unidata / @ana.v.espinoza
• Brenda Javornik / NSF NCAR EOL / @leavesntwigs
• Nick Cote / CISL/HPCD / @NicholasCote
• Michael Levy / CGD / @mnlevy1981
• Justin Richling/ CGD/ @justin-richling
• John Clyne / CISL / @clyne
• Julia Kent / CISL / @jukent
• Erik Johnson, EOL, @erikj
• Ward Fisher / NSF Unidata / @wardf
• Sean Arms / NSF Unidata / @lesserwhirls
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Bob Dattore / CISL/ISS / @rda-dattore
• Rich Neale / CGD / @swrneale
• Anissa Zacharias / CISL / @anissa111
• Helen Kershaw / CISL / @hkershaw-brown
• Drew Camron / NSF Unidata / @dcamron

30 attendees

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• August 18th - NASA earthaccess | Data Commons

• ESDS collaboratory 🧑‍💻

• Thanks to those who attended!
• Look forward to another this fall 🍁

• NCDC Slack

• Reach out to Katelyn FitzGerald with any feedback or input

• Community updates (resources, events, suggestions, challenges, etc.):

• Foundation Models for Science Workshop (November 3-5, 2025) @ University of Toronto

• Presentation #1:

• Title: NetCDF: Maintaining a healthy, community-supported software project
• Presenter: Ward Fisher, Unidata
• Slides: Google Slides Link
• Links:

• Unidata: https://www.unidata.ucar.edu/
• NetCDF: https://www.unidata.ucar.edu/software/netcdf/

• Description:
• Notes:

• What is Unidata?

• Great overview here: https://www.unidata.ucar.edu/about/tour/index.html

• What is NetCDF?

• Data Model
• Data Files
• Software Library / API

• When is NetCDF used?:

• Array-oriented scientific data (though not strictly this)
• Portable, self-describing*
• Emphasizes efficient, direct access to data within files
• Avoids deps on external tables and registries
• Emphasizes simplicity over power, serving common use cases

• Current status - active project

• Midway through the 4.10.0 dev cycle for the core lib
• Efforts

• Zarr

• ncZarr v2 support - implemented
• ncZarr v3 support - partial in v4.9.3, full coming in 4.10.0
• Unidata active in the Zarr community

• General bug fixes and improvements

• Broad recognition of the importance

• Challenges

• Resources vs work to do - bugs, PR review, support, proposals, service, and other hats
• Deep expertise in Fortran and MPI parallelism
• Progress and stability under limited resources

• Community

• Users and developers
• Support in third-party applications
• Third-party APIs for many other languages (e.g. R)
• CF Conventions
• Sharing and advocating
• Early, often, engaged, and sustained communication
• Management, cultivation, and governance of the community and project

• Presentation #2:

• Title: Thematic Real-time Environmental Distributed Data Services (THREDDS) –  a polyglot future
• Presenter: Sean Arms, Unidata
• Slides: Google slides link
• Links:

• TDS - https://www.unidata.ucar.edu/software/tds/
• Siphon - https://www.unidata.ucar.edu/software/siphon/

• Description:
• Notes:

• Thematic Real-time Environmental Distributed Data Services (THREDDS)

• Client (netCDF-Java) / Server (THREDDS)
• Originally funded for university usage
• Many (~20) universities running these and sharing their data
• Many other folks across the world (EDU / GOV / other - roughly even split between each)

• See slides for the architecture diagram
• Siphon (client for access)
• Data Access Layer - Common Data Model (CDM)

• Future direction:

• Contributions welcome
• Revisiting the approach for sustainability
• Challenges with legacy code / tech
• Cross-language TDS

• Phases:

• Bottom-up
• Horizontal-out

• Cross-language communication that works on the CDM level
• gRPC CDM - https://grpc.io/docs/what-is-grpc/introduction/

• Looking at various options to replace / enhance TDS services (Apache SDAP, EarthMover, publish, WIS 2, pygeoapi, pycsw, …)

June 23, 2025

Sign-In:

Name / Organization / @GitHub Handle:

• Harsha Hampapura / CISL/ @hrhampapura
• Brian Dobbins / CGD / @briandobbins
• Tom Cram / CISL / @tcram
• Katelyn FitzGerald / CISL / @kafitzgerald
• Brian Bonnlander / CISL / @bonnland
• Helen Kershaw / CISL @hkershaw-brown
• Brian Medeiros  / CGD / @brianpm
• Anissa Zacharias / CISL / @anissa111
• Nathan Lenssen / CGD / @nlenssen
• John Clyne / NCAR / @clyne
• Thomas Martin / NSF Unidata / @thomasmgeo
• Wayne Chuang / @wkchuang
•  Juila Kent / @jukent
•  Matt Rehme / CISL / @mattrehme

29 attendees

Agenda:

• ESDS announcements:

• JK - ESDS Collaboratory on Wednesday!
• KF - ESDS Forum Organizer Position

• Rotating role with a 1-year term
• Ideally starting in fall 2025
• Reach out to Katelyn FitzGerald or Harsha Hampapura

• Upcoming ESDS Forums

• July 21st - Unidata projects
• August 18th - NASA earthaccess

• Community updates (resources, events, suggestions, challenges, etc.):

• Presentation #1:

• Title: The NCAR Climate Data Guide: Connecting Data Experts with Users
• Presenter: Nathan Lenssen, Climate Analysis Section (CGD-CAS)
• Slides: Google slides link
• Links:

• Climate Data Guide

• Description:
• Notes:
• Slide 1: Nathan’s journey to the current position in

• CAS and Colorado School of mines
• Research on understanding climate uncertainty

• Overview

• History, philosophy and successes of the CDG (Climate Data Guide)
• Current and future

• CDG: Accessible, equitable and repo for climate community’s wisdom
• DCENT example: https://climatedataguide.ucar.edu/climate-data/dcent 

• Overview of the dataset
• Consolidated metadata
• Strengths and limitations of the datasets provided by expert users/ developer of the data
• Data Access links
• Expert Developer Guidance: How to use/ NOT use the data, biases artifacts etc (most important part of the climate data guide)
• Expert guidance has mostly been solicited.
• Common questions that experts need to answer: See slides

• CDG collection

•  25 + datasets on precipitation, Climate indices, Sea ice indices, Land surface data, Radiation, Ice core and clouds
• Growing collection of ocean subsurface data
• 200+ datasets
• 2024: 250K + users. See slides for google analytics data

• Ongoing work

• Maintain and update pages
• Expand to include new variables + new datasets + new components of the Earth System
• Develop framework + tools for dataset intercomparison
• Example:  Open source notebooks, Journal articles

• Ways to get involved:

• Join the Board of Advisors (subject matter experts) - meets 3-4x / year
• Request or contribute a dataset

• ~500 words and some figures

• Found a broken link?  Email Nathan
• Data provider/ expert user ? Contribute your expert guidance
• Data curator? Link data download pages
• Collaboration ideas?  Reach out

• E.g. open source notebooks, intercomparison journal articles, etc.

• Q&A:

• RDA - opportunities for improved updates / linking
• Any restrictions on adding a new dataset ? Example: Hosting on AWS

• Presentation #2:

• Title: Tracking and Object-Based Analysis of Clouds (tobac)
• Presenter: Julia Kukulies 
• Slides: https://docs.google.com/presentation/d/101cJXbMwdZMjo7TUWlgvgLn7oCKHduj6/edit?usp=drivesdk&ouid=115876143777061300216&rtpof=true&sd=true
• Links:

• https://tobac.readthedocs.io/
• https://github.com/tobac-project/tobac
• Publications:

• https://doi.org/10.5194/gmd-12-4551-2019
• https://doi.org/10.5194/gmd-17-5309-2024

• Description:
• Notes:
• Julia K, postdoc at M^3, works with km-scale modeling, storm and cloud tracking
• Tobac

• Has existed for ~ 8 years
• Python library designed to identify, track and analyze clouds and/or storms
• Can be used with any variable on any gridded dataset (including observations)
• Doesn’t work on unstructured grids (interested though and on the list of priorities)
• Can identify and track features in both 2D and 3D
• Flexible and modular framework
• Example: See slides

• Why tobac?

• Input variable and grid agnostic
• Used by NASA’s AOS and INCUS satellite missions
• Can track any feature/ proxy for storms
• Examples: Cloud IR brightness, Flashing rates etc (See slides for more details)
• Developed and maintained by a large group of scientists and engineers

• How does it work ?

• Uses scikit-image and multiple thresholds (as opposed to a universal threshold) to detect features (e.g. detect both strong and weak storms)
• Works both in 2D and 3D, across periodic boundaries

• Bigger picture

• Storm tracking has focused on strong deeply convective systems by NWP and other communities
• But, need better tools for a broader community
• New km-scale models and high-resolution observations necessitate the needs for cloud tracking tools
• Need for multivariate tracking across scales and enable broader community participation

• Development

• Open source, open science
• Monthly developer meetings, github discussions
• Reach out/ COntact: tobac.io 
• Enhanced documentation and tutorials in the last 2 years
• Pythia cookbook under development

• Tobac workflow: See slides

• Feature detection: Input : xarray dataarray  → Filtering → Single feature points identified → Output as a pandas dataframe
• Track the features /merge or combine features

• Tracking clouds

• Predictive tracking using trackpy package: predicts based on motion of the feature
• Can combine multiple data sources to segment and track a feature

• What after detecting a feature ?

• Bulk statistics of the detected feature using tobac
• Example: Shallow clouds in LES (See slides)

• Future - https://github.com/tobac-project/tobac-roadmap

• Full xarray support
• Dask support
• UNstructured grids
• Multivariate tracking

• Q&A:

May 12, 2025

Sign-In:

Name / Lab / Division / @GitHub Handle:

• Katelyn FitzGerald / CISL / @kafitzgerald
• Tom Cram / CISL / @tcram
• Mike Levy / CGD / OS / @mnlevy1981
• Brian Dobbins / CGD
• Anissa Zacharias / CISL / @anissa111
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Katie Dagon / CGD / @katiedagon
• Doug Schuster / CISL / @dcschus
• Nick Cote / CISL/HPCD / @NicholasCote

13 attendees

Agenda:

• ESDS announcements:

• Up next:

• Break
• June 23, 2025 - Climate Data Guide | Tobac

• Community updates (resources, events, suggestions, challenges, etc.):

• Presentation #1:

• Title: PelicanFS
• Presenters: Emma Turetsky
• Slides: ESDS-Turetsky-May.pdf
• Description:
• Notes:

• Brief overview and review of OSDF
• PelicanFS

• GitHub repository
• Available via PyPI and Conda

• Why Python?

• Jupyter notebooks
• A lot of Python usage in ML, various science domains, etc.
• FSSpec

• Offered something to build upon
• Probably already use this even if you haven’t heard of it e.g. with Dask, Intake, Xarray, Zarr, Pandas
• Already has other object store protocols

• Demos: largely on GitHub
• Future work:

• Improving authentication
• Improved integration with ML libraries and OSPool
• Closing the split between PelicanFS and the command line client

• Additional examples from Harsha: https://github.com/NCAR/osdf_examples?tab=readme-ov-file
• NCAR/Pelican project: https://rda.ucar.edu/resources/ndcc-osdf/
• Have questions / requests!?

• Log an issue on the GitHub repo or upvote an existing one!

• Presentation #2:

• Title: ESDS Updates
• Presenters: Mike Levy and Katelyn FitzGerald
• Slides: 2025.05.12 -- ESDS Updates
• Description:
• Notes:

April 28, 2025

Sign-In:

Name / Lab / Division / @GitHub Handle:

• Katelyn FitzGerald / CISL / @kafitzgerald
• Sean Arms / NSF Unidata / @lesserwhirls
•  Harsha Hampapura/ CISL/ @hrhampapura
• Anissa Zacharias / CISL / @anissa111
•  Julia kent / CISL / @jukent
•  Keith Lindsay / CGD / @klindsay28
• Drew Camron / NSF Unidata / @dcamron
•  Cora Schneck / CISL / TDD / @cyschneck
• Nick Cote / CISL/HPCD / @NicholasCote
• Mya Sears / EOL / @myasea8

Negin Sobhani / CISL / @negin513

•  Thomas Martin / NSF Unidata / @thomasmgeo
• Michael Levy / CGD / @mnlevy1981
• Max Grover / Argonne National Lab / @mgrover1
• Tom Cram / CISL / @tcram
• Katie Dagon / CGD / @katiedagon
• Justin Richling/ CGD/ @justin-richling

29 attendees

Agenda:

• ESDS announcements:

• Zulip to Slack transition

• Timeline

• Slack invitations - today 🎉
• Zulip read-only - May 16, 2025
• Zulip decommissioned - May 30, 2025

• Data transfer and archival

• No data transfer
• Archive of public Zulip streams (no images)

• Pending name change to NCAR Compute and Data Commons (NCDC)
• Channels

• #monthly_meetings will transition back to the #general channel
• #upcoming-opportunities (workspace-wide)
• #python-users (hosted by ESDS)
• #esds-community (main ESDS channel)
• Ability to create channels
• What else are we missing?

• Slack invitation link for ESDS

• Upcoming ESDS Forums        

• May 12, 2025

• PelicanFS - Using the Pelican Python FSSpec to Access Data
• ESDS Updates

• Break until Jun 23, 2025

• Climate Data Guide
• Tobac - Tracking and Object-Based Analysis of Clouds

• Community updates (resources, events, suggestions, challenges, etc.):

• Presentation #1:

• Title: Intake-ESGF vs. Intake-ESM
• Presenters: Max Grover
• Slides: https://mgrover1.github.io/esds-intake-esm-esgf-2025/
• Description:
• Additional resources / links:

• intake-esm docs - https://intake-esm.readthedocs.io/en/stable/
• intake-esgf docs - https://intake-esgf.readthedocs.io/en/latest/
• Demo repo from today - https://github.com/mgrover1/esds-intake-esm-esgf-2025
• Intake-esgf pythia cookbook https://projectpythia.org/esgf-cookbook/notebooks/intro-search.html 

• Notes: Intake-ESM v/s Intake-ESGF

• What problems are these packages trying to solve ?

• Tracking millions of files that store ESM data
• Easy reproducibility

• Earth System Grid Federation

• https://esgf.llnl.gov/ 
• CMIP5 > 1.5 PB
• CMIP6 > 16.1 PB
• CMIP7 ~ 100 PB

• ESGF 2: Next gen ESGF

• Effort supported by DOE
• ElasticSearch | STAC

• Summary of Intake-ESM

• For more details, see  https://intake-esm.readthedocs.io/en/stable/

• Future ?

•  In maintenance mode since 2022
• Still actively used in NOAA, ACESS-NRI
• Improvements

• Change to polars based backend lead to speedup
• Improved performance in terms of memory
• Integrations with SQL ?
• Integrations with intake-esgf

• Intake-ESGF

• A catalog tool developed for ESGF-2-US
• Intake-like , but not a plug-in in the intake ecosystem
• More details: https://intake-esgf.readthedocs.io/en/latest/
• Limitations: Setting up elastic search
• In beta-mode. Adoption is growing
• Globus-backed search enabled
• Future: Support for WPS, intake-ESM catalogs, remote computing

• Demo:  https://github.com/mgrover1/esds-intake-esm-esgf-2025
• Questions

• https://pcmdi.github.io/obs4MIPs/
• Thomas Martin: Issues with moving from Pandas to Polars

•  No. Just drop-in changes

• Presentation #2:

• Title: CIRRUS Update
• Presenters: Nick Cote
• Slides: CIRRUS_ESDSUpdate4-28.pptx
• Description: CIRRUS Introduction
• Additional resources / links:
• Notes: On-Prem Cloud

• Flexible compute options proximate to NCAR data
• Uses Kubernetes

• Container management platform

• Open source, cloud native, reproducible, runs anywhere, connects to existing hardware, feature rich

• Infrastructure as Code and GitOps

• Systems are automatically deployed using git states
• Version controlled infrastructure
• Enables CI/CD using helm charts

• CIRRUS hardware config: See slides
• V1-beta release: Workflow

• Container image required
• App defined in YAML, YAML stored in git as helm chart
• CIRRUS admins add the repo to Argo-CD
• Changes to YAML sync automatically
• Apps can be made publicly available or restricted to the UCAR network
• Load balancers in place for publicly available apps

• Github actions runner scale sets: See slides

• You may have received emails about github actions from NCAR github admins

• Ready to use tools

• Harbor: Private container registry
• OpenBao: Web based secrets manager, use SSO to store sensitive info
• Jupyterhub: Dask gateway installed, read-only access to glade, GPU images for TensorFlow and PyTorch
• Binder: Share computational environment, use git repo to reproduce and share results, access to glade and dask gateway

• Next steps

• ML cluster done, NWSC cluster should be completed soon
• V1-beta release announcement
• Publish production documentation
• Workshop with user focus groups
• CIRRUS team can help if you are interested

• Questions

• Katelyn Fitzgerald: Timeline of production docs + service

• By the end of this week

April 14, 2025

Sign-In:

Name / Lab / Division / @GitHub Handle:

•  Harsha Hampapura/ CISL/ @hrhampapura
•  Allison Baker / CISL / @allibco
•  David Ahijevych
•  Brian Vanderwende / CISL / HPCD / @vanderwb
• Nick Cote / CISL/HPCD / @NicholasCote
• Negin Sobhani /CISL /HPCD / @negin513
•  Cora Schneck / CISL / TDD / @cyschneck
• Katie Dagon / CGD / @katiedagon
• Wayne Chuang / Columbia / @wkchuang
• Thomas Martin / NSF Unidata / @ThomasMGeo
• Bob Dattore / CISL/ISD/DECS / @rda-dattore
• NIhanth CHerukuru/ CISL/TDD / @NihanthCW
• Katelyn FitzGerald / CISL / TDD / @kafitzgerald
• Doug Schuster / CISL /@dcschust
• John Clyne / CISL / @clyne
• Tom Cram / CISL/DECS / @tcram  
• Anissa Zacharias / CISL / TDD / @anissa111
• Brenda Javornik / EOL / RSF / @leavesntwigs
• Joe Tribbia /CGD /AMP/ @tribbia

29 attendees

Agenda:

• ESDS updates

• Upcoming forums

• April 28, 2025

• Intake-esgf vs. Intake-esm - Max Grover
• CIRRUS update - Nick Cote

• May 12, 2025

• Using the Pelican Python FSSpec to Access Data - Emma Turetsky
• ESDS Updates

• ESDS Zulip announcement

• Transitioning to Slack over the next couple of months

• More details on the transition coming soon
• Zulip will be read only as of May 16, 2025

• Zulip archive for public streams

• Community updates (resources, events, suggestions, challenges, etc.):

• Digital Earths Global Hackathon - May 12 - 16

• Information for the NSF NCAR node
• Registration closes Wednesday, April 23!!!

• Presentation:

• Presenter: Brian Bockelman
• Description:

The Open Science Data Federation (OSDF) is an NSF-funded service that connects the nation's disparate scientific dataset repositories into a single fabric.  The OSDF aims to cover a broad swath of science content, including NCAR's Research Data Archive, providing high throughput access for workflows with high levels of reuse.

In this talk, we'll cover the basics goals of the OSDF, how the technologies work "under the hood", the targeted user experiences, and where the project is headed next.

• Slides: OSDF_2025_04.pdf
• Additional resources / links:

• PATh (project)
• OSDF - Open Science Data Federation (service)
• Pelican Platform (software)
• OSPool (computing capacity)

• Notes:
• What is OSDF?

• Infrastructure that enables distributed data access to open scientific data
• OSDF = Netflix for Science: Streaming data as opposed to the legacy `download and analyze’ model
• Allow computational workflows to stream data
• OSDF: Cloudflare for science i.e., infrastructure for scaling access to your datasets

• https://path-cc.io/ 
• OSDF enabling science: Examples

• LIGO data can be accessed via OSDF (~27 PB of objects were moved, 200TB of unique data)

• For both proprietary and public data

• PI from University of Hawai sharing data using OSDF

• https://github.com/ifauh/spin4d-data

• NSF NCAR examples:

• https://github.com/NCAR/osdf_examples

• OSDF overview

• Network of Caches and origins across the US (and world) that deliver data
• All data are treated as objects.
• Who uses it ?NOAA, NSF, NCAR, DOE

• Accessing data from multiple origins in a jupyter notebook hosted on Casper

• https://github.com/NCAR/osdf_examples/blob/main/notebooks/cmip6_temps_zarr.ipynb 
• Object requested is delivered from the nearest cache
• If not present in the cache, the object is fetched from a data origin, is copied to the cache and then delivered to the user by the cache

• Major use case for OSDF: OSPool

• https://osg-htc.org/services/open_science_pool.html 
• 220 million jobs, 3.2 billion files (~60 PB) transferred to OSPool access points last year
• Example: NRAO Large Scale Distributed Analysis performed using HTCondor software on the OSPool, with the data transferred using OSDF

• Future Directions

• Improving scalability
• Improving observability
• Improve the clients: CLI vs PelicanFS

• Q&A (feel free to add questions w/ your name if you’d like):
•  Douglas Schuster:

• What is the sustainability model for OSDF ?
• Internet2, NSF funding, sometimes community pays for maintaining caches
•  How can new data providers join the OSDF ?

•   Thomas Martin

• GO-based software stack: Building software stack from scratch vs using existing software

March 31, 2025

Sign-In:

Name / Lab / Division / @GitHub Handle:

• Katelyn FitzGerald / CISL / @kafitzgerald
• Doug Schuster / CISL / @dcschus
• Pablo Lichtig / ACOM / @blychs
• ana victoria espinoza / NSF Unidata / @ana-v-espinoza
• Negin Sobhani / CISL / CSG/ @negin513
• Brian Bonnlander / CISL / ISD / @bonnland
• Harsha Hampapura / CISL/ DECS / @hrhampapura  
• Mike Levy / CGD / @mnlevy1981
• Nick Cote / CISL/HPCD / @NicholasCote
• Julia Kent / CISL / @jukent
• Tom Cram / CISL / @tcram
• Thomas Martin / NSF Unidata / @ThomasMGeo
• John Clyne / CISL/ @clyne
• Charlie Becker / CISL TDD MILES / @charlie-becker
• Eric Nienhouse / CISL / @ericnienhouse  
• Nathan Hook / CISL / @nathan-hook
• Riley Conroy / CISL / DECS / @rpconroy
• Teagan King / CGD / @teaganking
• Ward Fisher / NSF Unidata / @wardf
• Katie Dagon / CGD / @katiedagon
• Ben Gaubert / ACOM
• Behrooz Roozitalab/ ACOM  
• Julien Chastang / Unidata / chastang@ucar.edu
• Joe Tribbia /CGD/AMP @tribbia

34 attendees

Agenda:

• ESDS updates

• Upcoming ESDS Forums

• April 14, 2025

• OSDF - Connecting the Nation's Datasets - Brian Bockelman

• April 28, 2025

• Intake-esgf vs. Intake-esm - Max Grover

• May 12, 2025

• PelicanFS - Using the Pelican Python FSSpec to Access Data - Emma Turetsky
• ESDS Updates

• Summer schedule

• Roughly monthly
• Sign up

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Digital Earths Global Hackathon - 12 - 16 May, 2025 - registration open (NCAR node registration closes April 23!)

• Presentation #1:

• Title: Enabling a Science Gateway at NCAR
• Presenters: Eric Nienhouse
• Slides: ESDS 3/31/2025  - Enabling Science Gateway
• Additional resources / links:
• Notes:

• The science gateway is a part of the larger Data Commons effort
• Effort to minimize time spent on technology and focus on the science
• Vision of the Science Gateway

• Discover workflows related to your research
• AI/ML research with a shorter learning curve
• Move away from the legacy download and analyze model for remote users: Time consuming and inefficient, requires technical skills

• Data Commons Workshop takeaways (report: http://dx.doi.org/10.5065/gfbq-8y08)

• Engagement: Multiple entry points for novice and new users, intermediate and advanced.
• Service and Tools: Facilitate creativity among users. Enable users to bring their own tools , code and share it with others in a searchable way

• Data Commons Vision statement (see slides)
• Science Gateway/ Virtual Research Environment

• Hide the complexity of the underlying architecture
• Enable scientists and educators to focus on research and training
• Established frameworks and best practices exist

• What is a Science Gateway ? https://www.nature.com/articles/s41597-023-02029-x 

• Nanohub and Sciserver examples discussed

• Work in Progress- Pain points

• CLIs are challenging for novice users, GUIs and web browsers preferred
• But, free-tiers of cloud computing limits scaling up
• Documentation and expert guidance are not easily accessible
• Help desks and other traditional support structures are `slow’
• Examples for tools and workflows are hard to find

• Common Solutions

• Jupyterhub and exemplar notebooks
• Zulip/ Slack/ instant messaging apps for community engagement and support
• Hackathon: Kick start projects and help transition users
• Integration with community data fabrics
• LLMs for help with documentation

• Current/ Next steps

• Gather user needs - ESDS, Unidata, Data Commons Workshop participants
• Identify a MVP for a science gateway
• Consolidate CISL data repos

• Contact: Eric Nienhouse (w/ feedback, suggestions, interest in doing a user interview, etc.)
• Q&A:
•  A clear vision for NCAR’s science gateway ?

• Not sure yet
• CU’s OpenOnDemand based jupyterhub discussed
• Providing access to dashboards, example notebooks, tools for routine tasks like re-gridding

• Can ESDS help reach out to novice users ? Any pain points that resonate with the ESDS community ?

• Presentation #2:

• Title: MELODIES-MONET
• Presenters: Pablo Lichtig
• Slides: ESDS MELODIES-MONET presentation
• Additional resources / links:

• Documentation: https://melodies-monet.readthedocs.io/en/stable/

• Notes:

• A FOSS, model eval framework for atmospheric chemistry using observations

• NSF NCAR & NOAA collaboration
• Expecting a v1 release in May

• Mandatory requirements for the code: monet (no longer actively developed, but monetio and this project are), monetio (developed at NOAA), pyyaml, pandas
• Structure of the code

• See the slides

• User interacts with the tool using a YAML file

• With 5 sections:  analysis, model, obs, plots and stats
• Examples provided in the docs
• User is expected to write the same 8-10 lines of code in the MELODIES-MONET script→ easy to use
• CLI

• Driver class in the code reads YAML file, pairs the data and plots
• Challenges

• Need different readers for different models
• Not all types of data and models are supported
• Can work with satellite and aircraft data. See slides for examples
• For satellite data: Need a averaging kernel

• NASA’s TEMPO tool discussed

• A geostationary satellite instrument
• Pair model and satellite data over each swath + regridding → Satellite data on the model grid

• Looking for devs and collaboration (e.g. UXarray)!  Can reach out to the presenter.
• Also looking to grow ground-based remote sensing

• Q&A:

March 17, 2025

Sign-In:

Name / Lab / Division / @GitHub Handle:

• Katelyn FitzGerald / CISL / @kafitzgerald
• Harsha Hampapura/ CISL/ @hrhampapura  
• Negin Sobhani / CISL / @negin513
• Anissa Zacharias / CISL / @anissa111
• Brian Bonnlander / CISL / ISD / @bonnland
• Nihanth Cherukuru / CISL / @NihanthCW
• Ian Franda / CISL / @ifranda
• Kirsten Mayer / CGD / @kjmayer
• Keith Lindsay / CGD / @klindsay28
• Mike Levy / CGD / @mnlevy1981
• Katie Dagon / CGD / @katiedagon
• Julia Kent / CISL / @jukent
• Mya Sears / EOL / @myasea8
• Shima Shams /RAL/ @sshams
• Curtis Walker / RAL / @curtiswalker
• Joe Tribbia /CGD/ @tribbia
• Eric Nienhouse / CISL / ISD / @ericnienhouse
• Pablo Lichtig / ACOM / @plichtig
•  Doug Schuster / CISL / @dcschus
• Benjamin Gaubert / ACOM

28 attendees

Link to the OpenVisus slides:
University of Utah Ncar_talk_Open visus.pptx

Agenda:

• ESDS updates

• Upcoming ESDS Forums

• March 31, 2025

• Enabling a Science Gateway at NCAR - Eric Nienhouse
• MELODIES-MONET - Pablo Lichtig

• April 14, 2025

• OSDF - Connecting the Nation's Datasets - Brian Bockelman

• Summer schedule

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• SEA ISS 2025 - 7-10 April, 2025 - registration open
• Digital Earths Global Hackathon - 12 - 16 May, 2025 - registration open

• Presentation:

• Title: OpenVISUS for Petascale scientific visualization
• Presenters: Valerio Pascucci and Aashish Panta

• <feel free to add contact info as well>

• Slides: <feel free to add a link here if you’d like>
• Additional resources / links:

• sci.utah.edu

• Notes:

• OpenViSUS software for large scale visualization
• Leveraging the National Science Data Fabric
• Doing user interviews (reach out if you’re interested)
• OpenViSUS

• Basic core techniques

• Slicing, volume rendering
• Topology
• Statistics

• Out of cord processing for optimizing data access
• Pipelines of progressive algorithms
• Coarse to fine construction of multi-resolution models
• Remote data streaming

• OpenVisus SDKs

• C++ core library
• Python
• Docker
• Spack

• Basic workflow

• Conversion

• Create an Idx file (metadata file) with fields, dims, timesteps - createidx
• Read existing data as numpy array
• Write IDX
• Apply compression (lossy or lossless) and upload to cloud (optional)

• Zip, ZFP, LZ4
• Does add compression / decompression steps

• 2 levels of quality (spatial resolution and numerical precision)
• Reading
• Multiresolution data streaming

• Huge impacts in terms of resources, costs, etc. (w/ data transfer)

• Cloud costs
• Better use of shared resources

• Exploring working w/ netCDF data

• Allows for use of Xarray w/ MR streamed data

• Dashboard examples

• https://chpc3.nationalsciencedatafabric.org:12347/dashboards
• Allows for downloading subsets, etc.
• Working on AI features (in progress)

• Works from Jupyter notebooks as well
• https://sciviscontest2026.github.io/
• https://aashishp.quarto.pub/nex-gddp-cmip6/ 

• Q&A:

• <questions here>
• From chat:

• “Lower resolution is achieved through subsampling?”

• Allow for different filters (subsampling is probably the fastest, but there are others e.g. wavelet transforms)

• Is the example notebook available? Yes (visus.org - link in slides) - https://aashishp.quarto.pub/nex-gddp-cmip6/
• How do Openvisus access patterns compare with the HEALPix-based algorithms for accessing geospatial data ?
• What does your user community look like right now and what are you targeting?

• Trying to avoid creating their own format for this reason
• A good bit of traction in Materials Science
• Working on Geosciences now (e.g. some NASA users)

• What about 3D visualization?

• Do have tools for this (started there)… but is still brittle w/ web based tools
• Working on dashboard plugins (VTK plugin)
• Often 2D is enough for users anyway

March 3, 2025

Sign-In:

Name / Lab / Division / @GitHub Handle:

• Brian Bonnlander / CISL / @bonnland
• Katelyn FitzGerald / CISL / @kafitzgerald
• Harsha Hampapura / CISL/ @hrhampapura
• Dave Ahijevych / MMM David Ahijevych
• Nathan Hook / CISL / @nathan-hook
• Nihanth Cherukuru/ CISL / @NihanthCW
• Nick Cote / CISL/HPCD / @NicholasCote
•  Julia Kent / CISL / @jukent
• Thomas Martin / NSF Unidata / @ThomasMGeo
• Anissa Zacharias / CISL / @anissa111
• Susan Stringer / EOL / @susanstringer760
• Brian Dobbins / CGD / @briandobbins
• Ana Victoria Espinoza / NSF Unidata / @ana-v-espinoza
•  Allison Baker / CISL / @allibco
• Teagan King / CGD / @TeaganKing
• Doug Schuster / CISL @dcschus
• Riley Conroy / CISL @rpconroy
• Brian Medeiros / CGD / @brianpm
• Eric Nienhouse / CISL / @ericnienhouse
• Ben Gaubert / ACOM
• David John Gagne / CISL / @djgagne
• Katie Dagon / CGD / @katiedagon
• Curtis Walker / RAL / @curtiswalker
•  Negin Sobhani/ CISL/ @negin513

33 attendees

Agenda:

• ESDS updates

• Upcoming ESDS Forums

• March 17 - OpenVISUS for scientific visualization
• March 31 - Enabling a Science Gateway at NCAR

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• SEA ISS 2025 - 7-10 April, 2025 - registration open

• Presentation:

• Title: Research Data Commons
• Presenters: Doug Schuster, Harsha Hampapura, and Riley Conroy
• Slides: Presentation 
• Additional resources / links:
• Notes:

• Part I:

• Current state:

• Current data services: data.ucar.edu
• Legacy download and analyze model
• Challenges with search
• Difficulty accessing data

• What’s next:

• Pivoting to NSF NCAR’s Integrated RDC
• Leveraging on-prem cloud
• AI/ML testbed
• Gather user needs
• Evaluate VRE platforms
• Consolidate CISL data repos
• Enhance data mgmt and discovery software
• Build and deploy AR/AI-optimized datasets

• Dashboard demo: https://ncar.nationalsciencedatafabric.org/neon-demo/v1
• Integrate with CI initiatives:

• Open Science Data Federation

• Data Commons Workshop Report - http://dx.doi.org/10.5065/gfbq-8y08

• Part II:

• ARCO data

• What is it?

• Analysis ready
• Cloud optimized - or rather optimized for scalable and performant access

• Current

• Several options, but generally a download and analyze model

• Data formats

• Zarr
• Kerchunk / VirtualiZarr

• Catalogs

• Intake
• Intake-ESM

• Demo w/ CESM2 LENS data https://github.com/NCAR/osdf_examples/blob/main/notebooks/cesm_gmst_ncar.ipynb.
• What data is this available for?

• CESM LENS right now
• Many more are coming soon!
• Contact Harsha or Doug

• Documentation coming soon as well - some SIParCS 2025 work on this

February 3, 2025

Sign-In:

Name / Lab / Division / @GitHub Handle:

•  David Ahijevych@ahijevyc
•  John Clyne / CISL / @clyne
•  Thomas Martin / NSF Unidata / @ThomasMGeo
• Nick Cote / CISL/HPCD / @NicholasCote
•  Harsha Hampapura / CISL/ @hrhampapura
• Katelyn FitzGerald / CISL / @kafitzgerald
• Negin Sobhani / CISL / CSG / @negin513
• Mya Sears / EOL / @myasea8
• Mike Levy / CGD / OS / @mnlevy1981
• Julia Kent / CISL / @jukent
• Daniel Howard
• Allison Baker / CISL / allibco
• Doug Schuster / CISL / @dcschus
• Brian Bonnlander / CISL / @bonnland
• Brian Vanderwende / CISL / @vanderwb
• Anissa Zacharias / CISL / @anissa111

17 attendees

Agenda:

• ESDS updates

• Welcome to our new ESDS Forum co-organizer Harsha Hampapura!
• Putting together our spring 2025 schedule - sign up and/or reach out with suggestions, questions, etc.

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Thomas M - Secured PSIF around AI/ML Training for the university community, also more noises around internal AI/ML training. Feel free to get in touch.
• Daniel H. - Saw a cool interface at AMS for interacting with data products of various weather/climate AI models https://aiweather.cira.colostate.edu/ 
• John C. - WCRP Digital Earths - Global Hackathon

• We are excited to announce pre-registration for the WCRP km-scale model hackathon, May 12-16th (US Time), at nodes around the world. US nodes include Princeton, NCAR and Berkeley. Gain experience working with km-scale global and regional climate and weather model outputs
• Please fill out this short interest form to pre-register.
• If you have questions, please visit the hackathon website or contact John Clyne (clyne@ucar.edu), Brian Medeiros (brianpm@ucar.edu), or Julia Kukulies (kukulies@ucar.edu)   for general questions.

• John C. - What are your AI needs? CISL wants to know?

• Access to tools, staff training, access to AI-ready data, etc.?
• Share your thoughts: clyne@ucar.edu

• GeoCAT - materials available from AMS short course on data visualization in Python

• Updated Project Pythia Cookbooks
• Overview slides
• Resources - cheat sheets, galleries, etc.
• Contact: geocat@ucar.edu

• Brian V - New HPC JupyterHub language kernels

• Now: NPL-2025a, R-4.4; coming soon: Matlab R2024b, IDL 9.1.0, Julia 1.11.2

• New Project Pythia Cookboooks 🎉 - https://cookbooks.projectpythia.org/

•  EOFs Cookbook
• Ocean Biogeochemistry Cookbook
• Investigating interhemispheric precipitation changes over the past millennium
• ESGF Cookbook
• Wavelet Cookbook

October 28, 2024

Sign-In:

Name / Organization (and LCPO if relevant):

•  
• Katelyn FitzGerald / CISL
• John Clyne / CISL/TDD
• Anissa Zacharias / CISL / TDD
• Steve Yeager / CGD / @sgyeager
• Brian Vanderwende / CISL / HPCD
• Philip Chmielowiec / CISL/ TDD
• Negin Sobhani / CISL/ CSG
• Ana Victoria Espinoza / NSF Unidata
• Teagan King / CGD / @teaganking
• David Ahijevych / MMM
• Nick Cote / CISL/TDD / @NicholasCote
• Daniel Howard / CISL / HPCD
• Harsha Hampapura/ CISL/ DECS/ @hrhampapura
• Julia Kent / CISL / @jukent
• Mike Levy / CGD
• Wayne Chuang / Columbia LEAP / wkchuang
• Thomas Martin / Unidata / ThomasMGeo
• Joe Tribbia /CGD/AMP @tribbia
• Katie Dagon / CGD / @katiedagon
• Orhan Eroglu / CISL / @erogluorhan
• Susan Stringer / EOL
• Michael Waxmonsky / CGD/CISL

33 attendees

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• November 11th - ESPAT and ESDS
• November 25th - Interactive Data Visualization

• Looking for a new ESDS Forum Organizer
•  

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Pangeo Showcase
• Xarray v2024.10.0 release - DataTree
• CISL at SC24 
• CISL Cyberinfrastructure: Empowering Earth System Science Research (AKA "CISL Roadshows")

• Learn about CISL’s resources, services, and products in support of Earth System Science
• Next stop: CGD, Tuesday, Oct 29 (tomorrow), 1pm, Mesa Main Seminar Room.
• CGD Event info here.
• Complete details on the Road Show website.

• Presentation #1:

• Title: GeoCAT Applications
• Speaker: Anissa Zacharias
• Slides: GeoCAT-Applications-20241028.pdf
• Additional resources / links:

• GeoCAT - https://geocat.ucar.edu/ (major updates coming soon)
• GeoCAT Applications

• GitHub repository: https://github.com/NCAR/geocat-applications
• Website: https://ncar.github.io/geocat-applications/

• NCL Applications: https://www.ncl.ucar.edu/Applications/ 
• GeoCAT suggestion box (we would love to hear from you!)

• Discussion / suggestions:

• Presentation #2:

• Title: CISL On-Prem Cloud Update (CIRRUS)
• Speaker: Nick Cote
• Slides: ESDS CIRRUS Update - 10/28/24.pptx
• Additional resources / links:

• More workshops to come!

September 30, 2024

Sign-In:

Name / Lab / Division / @GitHub Handle:

• Katelyn FitzGerald / CISL / TDD / @kafitzgerald  
• Joseph Gum / CISL / ISD / @asx-
• Anissa Zacharias / CISL / TDD / @anissa111
• Julia Kent / CISL / TDDD / @jukent
• Holly Olivarez / CPAESS NOAA C&GC Postdoc / @holivarez
• Nick Cote / CISL/HPCD / @NicholasCote
• Cora Schneck / CISL / TDD / @cyschneck
• Joe Tribbia /CGD/AMP/ @tribbia
• Meg Fowler / CGD / @megandevlan
• Harsha Hampapura / CISL/ ISD/ @hrhampapura
• Thomas Martin / NSF Unidata / @ThomasMGeo
• Sam Rabin / CGD / @samsrabin
• Paul Prestopnik / RAL /@prestopUCAR
• Alice DuVivier/CGD/@duvivier
• Daniel Howard / CISL / CSG
• John Clyne / CISL / @clyne

24 attendees

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• October 28th - GeoCAT and on-prem cloud updates
• November 11th - ESPAT and ESDS

• Looking for a new ESDS Forum Organizer

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Upcoming Pangeo Showcase Presentations

• Project Pythia Cookbooks: challenges and progress in breaking down barriers to Open Science - Wednesday, October 02, 2024 at 10 AM MDT
• Full list of presentations for the fall session

• Presentation:

• Title: Indigenous Perspectives on Data Management: Rising Voices Changing Coasts Geosciences Hub
• Speaker: Wai Allen, Postdoctoral Researcher at the Haskell Foundation and soon to be Assistant Professor at Arizona State University (January 2025)
• Notes:

• Indigenous communities should not be seen as “broader impacts”
• Transition from western science to spotlighting Indigenous knowledges
• Indigenous perspective on geology

• Early mapping and mineral discoveries that led to land dispossession and boundary drawing
• Specimens taken w/out permission
• Culturally sensitive regions
• Ask: What is meant by whatever statements are being made?
• Who benefits?

• RVCC - NSF funded project

• Largest grant to a tribal university
• Collab w/ NSF NCAR

• Climate adaptation in Indigenous communities

• Recorded over multiple generations
• Climate and societal impacts

• Convergence science

• Interfaces between Indigenous communities and western science
• Co-production
• How do we do this?
• Considerations for data management

• Data sovereignty
• Do repositories and program constraints support the needs and concerns??

• Navigating meaningful engagement with Indigenous communities

• Positionality

• Power
• Identities
• Bias
• Requires critical self reflection

• Relationship building and trust

• What about after the end of the grant cycle?

• A lot of important historical context (e.g. colonial, resource extraction, prior interactions w/ researchers)
• Research legacy impacts
• Recognition
• Differences between communities and cultures
• Timing
• Training needed for scientists to engage w/ Indigenous communities
• Ethical considerations

• Data management in the RVCC

• Types of data: educational data from internships, storytelling / oral history, observational data, interviews, video / audio
• Accessibility w/ permissions for communities to decide what is shared
• Community access to model output data (important one for NSF NCAR)
• FOIA w/ federally funded projects
• FAIR and CARE (still early - much discussion on what this looks like in practice) principles

• Realities of working with Indigenous Communities

• Time
• Capacity
• Data
• Questions

• What types of recommendations can the RVCC offer to entities like NSF?

• Currently it’s challenging to interface w/ organizations like NSF - e.g. capacity building needed, differences in timelines / priorities / recognition

• Are there types of training that can come from lessons learned with the RVCC?

• Future directions

• Design database to hold diverse datasets that uphold CARE and FAIR
• Working toward long term data storage solution

• Will be in Boulder Oct 15 - 18th
• wallen@haskellfoundation.org 

• Links and resources:

• Rising Voices, Changing Coasts - The National Indigenous and Earth Sciences Convergence Hub

• Discussion / Q&A:

• What do you think might be the biggest change / need?

• Funding +
• Still a lot to be done on the NSF side of things in order build the capacity to better engage w/ communities

• Need to be talking with the appropriate people
• Need for advisory panels
• More familiar w/ some communities than others

• Positive examples to model?

• Building trust

• Encouragement to think about how we can use our positions to support Indigenous scientists in academia

• Any additional ways to do this or things to reflect upon?

• Promote Indigenous scientists’ publications, talks, social media posts, etc.
• Attend events hosted by Indigenous scientists (even if they are not science-related in name)
• Ask Indigenous scientists how you may support them: We can offer a lot of behind-the-scenes work/research and provide it to Indigenous scientists so they may continue on their path in their own relationships with Indigenous communities

• Should indigenous communities trust in storing their data at western research institutions or building their own or a mixed approach?

• Likely a mixed approach needed
• Still some (especially technical) capacity building needed, but also trust and access control needs to be built / addressed by western research institutions (e.g. flagging data as culturally sensitive)
• Trust - importance of continuity and communication

• How may we support you, Dr. Allen?

• Group at OSU working on effectiveness of executive orders wrt Indigenous communities (policy evaluation)
• CARE principles and fossils / specimens
• Geochronology and how it could be more culturally sensitive (would like to see an inventory and more transparency) - acknowledgement as a first step

Sept 16, 2024

Sign-In:

Name / Lab / Division / @GitHub Handle:

•  Katelyn FitzGerald / CISL
• John Clyne / CISL
• Steve Yeager/CGD/@sgyeager
• Harsha Hampapura / CISL / @hrhampapura
• Michael Levy / CGD / @mnlevy1981
• Anissa Zacharias / CISL / @anissa111
• Jo Tribbia /CGD/AMP @tribbia
• Nick Cote / CISL/TDD / @NicholasCote
• David Ahijevych/MMM David Ahijevych
• Julie Prestopnik / RAL/JNT/DTC / @jprestop
• Joseph Gum / CISL/ISD / @asx-
• Allison Baker / CISL / @allibco
• Minna Win /RAL/DTC/JNT/@bikegeek
• Tom Cram / CISL/ISD / @tcram
• Bob Dattore / CISL/ISD / @rda-dattore
• Wayne Chuang / LEAP / CGD / @wkchuang
• Brian Medeiros / CGD / @brianpm
• Curtis Walker / RAL/EdEC / @curtiswalker
• John Halley Gowtay / RAL/DTC John Halley Gotway

30 attendees

Agenda:

• ESDS announcements:

• Welcome back!
• Upcoming ESDS Forums

• September 30th - Wai Allen on indigenous data sovereignty
• October 28th - GeoCAT updates

• Looking for a new ESDS Forum co-organizer

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Teagan King (not present today) - The NSF NCAR Data Stewardship Engineering Team (DSET) is trying to figure out where data is coming from and going to around NCAR in order to gauge current priorities for organizational data services. Help us by filling out this 2-minute survey before September 20th and sharing how you use and archive data!
• John Clyne - The CISL on-prem research cloud is transitioning to operations!

• “Soft opening” (NCAR only) targeted for October
• Initial services are TBD
• Nick Cote will continue to lead the effort.

• John Clyne - CISL Road Show, coming to your LCPO soon!

• Presentation:

• Title: NCAR archiving with Zenodo
• Speaker(s): Joseph Gum
• Slides: ESDS 202409 - Archiving with Zenodo
• Notes:

• Zenodo - “Zenodo is a general-purpose repository for research outputs that can't be stored in discipline-specific repositories. It's open to researchers from all over the world and from any discipline.”
• Can still submit to Zenodo yourself
• Zenodo community

• https://zenodo.org/communities/nsfncar/
• Only an NSF NCAR Zenodo community so far
• Submitting to the community is opt-in
• Working on the file size limit (should be possible and happening soon - maybe next Monday)

• Current 100 files / 50GB
• Next 100 files / 200GB

• Plug-in system in development
• Sundog page coming soon
• Process

• Submitting to the community is opt-in and done via the web
• Once submitted, curators are notified to approve/edit the submission

• Record will be visible if you created it outside of the community while waiting for approval
• Multiple types of communities in Zenodo (e.g. you can be part of both a project and community at the same time)

• Example: FastEddy-model Software Release

• DASH - submission site

• Dataset submission resource
• GDEX

• Not accepting new submissions to GDEX moving forward
• Will be migrating data from GDEX likely to Zenodo or RDA
• DOIs will be migrated as well

• Discussion / Q&A:

• Feel free to drop any questions / discussion items here w/ your name or anonymous

• <name> - <question / item>
• Should we list datasets in multiple organizations?

• For example there is also a Developmental Testbed Center community on Zenodo
• Suggestion is to leave as is and add additional as possible
• May be some rough edges e.g. where only one community can be the main / headlining community

• Usage for software vs datasets at NSF NCAR?

• Guess - likely more software DOIs at the moment
• Working on integrating from Zenodo to the library holdings for software DOIs as well

July 22, 2024

Sign-In:

Name / Lab / Division / @GitHub Handle:

• Negin Sobhani/ CISL /CSG @negin513
• John Clyne / CISL/TDD / @clyne
• Katelyn FitzGerald / CISL / @kafitzgerald
• Ben Kirk / CISL / CSG @benkirk
• Julia Kent / CISL / @jukent
• Sam Levis / CGD/TSS / @slevis-lmwg
• Lev Romashkov / CGD / @rmshkv
• Nick Cote / CISL/TDD / @NicholasCote
• Thomaas Martin / Unidata / @ThomasMGeo
• Bob Dattore / CISL / ISD / @rda-dattore
• Brian Bonnlander / CISL / ISD / @bonnland
• Michael Levy / CGD / OS / @mnlevy1981
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Joe Tribbia /CGD/AMP/ @tribbia
• David Ahijevych/MMM/ @ahijevyc
• Ryan Sobash / MMM / @rsobash
• Orhan Eroglu / CISL / @erogluorhan

21 attendees

Agenda:

• ESDS announcements:

• Forum organizer role opening! See description here: ESDS Forum Organizer - Position Ad, and contact Katelyn (katelynw@ucar.edu) or Lev (eromashkova@ucar.edu) with interest
• Workshop:

• From Jupyter Notebook to Web Server: Automating Container Deployments with GitHub Actions (Part II)

• August 8, 2024 from 9:00a to noon MDT at the Mesa Lab and virtually
• Registration closes August 4th

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Joseph Gum - Hack the Hackathon workshop registration is open for November 18-22 in San Diego, closes August 23 - if you’d like to connect with other hackathon, code sprint, cook-off, etc organizers and researchers and find out about how they run their events please apply! Registration is free and there is travel support available, pending final disbursement from NASA.
•  

• Presentation:

• Title: AWSIG - Casper Augmentation
• Speaker(s): Ben Kirk & Brian Vanderwende
• Slides: 2024-07-22 Casper Status
• Notes:

• Recent software environment updates

• New modules on systems and JupyterHub kernels

• Updated MATLAB, Julia, IDL

• New NPL environment - 2024b

• Still Python 3.11 but many package update, next version will probably have Python 3.12

• Casper hardware refresh status

• Components selected in response to previous Casper user survey
• 3 new hardware classes that have already been procured and integrated
• Final addition planned later this year, requiring HPC systems outage in August (8/12-8/15)
• Legacy components will remain available, but are beyond supported lifespan from vendors. Will be “run to failure” so there will be some fluctuation in availability over next few years
• Specialty nodes

• Added 6x GPU visualization nodes
• Added 2x H100 GPU nodes
• Added 6x high memory data analysis nodes

• Later this year, adding 64x high throughput data analysis nodes, nearly doubling Casper’s HTC core count

• Accessing different types of resources

• Accessing L40 visualization nodes: Use VNC desktop https://ncar-hpc-docs.readthedocs.io/en/latest/compute-systems/casper/remote-desktops 
• Accessing H100 GPGPU nodes: Accessed similarly to other GPGPU resources https://ncar-hpc-docs.readthedocs.io/en/latest/compute-systems/casper/starting-casper-jobs 
• Accessing large memory nodes: Similar to typical HTC nodes (qsub or PBS job script), but must request >361 GB of memory

• Best practices for monitoring memory usage: https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/checking-memory-use 

• Ongoing HPC systems and storage maintenance

• Hardware replacements and file system reconfigurations for /glade/campaign and /glade/work - should increase interactive responsiveness and modestly increase capacity
• Systems down for maintenance August 12-15

• Expansion and updates to Bifrost network to prepare for addition of Casper nodes later this year
• Glade, Casper, and Derecho will be down for first 2-3 days of outage, progress dependent

• Resources:

• https://ncar-hpc-docs.readthedocs.io/en/latest/compute-systems/casper/remote-desktops 
• https://ncar-hpc-docs.readthedocs.io/en/latest/compute-systems/casper/starting-casper-jobs 
• https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/checking-memory-use 

• Discussion:

June 24, 2024

Sign-In:

Name / Lab / Division / @GitHub Handle:

• Lev Romashkov / CGD/OS / @rmshkv
• Katelyn FitzGerald / CISL/TDD / @kafitzgerald
• Kevin Sampson / RAL HAP / @kmsampson
• Nick Cote / CISL / @NicholasCote
• Brian Bonnlander / CISL/ISD / @bonnland
• Orhan Eroglu / CISL/TDD / @erogluorhan
• Ana Victoria Espinoza / NSF Unidata / @ana-v-espinoza
• Allison Baker / CSIL/TDD / @allibco
• Kirsten Mayer / CGD / CCR / @kjmayer
• Tracy Hertneky / RAL/JNT / @hertneky
• John Clyne / CISL / @clyne
• David John Gagne / CISL / @djgagne
• Negin Sobhani / CISL/CSG / @negin513
• Sheri Voelz / CISL / @mickelso
• Julia Kent / CISL / @jukent
• Rachel Tam / CISL / @rytam2
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Brenda Javornik / EOL / RSF / @leavesntwigs
• Thomas Martin / Unidata / @thomasMgeo
• Wayne Chuang / CGD / LEAP / @wkchuang
• Kwesi Quagraine/ACOM/CGD/ @akumenyi
• Ryan Sobash / MMM / @rsobash

31 Attendees

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums:

• July 22nd - Ben Kirk and AWSIG team presenting on Casper 2024 Augmentation Status, and general AWSIG (Analysis Workflow Special Interest Group) update

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• SciPy 2024 - July 8 - 14th
• Orhan - Joint WRF/ MPAS workshop this week (starting tomorrow)

• Agenda
• UXarray talk Tuesday, mini UXarray tutorial Friday (UXarray: unstructured grids Python package)

• Presentation:

• Title: MILES ML Software
• Speaker: David John (DJ) Gagne
• Slides: Gagne_esds_ml_2024-06-24
• Notes:

• MILES: Machine Integrations and Learning for Earth Systems

• Work with almost all parts of NCAR/UCAR

• Motivation

• AI for Earth system science advancing rapidly
• AI models all have own biases and artifacts

• Distributed quantile scaling

• Not all important quantities are Gaussian
• How do we transform non-Gaussian variables into exactly Gaussian distributions for ML? Quantile transform

• Serial quantile transform
• T-Digest - Compressed, distributed quantile transform

• Distributed quantile scaling has been applied to ERA5 data (3 hrs, 2048 CPUs)
• Bridgescaler - standardized way to rescale both tabular and deep data in serial or distributed patterns

• AI numerical weather prediction

• Goals:

• Extremely fast global + regional atmospheric predictions with little cost
• Advances in forecast skill
• Quickly conduct previously infeasible simulations
• Low-latency interaction w/ running simulations for coupling + counterfactual scenarios

• Various benefits of NCAR-supported AI atmosphere model versus relying on externals (private sector or foreign partners)
• Training data

• ERA5 reanalysis

• WXformer - NCAR’s first digital twin
• Current AI weather prediction groups have released model weights, but not code to train at scale - NCAR developing open framework to both train and run global + regional models
• Goal for initial release summer 2024 (reach out if interested)
• ECHO: hyperparameter optimization

• Distributed hyperparameter optimization on HPC systems

• Integrating Python-based AI with Fortran weather and climate models

• Major bottleneck has been running AI/ML models in Fortran environments

• MILES-GUESS: Machine learning uncertainty quantification

• Machine learning uncertainty quantification with evidential and ensemble models
• Paper

• Hagelslag: Scalable object-based data analysis and evaluation

• Python weather blob segmentation [...]

• Resources:

• NSF NCAR MILES group website
• Software repositories on GitHub:

• bridgescaler
• echo-opt
• miles-guess
• hagelslag

• Data:

• https://mldata.pangeo.io/
• https://www.ametsoc.org/index.cfm/stac/committees/committee-on-artificial-intelligence-applications-to-environmental-science/ai-datasets/
• https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim

• Discussion:

• <feel free to jot down questions or topics here>
• Potential topics:

• Experience with AI/ML integration with Fortran models
• Unmet needs for ML software at NSF NCAR
• Interest in integrating into your workflows and points of friction
• What other software are people using?
• What are the community needs with respect to ML software?

• Any challenges with using NCAR data?

• Not a lot of access issues
• Depends upon the dataset
• Some work required to preprocess and transform it
• Needs from the broader ML community in terms of understanding weather / climate data (requires a lot of domain expertise)

• What are some of the biggest sources of friction in supporting a broader user community, encouraging participation, etc? Any generic tools or support that would help?

• Supporting people’s different Python environments and diagnosing issues that come up, esp. with dependencies
• When working with domain-oriented collaborators, not everybody is as experienced with tools like Github
• Hard to support documentation as a small group, hard to outsource

• Pros and cons of tensorflow vs pytorch?

• MILES group uses both depending on project, moving more towards pytorch
• Also using Keras 3
• Easier to use custom neural networks with pytorch
• Recommend pytorch or Keras 3 for starting a new project

• Any plans to make data-driven forecasts on longer timescales than weather (S2S, decadal)? If so, how do you plan to do this with limited reanalysis?

• S2S definitely feasible, question is more how to make the predictions useful, need more components (land, ocean) coupled

• Opinion on physics-informed neural networks?

• Lots of other ways to incorporate physics
• Work better for idealized problems than larger-scale, has seen pretty mixed performance

• Are more “traditional” modeling groups (CESM?) welcoming these tools?

• Interest definitely exists, difficulties with compatibility, cultures around different tools (pace of change), concerns around support

• What challenges are folks experiencing?

• Time and pace of change (keeping up)
•  

• What AI/ML software are folks using?

• Data:

• https://mldata.pangeo.io/
• https://www.ametsoc.org/index.cfm/stac/committees/committee-on-artificial-intelligence-applications-to-environmental-science/ai-datasets/
• https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim

• Where to stay tuned for updates:

• https://www.cisl.ucar.edu/miles
• https://twitter.com/djgagnedos
• https://bsky.app/profile/djgagnedos.bsky.social

May 13, 2024

Sign-In:

Name / Lab /  Division / @Github Handle:

• Lev Romashkov/ CGD/OS / @rmshkv
• Negin Sobhani/ CISL/CSG / @negin513
• Ana V. Espinoza / NSF Unidata / @ana-v-espinoza
• Joseph Gum / CISL/ISD
• Joe Tribbia /CGD/AMP @tribbia
• Gary Strand / CGD / CESM / @strandwg
• Kirsten Mayer / CGD / CCR / @kjmayer
• Bob Dattore / CISL / ISD / @rda-dattore
• Matt Mayernik / Library / @mayernik
• Nick Cote / CISL / @NicholasCote
• Brian Bonnlander / CISL/ISD / @bonnland
• Katie Dagon / CGD / @katiedagon
• Riley Conroy / CISL / @rpconroy
• Anissa Zacharias / CISL / @anissa111
• Jiang Zhu / CGD / @jiang-zhu
• Tom Cram / CISL / @tcram
• George McCabe / RAL AAP / georgemccabe
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Seth McGinnis / RAL / @sethmcg
• Curtis Walker / RAL / @curtiswalker
• Doug Schuster / CISL / @dcschus
• Kwesi Quagraine ACOM/CGD/ @akumenyi
• Wayne Chuang / CGD / LEAP / @wkchuang
• Daniel Howard / CISL HPCD / @dphow
• Julie Prestopnik / RAL JNT/ @jprestop
• Paul Prestopnik / RAL AAP / @prestopUCAR
• Katelyn FitzGerald / CISL / @kafitzgerald
• Daniel Adriaansen / RAL / AAP / @DanielAdriaansen
• Harsha Hampapura/ CISL/ @hrhampapura
• David John Gagne / CISL / @djgagne

40 Attendees

Agenda:

• ESDS announcements:

• No forum on Memorial Day (May 27)
• ESDS forum:

• June 24th, David John Gagne presenting on MILES Machine Learning Software

• ESDS workshop:

• June 20th (9a - noon) - From Jupyter Notebook to Web Server: Containerizing Interactive Visualizations - more details coming soon

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Pangeo Showcase:

• May 15th - Tom Nicholas - VirtualiZarr: Create virtual Zarr stores using xarray syntax
• May 22nd - Joe Hamman - Zarr-Python 3 and why you should be excited!

• Presentation:

• Title: FAIR Data, Data Repositories, and Data Citation
• Speaker: Matt Mayernik and Bob Dattore
• Slides:

• Matt slides
• Bob slides

• Notes:

• Repositories and FAIR

• Findable - persistent identifiers (DOI), metadata
• Accessible - open access, standard access protocols
• Interoperable - common file formats, links to related resources
• Reusable - long term preservation, open copyright licenses

• NCAR dataset submission

• 
• All DMP requests should go through DASH Jira system (linked below)
• Will help evaluate info from PIs to determine best repository choice
• No-cost storage for up to 1 TB (supported by CISL)
• Able to help with DMPs for all proposals, not just those on NCAR systems
• DMP budget should be done at project proposal stage, otherwise costs will be higher

• Thoughts on different systems

• Github

• Strengths: collaboration, version tracking, distribution, ticketing
• Weaknesses: preservation (Github is for-profit, owned by Microsoft, no guarantee it’ll stick around), discovery, consistency

• Zenodo

• NCAR supported
• Better awareness of data and software input
• Include Zenodo-based resources in citation compilations

• Leveraging DOIs for discovery

• Datasets with DOIs allow you to track which papers cite them

• Tracking citations of UCAR assets

• Crossref
• Scopus (Elsevier), Web of Science
• Working on Google Scholar
• Citation finder runs as cron task 2x/month
• https://github.com/NCAR/rda-citefind
• Crossref, Scopus, Web of Science

• Pros: well-developed APIs, high-quality peer reviewed results
• Cons: Miss in-text citations that include title or URL but not DOI - especially relevant to older works

• Google Scholar finds these datasets that other services wouldn’t pick up, but difficult to automate. Working on adjusting NASA’s methods to work for NCAR/UCAR.

• Any API feedback is welcome - try checking for DOIs that you own
• Questions for the group:

• Where do you archive your data? Software?
• What frustrations or challenges do you encounter with regard to data or software archiving?
• How do you do citation tracking for data or software? What challenges have you encountered?

• Instructions for requesting to be a part of the NSF NCAR Community:

• If you are the owner of the Zenodo record, you should see “Communities” as a section on the right hand side.
• Click on the gear and select “Submit to community”
• Search for “NSF National Center for Atmospheric Research” and click “Select” next to the community once that community is visible.
• Note that curator members of the community will have access to view and edit your upload’s metadata and files.
• If you are sure you would like to join the community, check the box and then click “Submit to Community”, adding a message to the curators with any information you’d like to share or any questions you have (please include your email address).
• It looks like here are the official instructions with images and all: https://help.zenodo.org/docs/share/submit-to-community/#submit

• Resources:

• General

• FAIR Principles
• AGU Data & Software for Authors

• NSF NCAR

• NSF NCAR Data Stewardship Engineering Team (DSET)
• NCAR Dataset Submission Request System
• NCAR Github
• NCAR Zenodo

• UCAR

• UCAR Citation API Guide

• Discussion

• <name / anon> - question
• Zenodo - is there a way to link existing items into the community?

• Not entirely sure, but may be able to request to do so
• Julie Prestopnik (who set this up originally)  will check and add to notes
• Possibly these instructions: https://help.zenodo.org/docs/share/submit-to-community/#submit 

• What should we be thinking about in terms of tracking citations and impact?

• Who do we want to be reaching?
• What do we want to know?
• Used at the high level (org level) to understand data usage

• Site visits and other
• Asked about usage stats, but downloads are problematic

• Internally

• Could help w/ search as well eventually
• Could think about using for data retention of model outputs

• Potential use at the individual or project level
• Interest at the lab level
• Haven’t heard of targets so far
• EOL and CISL have worked on integrating the tool presented into resources to show citation information

• What’s the turnaround time when you submit a ticket for help with a DMP?

• If no-cost request (<1 TB). within the day, and will provide boilerplate template
• Larger model output that might go to RDA - first consult with Doug and determine budget, 1-3 day timeframe
• Don’t wait til the last minute, but the turnaround is pretty quick!

April 29, 2024

Sign-In:

Name / Lab /  Division / @Github Handle:

• Katelyn FitzGerald / CISL / @kafitzgerald
• Allison Baker / CISL/ @allibco
• Negin Sobhani / CISL / @negin513
• Harsha Hampapura / CISL/ @hrhampapura
• Kirsten Mayer / CGD / @kjmayer
• Benjamin Gaubert / ACOM
• John Clyne /CISL / @clyne
• Doug Schuster / CISL / @dcschus
• Kevin Sampson / RAL / @kmsampson
• Nick Cote / CISL / @NicholasCote
• Keith Lindsay / CGD / OS / @klindsay28
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Anissa Zacharias / CISL / @anissa111
• Teagan King / CGD / @teaganking
• Dave Stepaniak / CISL / David Stepaniak 
• Katie Dagon / CGD / @katiedagon
• Thomas Martin / NSF Unidata / @ThomasMGeo
• Joe Tribbia /CGD/AMP/ @tribbia
• Julius Busecke/ LDEO/@jbusecke

34 Attendees

Agenda:

• ESDS announcements:

• Next ESDS forum: May 13th, Matt Mayernik and Bob Dattore presenting on Data management and data citation tracking
• No forum on Memorial Day (May 27)

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
•  Thomas Martin - Community ML Education Module: https://elearning.unidata.ucar.edu/enrol/index.php?id=12        
• April 30th, MathWorks Tech Session - Artificial Intelligence with MATLAB & Python Interoperability - for NSF NCAR & UCAR staff only
• 2024 Project Pythia Cook-off, June 11-14th (registration closes May 26)
• Negin Sobhani - Sign up or suggest ideas for future NHUG meetings
• Pangeo Showcase:

• April 24th - Tom Nicholas - VirtualiZarr: Create virtual Zarr stores using Xarray syntax
• May 1st - Max Jones - Pangeo ML: Open source tools and pipelines for cloud-optimized machine learning

• Presentation:

• Title: Analysis-Ready Cloud-Optimized Data for your community and the entire world with Pangeo-Forge
• Speaker: Julius Busecke
• Slides: https://speakerdeck.com/jbusecke/analysis-ready-cloud-optimized-data-for-your-community-and-the-entire-world-with-pangeo-forge
• Notes:

• Involved in:

• LEAP
• M2LInES

• Why open science?

• Agile science - speed counts! (how do we iterate faster?)

• Understanding         limited portion

• Needs:

• Collaboration
• Reproducibility

• Tech / infrastructure limited portion

• Needs:

• Open + fast data access
• Community OSS tools
• Infrastructure support

• Open data as the bedrock of open science
• More inclusive science!

• Bootcamps
• academy.climatematch.io

• Greater need for climate data from broader communities (including the private sector)
• Positive feedback re: CMIP data in the cloud
• Key ingredients: ingest, storage (cloud-like), discovery (still a lot of need / work to be done here)
• “Collaborative science means community hopping”

• Portability of tools is key
• People move around and have common needs across different institutions / communities - communities are “semi-permeable”

• Future vision

• Separation between data and compute
• Public access more important than compute speed
• Decrease “time to first plot”
• Increased collaboration across communities

• LEAP-Pangeo

• “Cloud platform”

• Data (LEAP ingested + produced)

• Community bakery via Pangeo-Forge (recipes reusable)

• https://github.com/leap-stc/metaflux_feedstock 

• Compute
• Interface - JupyterHub
• Discovery - LEAP Hub

• CMIP6 recipe

• Intake ESM to read the files produced

• Dynamic chunking

• “StoreToZarr” accepts a custom function for chunking

• Lessons learned:

• Data ingest is hard work, but incredibly worthwhile (relying on short term grant funding risky)
• It’s a good time to get involved with pangeo-forge
• Not dependent upon commercial cloud, but rather cloud-like storage

• Resources:

• https://github.com/jbusecke/presentation_wcrp_open_science_conference
• SciPy presentation
• Pangeo/ESGF CMIP6 Zarr data. I would love to get feedback or requests for new data.

• Discussion

• <name / anon> - question
• S3 cloud object storage really seems to be what folks are moving to
• What is needed?

• Funding and…
• Steady support for maintenance (especially) and development (engineering time)
• Ambitious vision where folks can align and be inspired to move toward and rewarded for doing so

• Guided by open community that isn’t tied to a specific institution

• How do we better recognize open science efforts?

• Traditional publication process / incentives are limiting
• Make it easier to do the “right thing” for both producers and consumers (reduce friction)

• Commercial vs on-prem storage

• Can be both!
• We need “fast enough” not necessarily faster and faster
• Egress a huge problem (difficult to calculate / know cost)
• Excited about Open Storage Network approach

• How does this change w/ CMIP7?

• Advocating for streaming access vs download and analyze

• ESGF roadmap issue. Voicing support here would be great.
• We are also prototyping a Stac endpoint 

• Probably not ideal to convert to Zarr stores for a project like this for now

• Please check in this repo for progress on the prototype for virtual zarr ingestion.

• Excited about virtual Zarr / Kerchunk to help w/ this
• Modeling centers can:

• QC the formatting of the data in addition to the metadata
• Data chunking is a big deal (~10-100MB but depends a bit)
• Need to think about implications here (will link some GitHub issues to explore / advocate for)

• Zarr languages / implementations

• Not just Python: https://zarr.dev/implementations/

• Happy to chat more if folks would like to use this at NCAR and/or take further questions

April 15, 2024

Sign-In:

Name / Lab /  Division / @Github Handle:

•  
•  Katelyn FitzFerald / CISL / @kafitzgerald
•  Lev Romashkov / CGD / @rmshkv
• Katie Dagon / CGD / @katiedagon
• David Ahijevych/MMM/@ahijevyc
• Brian Medeiros / CGD / @brianpm
•  Thomas martin / Unidata / @thomasmgeo
•  Harsha Hampapura /CISL/ @hrhampapura
•  Mike Levy / CGD / @mnlevy1981
• Feng Zhu / CGD / @fzhu2e
• Wayne Chuang / CGD / LEAP / @wkchuang
• Teagan King / CGD / @teaganking
• Maryam Abdi-Oskouei/UCP/JCSDA
• Ben Gaubert / ACOM
• Brian Bonnlander / CISL / ISD / @bonnland
• Joe Tribbia /CGD/AMP @tribbia
• Ryan Sobash / MMM / @rsobash
• David John Gagne / CISL / @djgagne
• Kwesi Quagraine/ACOM/ CGD/ @akumenyi

27 Attendees

Agenda:

• ESDS announcements:

• 4/29/24 - ESDS Forum - Julius Busecke, Columbia University - CMIP analysis (title TBD)

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Pangeo Showcase:

• April 24th - Tom Nicholas - VirtualiZarr: Create virtual Zarr stores using Xarray syntax
• May 1st - Max Jones - Pangeo ML: Open source tools and pipelines for cloud-optimized machine learning

• NHUG Meeting:

• April 16th - Using AI in VS Code for Developing Code

• 2024 MetPy User Survey (for users of MetPy)
• 2024 Project Pythia Cook-off, June 11-14th (registration closes May 26)

• Presentation:

• Title: jax/numpy (part 1), Xarray and ML (part 2)
• Speaker: Thomas Martin, Unidata
• Slides: Notebooks here: https://github.com/Unidata/MLscratchpad/tree/main/ESDS_April24 
• Notes:

• Part 1: Faster array manipulations with JAX

• JAX is one of several ways to make Python code faster
• Drop-in replacement for numpy
• Good for if you’re writing a lot of raw numpy code
• Speeds up basic numpy tasks on the order of 40-60%, depending on type of function
• Is it useful with Xarray?

• There’s an example in their resources, they’re working on making it less clunky.
• For now, might be an easier win to just use as a drop-in for numpy alone
• It looks like graphcast has xarray_jax, but it hasn’t been released as a standalone project.  Related GitHub issue: https://github.com/google-deepmind/graphcast/issues/9

• Where is the speedup actually coming from?

• Under the hood, similar to numba but with easier UI

• Array API Standard

• Part 2: Xarray and ML

• Using a TensorFlow data loader or similar will strip out the units that initially came with your CF-compliant dataset
• Can extract units from the initial dataset with a one-liner shown in the notebook and add them back to the ML output

• Xarray full_like fills both values and attributes

• What’s the best way to keep track of this while doing ML work? How do we limit this bookkeeping?

• David A. - Would be really nice if ML packages would handle this more automatically
• Thomas M - Scikit-learn handles this with Pandas dataframes in recent versions
• DJ - [Missed this question, please summarize it here if you want!]

• Katelyn F - Is this something that Unidata is interested in from an educational perspective, or are you interested in contributing to development?

• Thomas M - Unidata is pro-CF and pro-units, so from that perspective we want to minimize the extra work caused by stripping off units

• Katelyn F - What do other people’s ML workflows look like?

• Katie D - Usually see people creating an Xarray dataset after the ML processing.
• Thomas M - Generally lots of intro level resources available online, with solutions to some of these problems, but not for very large datasets
• KD - Dask challenges
• KD - GPU usages

• DJ - Maybe JAX could be handy for this - JAX GPU applications

• KF - Connects to array interoperability standard in Python
• TM - Migrating between some of these packages can be easier than you expect, affect speed in some cases

• DJ - But JAX and Pytorch don’t play nicely with each other

• DJ - How many people on call use ML in their daily workflows?

• Around half of attendees

• Katelyn F - People doing work to make data more FAIR and AI-ready at NCAR, how does that work relate to this?

• TM - Feels like two different worlds, where ML just wants a 3D tensor

• Resources:

• Jax.numpy
• JAX NVIDIA
• Further information: AMS 2024: Flexible Vectorization In Xarray Using JAX
• MetPy & Xarray

• Discussion

April 1, 2024

Sign-In:

Name / Lab /  Division / @GitHub Handle:

• Janine Aquino / EOL / RAF / @janineA
• Orhan Eroglu / CISL / @erogluorhan
• Philip Chmielowiec / CISL / TDD / @philipc2
• Sam Rabin / CGD / TSS / @samsrabin
• Isabel Suhr / EOL ISF / @isabels
• Ana Victoria Espinoza / UCP Unidata / @ana-v-espinoza
• Cora Schneck / CISL / TDD / @cyschneck
• Keith Lindsay / CGD / OS / @klindsay28
• Harsha Hampapura/CISL/ @hrhampapura
• John Clyne / CISL-TDD/ @clyne
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Wayne Chuang / LEAP / @wkchuang
• Bob Dattore / CISL / ISD / @rda-dattore
• Joe Tribbia /CGD/AMP/ @tribbia
• Adrianna Foster / CGD / TSS / @adrifoster
• Elizabeth Homa / RAL @ehoma
• Steve Vahl / UCP / JCSDA / @svahl991
• Drew Camron / NSF Unidata / @dcamron
• Dan Adriaansen / RAL / @DanielAdriaansen
• Amir Mazrooei / RAL / HAP / @amazroo
• Naoki Mizukami / RAL/ @nmizukami

# of attendees: 35

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• April 15th  - Xarray and ML

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Upcoming Pangeo Showcase talks

• April 3rd - Open Radar Stack 

• 2024 MetPy User Survey (for users of MetPy)
• 2024 Project Pythia Cook-off, June 11-14th (registration closes May 26)
• 2024 Improving Scientific Software (ISS) Conference, April 15-18th (virtual registration closes April 12) - includes a UXarray tutorial and many relevant software and data talks

• Presentation:

• Title: VS Code
• Speakers: Adrianna Foster and Sam Rabin
• Slides:
• Notes:

• Linters
• Building, Executing, and Debugging
• Other items

• Custom themes
• Guide bars
• Version control integrations
• Many different panes / views

• Python linting in VS Code - demoed pylint (can specify the version you want)
• Color themes
• “Open preview to the side” for viewing rendered RST files

• Resources:

• Videos

• Adrianna's talk recorded at SEWG meeting: "VS Code and You: Why You Should Make Your Computer Work for You (and How!): https://youtu.be/8CYoJHoN2Gc?feature=shared&t=6824
• Sam’s talk recorded at SEWG meeting (demo of various features): https://www.youtube.com/watch?v=8CYoJHoN2Gc&t=7995s
• Supreeth's talk recorded at SEWG meeting: "Accelerate productivity with Al: Github CoPilot": https://youtu.be/8CYoJHoN2Gc?feature=shared&t=9369

• Tutorials

• basic intro to vscode: https://github.com/ESCOMP/CTSM/discussions/2139
• live docs: https://github.com/ESCOMP/CTSM/discussions/2093
• Avoid having to re-auth over and over: https://nhug.readthedocs.io/en/latest/blog/streamlining-two-factor-authentication-with-ssh/

• Discussion:

• Potential topics:

• Q&A for the presenters
• Resource sharing: e.g. recommendations for extensions, configuration options, useful features, etc.
• Usage questions
• Challenges

• <name/anon> - <question>

• Janine - How did y’all learn all of the cool features of VScode? Just playing? Any suggested references / tutorials?

• Nice tutorial on the CTSM GitHub repository and another (will drop links in the notes)
• A lot of learning from experience / Google
• Built in tutorial and documentation (specific ones for certain languages and types of work)

• Janine - Has anyone run into memory management issues running vscode on a Mac? On my Intel mac, VScode was a hog and I eventually deleted it. I have an M2 now, so maybe not such an issue…?

• Careful with your workspace (maybe open a single folder / project at a time)
• Especially problematic on HPC

• Suggestions for screen and window management?

• Splitting screens
• Can have multiple windows per project as well
• Multiple terminals
• Larger monitor

• Suggestions for debugging MPI codes?

• Yes, but would need to configure another debugger

• What are some of your favorite extensions?  

• GitLens
• CSV Rainbow
• ColorPicker
• VS Code Pets

• What are some of your favorite features?  

• Remote development via SSH (CSG may have some tips)

• What pain points have you encountered?

• Has anyone used Live Share extensively for debugging and/or pair programming?

• Any recommendations for further tutorials to reference?

• VS Code tutorials

• Some good tutorials for different languages, types of work, etc.

• Tutorial from SciPy 2023

• Covers basic use cases, Live Share, Dev containers, remote development, and some scientific python specific recommendations

• Tips for working on HPC:

• https://nhug.readthedocs.io/en/latest/blog/streamlining-two-factor-authentication-with-ssh/ 
• https://nhug.readthedocs.io/en/latest/blog/jupyter-for-vscode/

March 18, 2024

Sign-In:

Name / Lab /  Division / @GitHub Handle:

• Lev Romashkov / CGD/OS / @rmshkv
• Katelyn FitzGerald / CISL / @kafitzgerald
• Negin Sobhani / CISL - CSG/ @negin513
• Hannah Veitel / COSMIC / @huelsing
• John Clyne /CISL / @clyne
• Helen Kershaw /CISL/DAReS / @hkershaw-brown
• Francois Vandenberghe JCSDA/UCP/UCAR @fcvdb
• Ana Espinoza / NFS Unidata / @ana-v-espinoza
• David John Gagne / CISL/MILES / @djgagne
• Anissa Zacharias / CISL / @anissa111
• Mike Levy / CGD / @mnlevy1981
• Doug Schuster / CISL / @dcschus
• Katie Dagon / CGD / @katiedagon
• Brian Bonnlander / CISL / ISD / @bonnland
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Thomas Martin / Unidata / ThomasMGeo
• Isabel Suhr / EOL ISF / @isabels
• Teagan King / CGD/CCR / @teaganking
• Ward Fisher /NSF Unidata (UCP) / @wardf
• Tom Cram / CISL / @tcram
• Riley Conroy / CISL / @rpconroy
• Bill Brown / EOL ISF / wbrown@ucar.edu
• Ryan Sobash / MMM / @rsobash
• Joe Tribbia /CGD/ AMP @tribbia
•  Harsha Hampapura /CISL/ harshah@ucar.edu
• Drew Camron / NSF Unidata / @dcamron

# of attendees:40

Agenda:

• ESDS announcements:

• Monthly ESDS Forums this summer
• Looking for some additional presenters this spring and summer - please reach out to Lev Romashkov and/or Katelyn FitzGerald if you’re interested and / or have suggestions for speakers or topics of interest
• 2024 ESDS Annual Event Recap blog post
• New ESDS YouTube channel with recordings from the event: https://www.youtube.com/@ncar_ucar_esds 

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• John C: Registration now open for the The Pythia Cookbook Cook-Off (hackathon) June 11 - 14, 2024 at the Mesa Lab.

• Contribute to the Project Pythia Cookbook gallery, and improve your scientific python coding skills! All levels of experience are welcome!
• Requests for travel support due March 29
• Registration closes May 26
• There is a $50 registration fee for on-site participants

• Negin S. : SEA - ISS 2023 Registration is open now! https://sea.ucar.edu/conference/2024 ⌨️
• Doug S. Introduce Harsha. 👋
• Negin S. : Tomorrow NHUG Meeting - VS Code presentation 🧑‍💻
• DJ Gagne: MILES-GUESS package for uncertainty quantification 📦

• Presentation:

• Title: MetPy: Your Friendly Community-built Python Toolkit
• Speaker: Drew Camron and Ryan May
• Slides:  2024 ESDS 3/18 MetPy Update & Discussion
• Notes:

• MetPy - Python toolkit for meteorology, based on scientific Python stack
• Core features:

• Support for physical units - required for calculations
• Library of constants, with citations
• Xarray integration - if data is CF-compliant, MetPy has tools to automatically pull out relevant coordinates. New features for more accurate calculations that take into account grid info
• Cartopy integration - automatically determine Cartopy CRS, etc. for easier plotting

• Recent additions:

• Community-contributed new calculations (see slides for a list!)
• Plotting fronts - built on matplotlib
• Standardizing relative humidity definition
• And more, see slides!

• Soon:

• High/low center identification
• Easier text plotting
• dBz recognized as unit
• Performance work and benchmarking (incl. support for Dask arrays)
• Clients for AWS cloud datasets

• MetPy’s community

• MetPy mentioned or cited in almost 300 theses and peer reviewed articles!
• Documentation views + package downloads also tracked
• Want community contributions and involvement!
• Open calls every other Thursday at 11:30 MDT to discuss the project - see calendar

• Resources:

• MetPy documentation
• GitHub repository 
• Unidata Example Gallery
• Pint documentation
• Climate and Forecast Metadata Conventions
• Twitter & Bluesky @MetPy
• Calendar w/ community meetings

• Discussion (feel free to add questions here):

• Doug S - CISL ISD is developing a service to track citation counts (and what works from cited a UCAR DOI) at https://api.rda.ucar.edu/citations/. This may be useful for your "impacts" tracking.  We plan to add mentions, etc captured by google scholar in the near future and this will increase "reference" counts significantly.

• For example, try:  curl https://api.rda.ucar.edu/citations/doi/10.5065/D6WW7G29
• Could be opportunities to collab
• Maybe an ESDS presentation at some point?

• Brian M - MetPy vs GeoCAT

• Different origins

• GEMPAK+ for MetPy
• NCL+ for GeoCAT

• Some collaboration and contributions
• Interested in supporting each other rather than duplicating work
• Orhan E - +1 on this. Shouldn’t be much overlap between the two
• Feedback welcome

• Helen K - Tools for reading and working with observational data?

• Yes, they exist for gridded and other data
• No support for BUFR yet

• Katie D - What led to the implementation of IDing high/low centers? Community need? How are you doing it?

• A collaborator needed it (in a teaching context?)
• Using persistent topology for peak detection
• https://github.com/Unidata/MetPy/pull/3110 

February 5, 2024

Sign-In:

Name / Lab /  Division / @Github Handle (29 in call):

• Lev Romashkov / CGD / OS / @rmshkv
• Katelyn FitzGerald / CISL / TDD / @kafitzgerald
• Heather Carrigan/ UCP / BSS / @hcarrigan
• John Clyne / CISL / TDD, @clyne
• Anissa Zacharias / CISL / TDD / @anissa111
• Cora Schneck / CISL / TDD / @cyschneck
• Mike Levy / CGD / OS / @mnlevy1981
• Allison Baker/ CISL/ TDD/ @allibco
• Julia Kent / CISL / TDD / @jukent
• Brian Vanderwende / CISL / HPCD, @vanderwb
• Katie Dagon / CGD / @katiedagon
• Brian Bonnlander / CISL / ISD / @bonnland
• Ana V. Espinoza / UCP; EODS; NSF Unidata / @ana-v-espinoza
• Joe Tribbia /CGD/AMP @tribbia
• Teagan King / CGD / CCR / @TeaganKing
• Keith Lindsay / CGD / OS / @klindsay28
• Nick Wehrheim / NRIT / @nwehrheim
• George McCabe / RAL/AAP / @georgemccabe
• David John Gagne / CISL / TDD / @djgagne
• Nick Cote / CISL VAST / @NicholasCote
• Amir Mazrooei / RAL / HAP / @amazroo

29 Attendees

Agenda:

• ESDS announcements:

• Upcoming ESDS forums…we have open slots in March!

• Sign up here - ESDS Forum Presentation Signup
• Reach out to Katelyn FitzGerald and Lev Romashkov with suggestions for speakers and/or topics you would like to see!

• Updates from AGU - AGU 2023 Recap - ESDS Forum 2/5/24
• Updates from the ESDS Annual Event - ESDS Forum 2/5/24 - ESDS Event Summary

• Should have recordings posted soon (hoping in the next week or so) and will share in an ESDS blog post!

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Any updates from folks who attended AMS?
• Pangeo Showcase starting back up again on Wednesday

• VAPOR Python API - A 3D data visualization Python package for Earth Science datasets, Nihanth Cherukuru, Feb 28, 2pm MT

• Presentation:

• Title: Benchmarking your Scientific Python Packages Using ASV and GitHub Actions
• Speaker: Anissa Zacharias
• Slides: ASV-ESDS-02052024.pdf
• Notes:

• Resources:

• ASV github, documentation
• ASV commands reference
• Links from presentation:

• geocat-comp

• asv.conf.json
• asv-benchmarking.yml

• geocat-comp-asv

• asv.conf.json
• make-publish.yml

• geocat-comp asv benchmark dashboard

• Discussion

• What are good benchmarking practices?

• Are other people running into limitations with the GitHub NCAR organization and fine-grained personal access tokens or bots?

• How are others going about benchmarking (doesn’t just need to be packages)?

January 8, 2024

Sign-In:

Name / Lab /  Division / @Github Handle (40 in call):

• Katelyn FitzGerald / CISL / TDD / VAST / @kafizgerald
• John Clyne / CISL/TDD / @clyne
• David John Gagne / CISL/ TDD/ @djgagne
• Brian Dobbins / CGD / CSEG / @briandobbins
• Katie Dagon / CGD / @katiedagon
• Lev Romashkov / CGD / OS  / @rmshkv
• Joe Tribbia /CGD/AMP/ @tribbia
• Negin Sobhani /CISL/ CSG/ @negin513  
• Anissa Zacharias / CISL / TDD / @anissa111
• Ana Victoria Espinoza / NSF Unidata / Science Gateway / @ana-v-espinoza
• Julien Chastang / NSF Unidata / Science Gateway / @julienchastang
• Teagan King / CGD / CCR / @TeaganKing
• Cora Schneck / CISL / TDD / @cyschneck
• Bob Dattore / CISL / ISD / @rda-dattore
• Julia Kent / CISL / TDD/ VAST / @jukent
• Daniel Howard / CISL / CSG / @dphow
• Thomas Martin / Unidata / @ThomasMGeo
• Brian Medeiros / CGD / @brianpm
• Doug Schuster / CISL / @dcschus
• Michael Levy / CGD / OS / @mnlevy1981
• Keith Lindsay / CGD / OS / @klindsay28
• Tom Cram / CISL / @tcram
• Amir Mazrooei / RAL / HAP / @amazroo
• Riley Conroy / CISL / @rpconroy
• Nick Cote / CISL/VAST / @NicholasCote
• Jared Baker / CISL / HPCD / @jbaksta

Agenda:

• ESDS announcements:

• ESDS Annual Event - January 18-19, 2024 - if you missed the deadline and would like to join, let us (Teagan or Katelyn) know
• Upcoming ESDS Forums

• Looking for speakers this spring! - ESDS Forum Presentation Signup
• 1/22/24 forum canceled immediately following the ESDS Annual Event

• Community updates (e.g. resources, updates, upcoming events):

• <name> - <update>
• Katie - relevant events at CU Boulder ESIIL/Earth Lab

• Environmental Data Science seminar series weekly on Tuesdays at 11am
• Forest Carbon CodeFest from March 12-14, 2024

• John - AMS attendees: Register for GeoCAT and ViSR’s upcoming tutorial,  Visualizing 2D and 3D Geoscience Data in Python
• Katelyn (or someone from MetPy) - AMS short course - MetPy: Creating Meteorological Python Workflows from Scratch
• John - Three new exemplar visualization cookbooks available in the Project Pythia Cookbook Gallery

• VAPOR Python
• Advanced Visualization
• Unstructured Grid Visualization
• NA-CORDEX Visualization

• John - Pythia Cookoff - 2024. Save-the-date: June 10 - 14
• Katelyn - UCAR SEA is hosting an open discussion on code review tomorrow (details on the Staff-Only Events calendar)
• Katelyn - Lot of great internship opportunities for undergraduate and graduate students at NSF NCAR/UCAR with deadlines soon approaching: https://www.ucar.edu/opportunities/internships
• Daniel - Open Hackathon with NOAA/NCAR/NREL late Feb. Intend to host a talk by Kristina V. (AGU Open Science Program Manager) for hack participants and broader community at NCAR
• May 29-31 Workshop at NCAR ML: Community Expectations for a Geoscience Data Commons

• Presentation #1:

• Title: Pivoting to NCAR’s Next Generation Geoscience Data Exchange (GDEX), Integrated Research Data Commons
• Speaker: Doug Schuster
• Slides: ESDS-DataCommonsPresentationV1
• Notes:

• Why a data commons?

• Currently fragmented and largely supports a data download model
• Some unification through data.ucar.edu
• Lots of copies of datasets, difficulty w/ access / permissions
• Some data access limited to HPC

• What’s next?        

• Data infrastructure to connect analysis and AI ready geoscience datasets with community developed analytics tools
• Jupyter and associated analytics tools
• Commercial cloud and on-prem cloud / HPC
• Reference: Ten lessons for data sharing with a data commons

• Examples (projects this might build from)

• Copernicus Climate Data Store
• SciServer - NSF funded infrastructure

• User communities / use cases

• ESDS
• Project Raijin
• Project Pythia

• Related initiatives (funded to work with)

• NSF OSDF
• NSF Discovery Cloud for Climate

• Goals

• Common trusted repository services
• Data proximate analytics platform

• Opportunities for collaboration

• Input on plans
• Collaborative development
• User feedback
• Upcoming workshop

• Presentation #2:

• Title: On-Prem Cloud Pilot Project
• Speaker: Nick Cote
• Slides: ESDS Forum 1-8-2024
• Notes:

• Disclaimer: this is a pilot project and still in development
• On-Prem Cloud Pilot Project
• What do you need to sign up?

• GitHub account to access JupyterHub

• Will be hosting a workshop at the ESDS Event
• More in-depth workshop on 1/23 - registration here 
• Documentation (public)
• JupyterHub - Request access (seems like you need to be on the network for this)
• Hosting web applications

• Must be containerized

• Happy to help
• Can use GitHub applications to build and help deploy

• Use Helm & ArgoCD
• Initial deployment will require administrator support, but after that you just push code changes
• Containers have access to GLADE & Stratus
• Working on public access, but get things started now!

• Internal container registry - uses Harbor

• Can also cache images from other repositories

• Contact info
• Would love feedback!
• Link to harbor service is in the docs - https://hub.k8s.ucar.edu/

• Q&A

• <name or anonymous>-<question>
• Is there funding for data engineering / formatting?

• Still work needed
• Looking into this
• Lots of interest still in netCDF, but also interest in Zarr, etc.

• What do the URLs look like? <>.k8s.ucar.edu
• Who is harbor available for?

• On the UCAR network, but for all (controlled by CIT creds)
• Can pull if you’re not authenticated
• Need to be authenticated if you’re pushing

• Curious about NSF NCAR’s strategy for cloud computing (e.g. NDCC) and how this integrates with these efforts

• Chat more later

December 4, 2023

Sign-In:

Name / Lab /  Division / @Github Handle (24 in call):

• John Clyne / CISL / @clyne
• Katie Dagon / CGD / @katiedagon
• Katelyn FitzGerald / CISL / @kafitzgerald
• Julia Kent / CISL / @jukent
• Anissa Zacharias / CISL / @anissa111
• Dave Ahijevych / MMM / @ahijevyc
• Teagan King / CGD / @TeaganKing
• Orhan Eroglu / CISL / @erogluorhan
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Brian Bonnlander / CISL / ISD / @bonnland
• Brian Dobbins / CGD / @briandobbins
• Nick Cote / CISL / @NicholasCote
• Thomas Martin / Unidata / @thomasMgeo
• Tom Cram / CISL / @tcram  
• Joe Tribbia /CGD/AMP @tribbia
• Riley Conroy / CISL / @rpconroy
• Doug Schuster / CISL/ @dcschus
• Ryan Sobash / MMM / @rsobash
• Negin Sobhani /CISL/ @negin513
• Allison Baker/CISL/@allibco
• Lev Romashkov/ CGD / @rmshkv

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• This is the last forum of 2023 - see you next year!
• Next up will be Jan 8, with Doug Schuster and Nick Cote presenting on Data Commons and CISL’s on-prem cloud pilot, respectively

• Registration is open for the Jan 18-19 ESDS Annual Event!

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Katelyn FitzGerald - For those going to AGU, Joe Hamman put together this list of Pangeo related presentations: AGU Fall Meeting - Pangeo++ talks
• Pangeo dinner - sign up from Pangeo Discourse +  link on Zulip

• https://lu.ma/pangeo-agu-2023

• Folks from CISL VAST are putting together Pythia Cookbooks on data visualization, which is also going to be used for upcoming AGU / AMS half-day workshops

• Comment - would love to see announcement for published cookbooks!
• Register directly for the AMS tutorial

• Presentation:

• Title: Enterprise Architecture
• Speaker: Caroline Bain (UK Met Office)
• Slides:
• Notes:

• UK Met Office

• History

• Founded by Fitzroy in 1854
• NWP 1922 - Richardson
• NWP forecasts started in 1955

• Services
• Science
• Technology

• Moving to cloud
• AI for NWP project

• Building successful workforces

• “I” shaped and “T” shaped

• “I” shaped - deep expertise
• “T” shaped - depth + breadth

• Business strategy
• Technology
• Collaboration & community

• Bridging gaps
• Avoiding silos

• Diversity works!

• More technical literacy as a scientistic than most realize at first glance
• Good to move around!

• Mentoring & coaching
• Conscious career development (strengths / weaknesses analysis)
• What is enterprise architecture?

• 1) planning & maneuvering
• 2) community & assurance
• 3) engineering
• Building w/ intention and design
• Solutions architect would do this at the project level vs Enterprise architect does this at the organization level
• Examples

• Could we benefit from collaboration and sharing of strategies?
• Could we benefit from new technologies?
• Really networking and connecting folks
• Architecture Guild - seminar series
• Review and triage of critical services and processes
• C4 diagramming - great experience with this
• Decision record - provides history and context for decision making
• Ensuring there is longer term planning for temporary solutions

• Two key parts of most large orgs

• Doing
• Thinking

• Can be very opaque from the top…
• Lenses

• Organizational hierarchy

• Many layers + silos
• No common language
• Pace of change can be very slow

• Abstracted domain model (UK Met Office)

• National Capability - makes stuff
• Products and Services - produces stuff for customers
• Enabling Capability - anything to support these other areas
• Aligned upon value streams
• Benefits

• Common language and focus on value
• Less threatening and more collaborative
• Helpful for decision making and investment planning

• Caveats

• It’s a model (George Box quote)
• Things take time - still some struggle with change

• Burlton theory

• Detail vs Understanding matrix

• EA - targeting the elegant quarter

• Horizons (time)

• Categories
• Do they move you toward a vision?

• Approach

• Less is more - fewest structures with the fewest number of member - clear roles
• Strive for continual incremental benefit - acknowledge this is not a panacea
• Decisions at the appropriate level
• Consistency and not uniformity

• Organization needed
• But not everyone's the same

• “Learn as we go” - and reflect and refocus regularly

• Takeaways

• Worth trying
• Beneficial for the organization
• Lots of learning
• Excellent career training - opportunity to zoom out
• Tech and science are blurring (even more in data science)

• Roles changing

• Resources:

• C4 diagramming video - on YouTube

• Discussion / Q&A:

• Questions:

•  <name or anonymous> - <question>
• C4 diagrams

• Related to Burlton theory
• UML diagrams have a lot of detail in comparison for example
• Does this help w/ abstraction and high level understanding?

• Yes
• Allows for targeting varying levels of abstraction

• Hard to get folks to do the design pacs (sp?) - helpful to have diagrams that are updated are varying frequencies and can allow for varying formats

• How do you balance things that are new and disruptive?

• AI - changes coming

• Not as developed as some leading industries in weather / climate
• Actively coordinating w/ tech companies and being open minded about potential changes and roles

• Cloud also quite disruptive
• Still learning
• How do you balance shifting vs doing the work?

• Being a bit slow about this in her opinion
• People are exploring
• Probably need to be strategizing about training and changes
• Changes: Sagemaker / Copilot / others
• Be a part of the conversation

• Curious about the flexibility in your role

• How does this work in practice?
• What is the structure?

• Yes, there is a team
• And a team of folks from different backgrounds IT, science, etc.
• Now more folks from broader backgrounds

• What this intended to be a term role?

• Works out that way

• Where did the momentum for change come from?

• Shift from tech being a tool to driving strategy
• Started to be clear to directors and mgmt
• Big data - need to pay attention to this
• Enterprise design

• Info Arch
• Business Arch
• Ent Arch

November 13, 2023

Sign-In:

Name / Lab /  Division / @Github Handle (24 in call):

• Katelyn FitzGerald / NCAR / CISL / @kafitzgerald
• Elena Romashkova / NCAR / CGD / @rmshkv
•  Kevin Sampson / NCAR / RAL / HAP + GIS / @kmsampson
• Anissa Zacharias / CISL / TDD / @anissa111
•  Heather Carrigan/UCP/BSS/GLOBE
• Keith Lindsay / CGD / OS / @klindsay28
• Riley Conroy / NCAR / CISL / DECS @rpconroy
• Katie Dagon / CGD / @katiedagon
•  Cora Schneck / CISL / TDD / @cyschneck
• Steve Yeager / CGD/ @sgyeager
• Joe Tribbia /NCAR/CGD/AMP  @tribbia
• Bob Dattore / NCAR / CISL / ISD / @rda-dattore
• Mike Levy / CGD / OS / @mnlevy1981
• Julia kent / CISL / @jukent
• Samar Minallah / CGD / @minallah
• Nick Cote / CISL/TDD / @NicholasCote
• Jennifer Boehnert / RAL GIS / @boehnert
• Teagan King / CGD / @TeaganKing

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• December 4, 2023 at 11 AM hybrid in the Chapman room - replacing November 27th, off schedule forum by Caroline Bain, visitor from the UK Met Office
• Reminder to sign up for spring time slots!  Or reach out to Katelyn or Elena with suggestions / questions. ESDS Forum Presentation Signup

• Save the date: Jan 18-19 ESDS event (tutorials, collaborative work time, etc.)

• 2024 ESDS Annual Event Survey
• Look for a registration form soon!

• Followup from the Dask town hall forum

• NHUG is looking into more institutional support for Dask help in next fiscal year
• Meanwhile, existing resources include:

• ESDS and CSG office hours
• Zulip
• Upcoming ESDS Annual Event (Jan 18-19)
• Directly reaching out to Brian Vanderwende and Negin Sobhani

• Community updates (resources, events, suggestions, challenges, etc.):

• <name> - <update>
• Katelyn FitzGerald - For those going to AGU, Joe Hamman put together this list of Pangeo related presentations: AGU Fall Meeting - Pangeo++ talks
•  
•  

• Presentation:

• Title: “NCAR GIS Program - Advancing actionable and convergent Earth System Science with geographic information science and technology, geospatial analytics, and geovisualizations”
• Speakers: Olga Wilhelmi, Jennifer Boehnert, Matt Casali, and Kevin Sampson (RAL)
• Slides:
• Recording: ESDS Forum (2023-11-13 14:07 GMT-7) 
• Notes:

• NCAR GIS program

• Started in 2003
• Manage the ESRI licenses for NCAR
• Expanded to broader geospatial work (visualization + analytics) to support the NCAR mission

• Work on tools to make NCAR data more interoperable with common geospatial programs and tooling
• Focus on actionable science

• GIS community and also…
• Collaboration w/ decision makers and stakeholders
• Useful and usable information

• Convergent science

• Integrating physical and social science data and knowledge

• Education and Training

• Broadening participation and capacity building through GIS-focused education
• Free tutorials to teach GIS concepts and tools using weather and climate data
• BRIGHTE workshop series

• Data Interoperability

• CF Conventions
• Compatibility w/ various applications and tools: e.g. ArcGIS, QGIS, GDAL, Panoply, RioXarray, and others
• Other emerging geospatial standards: OGC GeoZarr, STAC
• Have made WRF-Hydro outputs CF compliant and working on others
• For WRF: github.com/NCAR/WINDOW

• Geospatial Analytics

• Analysis and workflows
• Jupyter and ArcGIS notebooks
• Exploring cloud-based climate analytics
• Accelerating Convergence in Earth System Science through geospatial cyberinfrastructure and climate analytics (ACCESS)
• Spatial Analytics using Jupyter Notebooks in ArcGIS Pro - workshop at NWA Pune, India with World Bank funding

• Geovisualizations

• Wilhelmi et al, 2023 Weather, Climate, and Society - https://doi.org/10.1175/WCAS-D-23-0044.1

• Geospatial Storytelling

• ESRI StoryMaps

• Geospatial Tools at NCAR

• ESRI site license

• Desktop (Windows-based) and cloud-based applications (ArcGIS Pro and ArcGIS Online)
• Also have some machines in the FL library with the desktop program installed (accessible remotely as well)

• Expertise in QGIS
• Geospatial data

• Internal NCAR server w/ many, many datasets
• ArcGIS Living Atlas of the World data repository

• Climate Inspector

• Interactive geospatial web application

• Geospatial dashboards

• Air quality in the US - presentation

• Apps with Streamlit

• Mapping
• Plotting
• Data export
• Open source software, Docker container, RAL server

• Resources:

• NCAR GIS Program
• NCAR/UCAR OGC Representatives

• Kevin Sampson
• Ethan Davis 

• Twitter/X - NCAR GIS

• Discussion / Q&A:

• Questions:

•  <name or anonymous> - <question>
• What features of the CF conventions do tools typically rely upon?

• Often working with time series data
• Also the gridded aspect of the standards (grid mapping, coordinate system, etc.)

• ESRI licensing

• Must be on NCAR/UCAR laptops (not personal machines)
• Some do install on MacBooks w/ a virtual machine or access the library computers via remote desktop
• There is a home use program with a limited license - https://www.esri.com/en-us/arcgis/products/arcgis-for-personal-use/buy
• QGIS is also a very nice tool these days

• Tutorials / Workshops

• https://gis.ucar.edu/learn-gis
• Also some educational materials through the NCAR/UCAR site license

October 30, 2023

Sign-In:

Name / Lab /  Division / @Github Handle (34 in call):

• Dan Adriaansen / RAL / AAP / @DanielAdriaansen
• John Clyne / CISL / @clyne
• Negin Sobhani /CISL/ CSG/ @negin513
• Katelyn FitzGerald / CISL / TDD / @kafitzgerald
• Elena Romashkova / CGD / OS / @rmshkv
• Anissa Zacharias / CISL / TDD / @anissa111
• Brian Medeiros / CGD / AMP / @brianpm
• Brian Vanderwende / CISL / CSG / @vanderwb
• Brian Bonnlander / CISL / ISD / @bonnland
• Anna Deppenmeier  CGD / OS / @ALDepp
• Kirsten Mayer / CGD / CCR / @kjmayer
• Cora Schneck / CISL / TDD / @cyschneck
• Teagan King / CGD / CCR / @TeaganKing
• Dave Ahijevych / MMM / @ahijevyc
• Mike Levy / CGD / OS / @mnlevy1981
• Thomas Martin / Unidata / ThomasMGeo
• Allison Baker / CISL / ASAP / @allibco
• Katie Dagon / CGD / @katiedagon
• Nick Cote / CISL/TDD / @NicholasCote
• Wayne Chuang / LEAP / Columbia / @wkchuang
• Keith Lindsay / CGD / OS / @klindsay28
• Sam Levis / CGD / TSS / @slevis
• Rich Neale / CGD / AMP @swrneale

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• November 13, 2023: Olga Wilhelmi, Jennifer Boehnert, title TBD (GIS)
• November 27, 2023 - canceled
• December 4, 2023 - replacing November 27th, off schedule forum by Caroline Bain, visitor from the UK Met Office (likely in the morning!)

• Time slots now open for spring ESDS Forum dates! ESDS Forum Presentation Signup
• Save the date: Jan 18-19 ESDS Event (tutorials, collaborative work time, etc.) 🎉

• 2024 ESDS Annual Event Survey (please help us shape this event with your input!)

• Community updates (resources, events, suggestions, challenges, etc.):

• <please add items here (with your name if you’d like to share)>
• Brian V - heads up regarding the Dask worker network interface on Casper - please use “ext” now instead of “ib0” - this is a consequence of a general move from Infiniband to high-speed ethernet.
• Some interesting upcoming talks in the Pangeo Showcase

• 11/1/23 - “Compression of Geospatial Data with Varying Information Density” by Ayoub Fatihi at ECMWF
• Also talks on Cubed and the Open Source Science Project

• Presentation:

• Title: Dask discussion
• Slides: DaskUserExperiences_ASWIG_Discussions
• Notes:

• Chunking best practices: ​​https://ncar.github.io/dask-tutorial/notebooks/06-dask-chunking.html#dask-chunking-best-practices
• Survey results

• Pain points - challenges that prevent you from using Dask

• Biggest is difficulty in debugging or troubleshooting
• Also lack of familiarity, concerns about scalability or reliability, insufficient documentation
• Plotting can still be very slow
• Shared HPC resources can be limiting
• Merging large, heterogeneous dataframes takes a long time - reindexing
• Documentation is too nested
• Hard to reproduce error conditions
• Mike L - My comment about reproducibility, I might be running at the limit of what workers are capable of with a hires dataset. Killed workers some days but not others.
• Anna-Lena D - Not always possible to run in same environment
• Can take a while to optimize a particular workflow, maybe investment in time for workflows that are used regularly

• Topics for future Dask tutorials

• Advanced topics (performance, tuning, optimization) most wanted
• Debugging and troubleshooting
• Dask with geoscience data

• Negin S - Maybe put together some tutorials with commonly used datasets, e.g. ERA5

• Dask for visualization
• How to use Dask dashboard
• Setting up Dask config files
• How to effectively use Dask with CPUs/GPUs
• Dask for 1D or unstructured data
• Running Dask with scripts submitted to queue
• Really understanding how to manage memory

• Support preferences

• Lots of people wanted NCAR-specific Dask tutorials/workshops
• Collection of example geoscience Dask workflows
• One-on-one consulting about workflows
• Dask working groups to work together
• Better advertising of tutorials (esp. to non NCAR/UCAR people)
• More complex examples

• Negin S - We often don’t know exactly what scientists need

• Guide on how to properly submit issue on Github related to xarray, dask, etc. (maybe ESDS can help?)

• Other issues

• Knowing who to reach out to (ESDS, other options?)
• More education about how to request Dask workers via dask-jobqueue
• More concerns about reliability

• Negin S - If you can avoid using Dask, don’t use Dask

• Brian V - The survey will stay open, please keep giving us feedback! (Zulip also works)

• Resources:

• NCAR Dask Tutorial Materials: https://ncar.github.io/dask-tutorial/README.html 
• Dask Cookbook : https://projectpythia.org/dask-cookbook/README.html 
• Dask Official Tutorial : https://tutorial.dask.org/00_overview.html 
• Dask Docs: https://docs.dask.org/en/stable/ 
• Dask Examples: https://examples.dask.org/ 
• Xarray Dask Docs: https://docs.xarray.dev/en/stable/user-guide/dask.html 
• CSG/ESDS consulting: https://ncar.github.io/esds/office-hours/ 
• Dask discourse: https://dask.discourse.group/ 

• Discussion / Q&A:

• <add questions / comments / resources here>
• Are you generally satisfied with your experience when using Dask (especially on NCAR resources)?

• If support staff should take away one focus area / action item relating to Dask, what do you think it should be?

• Thomas M - Resources about when not to use Dask, debugging why Dask is slow (chunking problems), especially with weird non-tutorial datasets

• Brian V - Is running slow/weird datasets more of lack of tutorial issue or lack of tooling issue?
• Thomas M - Both - real datasets often look too different from tutorial datasets. Something like a tutorial or notebook you can point people to for things to try

• Katie D - Best practices for dask-jobqueue usage (incl. dask-jobqueue vs. ncar-jobqueue). A lot of people’s workflows depend on out of date material, having content with more description of what you’re requesting and why. Also more understanding of adaptive scaling - don’t request too much workers / too much memory when you don’t need it.

• Brian V - Could use a new user to Dask on NCAR systems tutorial. Currently working on a major documentation upgrade in consulting

• Brian M - Get really confused when I’m trying to use Dask in a script rather than a notebook - just want to send a job off and not sure how to set up a batch script.

• Brian V - Heard this from multiple people in survey, maybe have been too notebook heavy. Definitely a gap in the documentation. Also depends on what system you’re running on, no simple answer.

• Are there any particularly recent problems in terms of your Dask experience? Also where would you ask a question first?

• Dan A - Ask on Zulip
• Keith L - Hard to ask specific questions, way too many different variables and moving parts to ask a question to diagnose a problem

• Thomas M - In some cases I’ve answered my own question by the time I’ve documented it top to bottom
• Brian V - Raises the question of what the best way to get support is

• Negin S - What’s causing this frustration with Dask in particular vs. other packages?

• Anna-Lena D - More complicated, so many layers to it
• Mari T - Same as Anna-Lena, code might work fine without Dask and then seems to clog up
• Steve Y - One of the pain points is that I use it for ingesting large ensemble datasets, often makes chunks that are non-optimal (Anderson recommended 140 MB size), by the end of a workflow the chunk size is reduced and not really meaningful anymore. Compute at the end may or may not work. Anderson recommended dask optimize, which seems to help in many circumstances. Would really like to know these sorts of tips and tricks
• Anna-Lena D - There are so many steps you take to make sure something works, every time you try to parallelize something you have to repeat steps again, makes it difficult to diagnose/optimize
• Katelyn F - I think it often requires users to know more about their datasets, hardware, and computational problems than they would need to otherwise.  (and the interactions between these)

• John C - Do people feel like there are adequate examples that are representative of their workflows? Do people feel comfortable with Dask’s profiling tools to improve performance?

• David A - Trying to use diagnostics graphs but unsure how to interpret whether it’s working well or not
• Katie D - Could be more examples of how to use the optimize tool that Steve mentioned. Other examples of profiling, trying to understand what kind of task graph you’re building before you run a compute command. Still come back to question of whether to use .compute() or .persist(), not sure which is better to use for a given workflow. How to troubleshoot this process on the fly?

•  Brian M - Should we be using something else instead of Dask, in general?

• Negin S - Not very familiar with alternatives
• John C - There are some alternatives being explored in the Pangeo community, but Dask is by far the best mature, and compatibility with xarray makes it particularly attractive for workflows that we use at NCAR - but maybe there’s something else out there

• This is a common challenge, but I think it applies here as well, but I think it’s often hard to know which resources are actively maintained and/or up-to-date.  Also, where to look for new resources.  

October 16, 2023

Sign-In:

Name / Lab /  Division / @Github Handle (21 in call):

• Katelyn FitzGerald / CISL / TDD / @kafitzgerald
• Justin Richling/ CGD / AMP @justin-richling
• Joseph Gum / CISL / ISD / @asx-
• Elena Romashkova / CGD / OS / @rmshkv
• Kirsten Mayer / CGD / CCR / @kjmayer
• Cora Schneck / CISL / TDD / @cyschneck
• Joe Tribbia /CGD /AMP/ @tribbia
• Teagan King / CGD / CCR / @TeaganKing
• Ward Fisher / Unidata / @WardF
• Mike Levy / CGD / OS / @mnlevy1981
• Nick Cote/ CISL/VAST / @NicholasCote
• Thomas Martin / Unidata / @ThomasmGeo
• Allison Baker / CISL / @allibco
• Doug Schuster / CISL /@dcschus
• Julia Kent / CISL / @jukent
• Keith Lindsay / CGD / OS / @klindsay28

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• October 30, 2023: AWSIG - CISL/CSG - Discussion on Dask
• November 13, 2023: Olga Wilhelmi, Jennifer Boehnert, title TBD (GIS)

• Save the date: Jan 18-19 ESDS event (tutorials, collaborative work time, etc.) 🎉

• 2024 ESDS Annual Event Survey

• Community updates (resources, events, suggestions, challenges, etc.):

• How many people use Zarr? - Thomas Martin

• 5+ a collection of thumbs up!

• Presentation:

• Title: “Caring for the ‘Data’ in Data Science”
• Slides: Caring for the "Data" in Data Science
• Notes:

• Survey: https://forms.gle/8SoQ7ex9HWGKGRFR8 
• Scientific data management:

• Definition:
• Examples: DOI, Zenodo, many many others
• Why?

• Required by funders and journals
• DOIs for software, data, etc.
• Reproducibility
• The net effect is accelerated time to science

• FAIR principles:

• Findable, Accessible, Interoperable, Reusable

• How to care for data:

• Archival and distribution
• Appropriate and as much metadata as required
• Data rescue

• Scanning old data, finding grants to store data for ongoing research that doesn’t have data management budget, future data management plans…

• Data proximity

• Data as close to compute as possible (on HPC or in the cloud)

• Resources:

• Broader initiatives of potential interest:

• Ocean Hack Week
• Climatematch Academy
• Hack the Hackathon

• Data resources:

• CARE data principles

• NCAR/UCAR resources:

• Sundog: create a data management plan & budget
• Email datahelp@ucar.edu for any data management questions

• Discussion / Q&A:

• Q: What is your favorite way to move around/store large datasets?

• Depends a lot upon the situation
• S3 seemed easiest, but that was in a certain situations

• Q: Question about future plans for NCAR’s data (curiosity from NOAA staff about what future access might look like)

• Doug Schuster:

• Would like to have a system that is accessible from both NCAR HPC and commercial cloud (i.e. multiple compute platforms).  Have some funding for a demo to work on this.
• Would also like to make resources more easily accessible / organized internally

• Q: Timeline:

• Depends on funding, ~3 year project

• Q: Where is the best place to find data management resources for NCAR/UCAR?  Is it the DASH page on Sundog?

• DASH Sundog page is probably the best place for now

October 2, 2023

Sign-In:

Name / Lab /  Division / @Github Handle (36 in call):

• Philip Chmielowiec / CISL / TDD / philipc2
• Katelyn FitzGerald / CISL / TDD / kafitzgerald
• Elena Romashkova/ CGD / OS / rmshkv
• Allison Baker/CISL/TDD/allibco
• Julia Kent / CISL / @jukent
• Negin Sobhani / CISL /CSG / @negin513
• Cora Schneck / CISL / TDD / @cyschneck
• Kevin Sampson /RAL/HAP/kmsampson
• Brian Bonnlander / CISL / ISD / @bonnland
• Joe Tribbia /CGD/AMP/ @tribbia
• Mike Levy / CGD / OS / mnlevy1981
• Katie Dagon / CGD / @katiedagon
• Anissa Zacharias / CISL / @anissa111
• Carl Drews / ACOM / Web / @carl-drews
• John Clyne / CISL/TDD / @clyne
• Orhan Eroglu / CISL / @erogluorhan
• Nick Cote / CISL/VAST / @NicholasCote
• Justin Richling CGD/ AMP @justin-richling
• Ben Gaubert ACOM
• Naoki Mizukami / RAL / @mizukami
• Hsiao-Chun Lin / COSMIC @hchunlin
• Christine Shields/CGD/ @shieldsca
• Daniel Adriaansen / RAL / AAP / @DanielAdriaansen
• Brian Medeiros / CGD / @brianpm

36 attendees

Agenda:

• ESDS announcements:

• Upcoming ESDS Forums

• October 16, 2023: Joseph Gum - Caring for the data in “data science”
• October 30, 2023: AWSIG - CSG - Discussion on Dask

• Request for input on the upcoming 2024 Earth System Data Science Annual Event

• 2024 ESDS Annual Event Survey
• Please share with others from NCAR/UCAR who might be interested!

• ESDS governance and intros - John

• Website: https://ncar.github.io/esds/about/
• Governance document: https://docs.google.com/document/d/1xjNzJqZjeRzUg5BMipQVUsy2xA1SRlEiyJLpPmUAs9E/edit?usp=sharing
• Looking for more folks to serve on the leadership team!  Especially from underrepresented labs/orgs.
• We also have working groups folks can get involved in outside of or in conjunction with the leadership team!

• ESDS community updates (resources, events, suggestions, challenges, etc.):

• NCAR HPC User Group (NHUG) Meeting 10/3 at 9:00 am MT - GLADE2 decommissioning, Derecho performance, and Derecho user experience
• CISL Seminar 10/5 at 1:00 pm MT - Good for Scientists, Bad for Society: What happens when our charts don't change minds? - Evan Peck, University of Colorado Boulder
• ESIP Community Fellows Program (for graduate students or postdocs) - application open until 10/6
• <please add additional relevant items>

• Presentation (will be recorded):

• Title: “Development of Unstructured Grid Visualization Functionality in UXarray”
• Slides: here
• Recording: https://drive.google.com/file/d/1-SbD7SQP75QTbyVJWQLaMb527mt9_RKl/view?usp=sharing 
• Notes:

• UXarray

• Unstructured grid visualization and analysis tool in Python
• Sponsored by NSF and DOE
• Can read:

• MPAS
• UGRID
• Exodus
• SCRIP

• Internal representation in UGRID
• UXarray’s role in visualization:

• Initial Encoding & Grid Representation
• Data Cleaning & Processing
• Conversion Methods
• Visualization Routines

• Resources:

• GitHub repo: https://github.com/UXARRAY/uxarray
• Documentation: https://uxarray.readthedocs.io/en/latest/
• Ian Franda’s presentation on UXarray visualization: https://youtube.com/watch?v=b8UNXgGyV-o&si=aLhanHUlWhnrGowp
• Holoviz example: https://uxarray.readthedocs.io/en/latest/examples/005-to-geodataframe-for-holoviz.html
• Datashader

• Discussion / Q&A:

• Any provisions for 3-D visualization?

• Quick answer - not right now
• VAPOR supports 3D visualization of MPAS grids. CAM-SE in the future

• Is there any interactive vector plotting or is that mostly through geopandas plotting capabilities?

• Most examples use Bokeh and many are interactive
• Can be very slow and require a lot of memory

• How much of this uxarray visualization is already available versus currently in development?

• Steps 1-3 are implemented (see usage examples)
• Working on a basic raster plot right now
• Native plotting routines soon

• Is the conversion overhead only per grid or greater?

• Only per grid

• <add your name or anonymous and question here>

• Reminder about the event survey and any additional community updates

September 18, 2023

Sign-In:

Name / Lab /  Division / @Github Handle (28 in call):

• Anna-Lena Deppenmeier / CGD / OS / @ALDepp
• Katie Dagon / CGD / @katiedagon
• Hendrik Grosselindemann / CGD / OS / @hgrosselindemann
• Katelyn FitzGerald / CISL / TDD / @kafitzgerald
• Keith Lindsay / CGD / OS / @klindsay28
• Orhan Eroglu / CISL / TDD / @erogluorhan
• Elena Romashkova / CGD / OS / @rmshkv
• Brian Vanderwende / CISL / HPCD / @vanderwb
• Brian Medeiros / CGD / AMP / @brianpm
• Philip Chmielowiec /  CISL / @philipc2
• Nick Cote / TDD / VAST / @NicholasCote
• Sam Rabin / CGD / TSS / @samsrabin’
• Julia Kent / CISL / @jukent
• Justin Richling CGD / AMP @justin-richling
• Wayne Chuang / LEAP / Columbia/ @wkchuang
• Matt Mayernik / NCAR library / @mayernik
• Teagan King / CGD / @TeaganKing
• John Clyne / CISL/TDD / @clyne
• Allison Baker /CISL/TDD/ @allibco
• Cora Schneck / CISL / TDD / @cyschneck
• Doug Schuster / CISL / ISD / @dcschus
• Joe Tribbia /CGD/AMP/ @tribbia

• 28 attendees

Agenda:

• ESDS announcements:

• New blog posts! And Contributors Guide

• "Thinking through CESM data access" (note: still in progress to update some data locations)
• "Analyzing and visualizing CAM-SE output in Python" 
• “ESDS at SciPy 2023”

• Planning an ESDS Fall / Winter Event!

• Will be looking for input at upcoming ESDS Forums

• Upcoming ESDS Forum on October 2nd with a presentation on UXarray visualization developments by Philip from the GeoCAT team at NCAR

• ESDS community updates (resources, events, suggestions, challenges, etc.):

• Orhan

• Welcome Cora to GeoCAT!
• Introducing Philip from GeoCAT, presenter with “UXArray visualization developments” at the forum on October 2!

• Presentation:

• Using catalogs for model diagnostics - Keith Lindsay NCAR/CGD/OS
• Slides: Catalogs in ESM Analysis
• Relevant links:

• Intake-ESM

• Documentation: https://intake-esm.readthedocs.io/en/stable/
• Links to examples: https://intake-esm.readthedocs.io/en/stable/reference/faq.html

• Intake

• Documentation: https://intake.readthedocs.io/en/latest/index.html

• Catalog building tools

• ecgtools - https://github.com/ncar-xdev/ecgtools

• Created by Anderson Banihirwe and Max Grover
• Building an Intake-esm catalog from CESM2 History Files (with ecgtools)

• esm_catalog_utils - https://github.com/klindsay28/esm_catalog_utils 

• Can update catalogs of ongoing runs
• Fewer built-in assumptions about filename conventions
• https://esm-catalog-utils.readthedocs.io/en/latest/notebooks/ex1_caseroot_hist.html

• ESDS Blog posts / Pythia cookbooks

• Building an Intake-esm catalog from CESM2 History Files
• Using Intake-ESM to Analyze Data from CESM2-LE
• Intake Cookbook 

• What is a catalog?

• CSV file, sometimes with associated JSON file, that specifies paths to data files, isolating those details from your code

• Benefits:

• Includes metadata in a relevant structure
• Abstracts from the directory structure
• Functionality for search, discovery, and data access

• Pain points:

• Building the catalog

• Discussion / Q&A:

• Questions for the speaker:

• Katie Dagon - curious about how generalizable the catalog creation tools might be

• Ecgtools can work with a wide range of use cases
• Esm_catalog_utils has been used for all components of CESM

• Doesn’t need to be caseroot
• May need to create helper functions for other models / use cases

• John Clyne - are you able to share these catalogs?

• Have shared with Elena
• Mostly for smaller projects
• Elena - some floating around the OS

• Katelyn FitzGerald - how do folks share info about catalogs and/or manage communication and maintenance?

• Can be a pain point
• Managed adhoc sometimes or in a GitHub repo (does seem to help)

• Brian Medeiros - portability of catalogs - what happens when the files move?

• Unsure
• There is an issue to help w/ this use case
• Could be tools to help especially if the directory structure is the same
• Relative paths could also be helpful - not sure how well tools are set up for this

• Doug Schuster

• Curious about these from the perspective of curated dataset collections such as those maintained in the RDA and Climate Data Gateway, etc.

• Imagine there would be interest
• Definitely from OS of CGD
• Curious what datasets folks would be interested in to get things rolling