1 of 34

How the NASA Openscapes community supports Earthdata users migrating workflows to the Cloud

Slides: https://nasa-openscapes.github.io

Julia Lowndes, Openscapes

Celia Ou • Catalina Taglialatela • Cassie Nickles • Nikki Tebaldi• Victoria McDonald • Jack McNelis, NASA Physical Oceanography Data Center (PO.DAAC) Openscapes Mentors

NASA Openscapes Mentors and the Openscapes Community

• Bri Lind • Catalina Taglialatela • Cassandra Nickles • Aaron Friesz • Ian Carroll • Michele Thornton • Daniel Kaufman • Alexis Hunzinger • Rupesh Shrestha • Mahsa Jami • Luis Lopez • Celia Ou • Alex Lewandowski • Chris Battisto • Jess Welch • Brianna Pagán • Sargent Shriver • Amy Steiker • Makhan Virdi • Brandon Bottomley • Stefanie Butland • Andy Teucher • Erin Robinson & more!

NASA Jet Propulsion Laboratory

May 23, 2024

2 of 34

What if we connected our skills & values as a daily practice, for climate?

data science + open science

mentorship + teamwork + community

“To address our climate emergency, we must rapidly, radically reshape society. We need every solution and every solver”. - Ayana Elizabeth Johnson & Katharine Wilkinson, All We Can Save

2

What if we connected our skills & values in our daily work, for climate solutions and justice? Connecting our values to our daily workflows with technical skills and leadership skills.

In our work we’ve seen that when we combine Data science <> Open science <> Teamwork & community is a way for us all to help address our climate emergency. As Ayana Elizabeth Johnson and Katharine Wilkinson say in the book all we can save: “To address our climate emergency, we must rapidly, radically reshape society. We need every solution and every solver”.

I’m going to spend this talk telling two stories of what this looks like and invite you to join us

But first I wanted to tell a bit about me and what drives me to this work. I am often asked why a marine scientist would need data science, so I want to make sure you leave here knowing that marine scientists, as scientists more broadly, work with data every single day to do our science, and it is critical that we are supported to work with data responsibly and to code collaboratively.

So let me start off when a peek at my PhD work, which was here at Stanford University.

3 of 34

What Openscapes brings

Focused on research teams, connecting to leadership
Centering climate
Kinder science, inclusion

Openscapes is an approach & community that helps shift culture by helping people move from lonely science to teamwork as we explore & navigate the open science landscape safely with our teams.

Kinder science for future us

4 of 34

NASA Openscapes:

Supporting Open NASA Earth Science in the Cloud

NASA Award# 20-TWSC20-2-0003 Leads: Julia Stewart Lowndes & Erin Robinson

A mentor community across NASA Earth science data centers that helps support users though:

Co-creating common tutorials
Hosting cloud training events
Practicing open science ourselves!

Proving ground for open source and Cloud community development within orgs

https://nasa-openscapes.github.io

5 of 34

End-Users

NASA Openscapes Mentor Community Values

NASA Data Center Staff

Open Science Community

Consistent trust, feedback, & iteration has shaped tools & approaches

>> Learn together first

>> Peer-teach, document, and onboard

>> Teach scientists/users (Openscapes Champions is one format where we get to know users deeply)

>> Connect with the broader open source community, collaborate & give back

6 of 34

6

What Open looks like - live learning, testing, & troubleshooting together

Mentors from 3 different NASA data centers (PO.DAAC, NSIDC, ORNL) testing cloud infrastructure developed by Luis López, NSIDC during regular Openscapes coworking sessions

7 of 34

7

What Open looks like - async contributions via GitHub

8 of 34

8

https://nasa-openscapes.github.io/news/2024-02-28-github-community-call, Bri Lind LP DAAC

What Open looks like - project mgmt in the same place as code & docs

9 of 34

Collaborative software to build habits & community

What Open looks like -

effective teams

Empathy
Time to reflect
Psychological safety

Where team members are not rejected or embarrassed for speaking up with their ideas & concerns (Edmundson)

Growth mindsets

The power of yet - “I don’t know how to do this yet”(Dweck)

Reflect and share ideas early

Meetings to learn together, invite others, role-model open

10 of 34

Tooling

tools & practices

People

teams & community

Technical & social infrastructure together

Some Openscapes stories…

11 of 34

earthaccess aims to be a technical and social bridge between NASA Earthdata and scientific research.

Reproducibility with less code makes open science more accessible.

https://nasa-openscapes.github.io/news/2024-03-04-earthaccess-tech-spotlight

12 of 34

earthacess: making things simpler

earthaccess is a Python library that simplifies data discovery and access to NASA Earthdata by providing an easy to use library to interface with NASA’s authentication and search APIs so that science can be done using less code.

Auth: earthaccess handles authentication with NASA EDL.

Search: earthaccess abstracts NASA’s search API (CMR) into a pythonic module.

Access: earthaccess can download or open data for both cloud and on-prem hosted datasets with the same code.

13 of 34

earthaccess new features save time & egress $$

13

ESDS Tech Spotlight: earthaccess

Luis López, Matt Fisher, Aaron Friesz, Qiusheng Wu, Amy Steiker & community

Blog w/ slides, notebooks & video: https://nasa-openscapes.github.io/news/ 2024-03-04-earthaccess-tech-spotlight

earthaccess and the cloud: the force awakens notebook - Luis Lopez demo

14 of 34

“Daskifying” code: from serial to parallel

Mahsa Jami (LP DAAC) brings a GEDI notebook to our biweekly Mentor Calls. Screenshares with Luis López (NSIDC) while we all watch & learn.

https://podaac.github.io/tutorials/notebooks/Advanced_cloud/basic_dask

PO.DAAC tutorials…

…are used in Mentors’ coworking…

…and then amplified to teach others

Mahsa Jami (LP DAAC) and Luis López (NSIDC) develop & teach coding strategies for parallelization to Champions Teams, May 17 (slides)

15 of 34

Parallelized science in the Cloud - first “fledging” story from 2i2c JupyterHub!

“It’s like a super power”: I did the analysis on my AGU poster: 150 TB of L1 and L2, and I needed some tiny fraction that were of interest to me - in the old way I would need to find a way to download, don’t have a machine big enough. But with Coiled, I can subset. Now once I realize that, I realize there are bigger datasets that just seem unworkable before.”

Aronne shared his story with the 2024 Champions Cohort science teams in April!

First Forays into the Cloud - Youtube

“In Champions Cohort we got a teaser of how to work in the Cloud. With Coiled and I did some simple things and now I’ve moved into my own workflow to Cloud. I have an Amazon account now ‘paying my own way’.”

- Aronne Merelli, University of Wisconsin

2021 Kevin Murphy call - what if this doesn’t work. We’ve iterated on Champions, and on teaching opps that Mentors have presented, to 4 workshops at AGU, to Tech Spotlight, Webinar, UWGs. They don’t have any hesitation, they’re not starting from scratch every time and not isolated when they do it

And, Champions. We’ve seen huge strides

Aronne - was not a cloud expert. Was a NASA data expert and python skills.

Learned to use the Cloud through materials Mentors developed. decided he was going to do something he needed to do anyways. AGU poster using our infra.

Now he’s paying himself, let’s all cheer for this.

Flywheel once - immensely hard to get to this place. But now, it will get immenesly easier

“Aronne took 3 years plus 30 mentors and all the NASA earthdata - it took all of us plus coiled to get here.”

We can support them using the Cloud. Aronne quotes “working in a new way” - answering new q

16 of 34

Ongoing work: onboarding & “fledging” users; maintaining community infrastructure

2024 Goals:

earthaccess

Python library

Also: earthdatalogin R package, MATLAB library

Cloud Cookbook: tutorials, contributing

Workshop planning automation

Cloud infrastructure (2i2c JupyterHub)

https://nasa-openscapes.github.io

NASA Earthdata Cloud Cookbook walkthrough video, by Cassie Nickles

The mentors are learning how to work in the Cloud together using a 2i2c JupyterHub, where they are supporting researchers to work there in Python, R, and Matlab. They are leading learning events using tutorials they created and reuse and iterate for different audiences and events for researchers and other NASA staff, They are also identifying user needs and developing where they see needs including the earthaccess python library; cheatsheets and guides. They describe how they are now working openly with each other across their silos, feel braver speaking up in other places, and that they are comfortable sharing imperfect work with each other and getting and receiving feedback.

2i2c

Work of the Mentor Community

How to grow it - working on real researchers in the science teams

Luis showing this real example of work the Mentor community has done

We are really grateful for our NASA Mentors, and to NASA who is supporting their time so that open science can be a paid part of their jobs.

We’re supporting Support Mentors towards establishing a common set of NASA Earthdata tutorials that they can then build off their DAAC-specific and science examples

Again, these recurring themes. Wanted to reuse existing so needed to identify what is unique to NASA earthdata and what’s not, tease that apart to reuse existing. Only contribute to combine and fill gaps rather than reinventing

We center researchers, and we focus on this feedback loop between tooling and people

---

17 of 34

PO.DAAC Connections in Openscapes

Ease burden on Earthdata end users by consolidating resources where possible and making access patterns and guides consistent
Decreasing duplication of work
Collaborating towards supporting interdisciplinary NASA data use
Openscapes 2i2c Cloud Hub for prototyping and onboarding early adopters and training at workshops

17

https://nasa-openscapes.github.io/

Openscapes: PO.DAAC Team members participating over past 3 years:

Catalina Taglialatela, Cassie Nickles, Celia Ou, Nikki Tebaldi, Victoria McDonald, Jack McNelis

Cloud & Open Science Mindset

DAAC Mentor Training & community building
User Training

18 of 34

PO.DAAC Connections in Openscapes

PO.DAAC Cookbook: was modeled after the Openscapes Cookbook, and links the common pieces (How To, Cheatsheets) to the NASA Earthdata Cloud Cookbook
Adoption of earthaccess in PO.DAAC data tutorials and workflows: consistent & easy for end-users across DAACs (2023 PO.DAAC summer intern easily learned and implemented earthacess in tutorials)
Openscapes Champions cohorts: engaging with end-users, listening, training

19 of 34

PO.DAAC Connections in Openscapes

Hosted or co-hosted several workshops using the 2i2c Openscapes Cloud Hub:

2021 Cloud Hackathon
2021 AGU Openscapes Cloud workshop
2022 SWOT Ocean Workshop
2023 AGU Openscapes Cloud workshop
2024 AGU Chapman: Remote Sensing of Water Cycle - SWOT Data Access workshop

Leveraging the 2i2c Hub to develop and prototype tutorials, learn ourselves (DAAC mentors), teach others, demonstrate using the Cloud at Meetings (e.g. SWOT ST meetings, SWOT Applications Early Adopters etc), onboard workshop partners
Cloud Playground

Much discussion happened in Openscapes sub-group

PO.DAAC tutorials - - - - - - Ayush

20 of 34

PO.DAAC Connections in Openscapes

Learning from each other across DAACs - troubleshooting, workshop skills
Gaining Github skills for code and project management
Motivation and encouragement - we all encounter the same questions
Exercising engagement with empathy (What is the listener looking to get out of this? Is this jargon?)
Always a work in progress - Openscapes community keeps me going
Community presence at AGU, ESIP
Carpentries training

2024 NASA Champions
Total # of teams	9
Teams with multi-DAAC	6
Teams listing PODAAC	3

21 of 34

Shifting culture to Open Science

Changes in how INDIVIDUALS operate

Develop technical skills, ex: Google Drive, R, Quarto, JupyterHub, Python.
Shift mindset - braver to speak up & ask questions, share ideas

Changes in how TEAMS operate

Build psychological safety in teams; tackle seemingly insurmountable challenges
"We weren't a team before Openscapes" - met 2x/yr, didn’t know what each other were working on
Our new workflow doesn’t rely on 90 emails; we have an automated Google form

Changes in how ORGANIZATIONS operate

NASA Earthdata has a common way to teach Cloud across data centers (DAACs)
NOAA Fisheries Open Science Vision & Open Science Lead (Eli Holmes)
CA EPA/Water Board has Open Science strategic plan, implemented in part via annual Champions cohorts - led internally “forked Openscapes”
Cross-government sharing, learning, and reusing what works

22 of 34

How we work: Flywheel to modernize teams & create self- sustaining culture change in orgs & science

Robinson & Lowndes 2022

Invest in learning and trust Everyone has something to learn, ask, teach; don’t need to be an expert in everything. Psych safety, growth mindset. Slowing down to speed up.

Flywheel

Create space and place to connect and collaborate; remove barriers to participation (paid time, part of jobs)

Empower

Learning culture

Engage

A Future Us mindset

Amplify

Open leaders

Work Openly Put what you learn into practice quickly, role- modeling sharing imperfect work. Identifying common challenges & opportunities. Documentation. Openness is a spectrum; first Future You and then Future Us.

Welcome

bright spots (be they mentors or researchers) –people who want to work better and collaborate

Leverage common workflows, skills, tools. This is where we speed up: Iterate, reuse, remix with each other and the broader community

Inspire broader research communities through celebrating people and small wins that accumulate – Open science shift

This is how we work: we’ve developed a cyclical process we talk about as the Openscapes Flywheel. We engage a Future Us mindset through welcoming and creating space and place, we Empower a learning culture by investing in learning and trust, and working openly, and we amplify open leaders by leveraging common open data science workflows, skills and tools and inspire broader scientific culture.

The Flywheel is a concept developed by Jim Collins in the book Good to Great; it’s the idea that when building something to last, there is no single defining action. Rather, the process resembles relentlessly pushing a giant, heavy flywheel, turn upon turn, building momentum until a point of breakthrough, and beyond. We’ve found these are the synergistic and overlapping elements for open data science movement building which we’ve tested, iterated, and refined in working with over 150 research teams since 2019. We are really focused on building habits that last, and institutional change that lasts. Let me share a few example:

https://www.jimcollins.com/concepts/the-flywheel.html

23 of 34

Growth mindset: concepts and tools to reimagine workflows

Wickham & Grolemund 2016

https://quarto.org

https://github.com

https://openscapes.github.io/series

https://openscapes.github.io/quarto-website-tutorial

Julie

The first was stepping back and thinking about the common parts of our data analysis.

This is HW’s conceptual diagram for data science where he illustrates the importance of tidying data first before beginning to ask your research question. this was criticatl for us to separate out the common parts we were all reinventing so we could collab on those common parts and then give more creative energy and time for the science.

Less reinventing

-----

This is probably the most impactful figure in my science-life. The idea that no matter your question, your study system, etc, you will need to go through these deliberate steps if you’re working with data. And it’s really in this grey box of Understanding where the Science starts. So this is separating your science questions from this first part, which is import and tidying.

The idea of tidying your data first and then being streamlined with the tools you can use for science is amazing. So not starting the science too soon. I’ve been burned by this, and it means forever building bespoke approaches to accommodate weird data formatting. And knowing that tidying data takes a huge percentage of analysis time

24 of 34

Growth mindset: Mentorship is a skill we can all develop

Develop growth mindset, learn from & join existing efforts

24

Generosity

Empathy

Asking questions

Listening

Not solutioneering

Meeting folks where they are

Reflecting, learning

Speaking up

Connecting values to daily actions

“Perfection is a colonial conditioning.

Worrying that you’re going to get it wrong holds you back from trying, and that erases.”

- Ta7talíya Michelle Nahanee, Nahanee Creative

Territorial Acknowledgements Interactive Workshop

25 of 34

25

“As fisheries scientists, we know that when we’re in rough seas, it’s important to keep the ship afloat AND get out of the storm.

Openscapes helps teams steer out of the storm of email chains with 37 versions of the same spreadsheet and to the calmer waters of open science and meaningful collaboration, using tools like R.”

Adyan Rios

NOAA Southeast Fisheries Science Center

26 of 34

Planning (during cohort): SEFSC IEA team

How Ecosystem Status Reports are currently made

Data: numerous sources; CSVs everywhere
Metadata: each researcher’s responsibility to learn/document
Analysis: Excel and R is used; GitHub - not used as a group
Version control: Final_final_REALLY FINAL
Storage: individuals own Google folders
Communication: periodic meetings and emails
Writing: Google docs /Rmarkdown used to create report in some cases

How we envision making this process better

Use of structured, relational databases
Data extract code for automated processing on GitHub
Use of single repository for ESR
Live document with Quarto to facilitate reproducibility
Metadata on how we pulled data; contact information and versions of data accessed
Github to increase cross-NOAA line office collaborations

Challenges to overcome

Fostering online collaborative culture (e.g., people at different levels of comfort with Github
Explicit best practices on collaboration and Github
Transferring old methods and code to Github

Mini ESR GitHub

27 of 34

Doing (2 months after cohort): NWFSC IEA team

27

Slide from NMFS R User Group Lightning Talks: Open Science Work in R Jan 31, 2023; (video)

28 of 34

Doing (years following cohort): NEFSC IEA team

https://noaa-edab.github.io/presentations/20211015_Openscapes_Bastille

29 of 34

Movement building

What’s possible because all this

More time on science & solutions

It’s not just about time saved - it’s better products, less lost day-to-day & during succession

Improved morale

Help get unstuck, less lonely, staying in jobs. Real relationships. Institutional culture change.

Climate and social change

Connecting our biggest challenges with our daily work. Collective agency, voice, action

2023 was the Year of Open Science. 2023 was also the Year of Taylor Swift, according the The New York Times. What can the Open Science movement learn from Taylor Swift, whose specificity in her storytelling has connected and empowered millions of people? "Taylor Swift has taken her story and made it big enough for everybody".

Taylor Swift
Issa Rae
Get out the Vote
All We Can Save
Recoding America

Learning from & joining other movements

30 of 34

Join us:

Mastodon, Bluesky: @openscapes

openscapes.org�NASA Openscapes

NOAA Fisheries Openscapes

Openscapes at the Water Boards

Learn, reuse, remix, support others, grow the movement

Earthdata Cloud Cookbook

Champions Lesson Series

Approach Guide

Thank you!

30

Julia Stewart Lowndes, Celia Ou

NASA Openscapes Mentors & Openscapes Community

Thanks to the people who made this possible!

Not pictured: more people!

Diverse, inclusive teams and communities are key

Slides: openscapes.org/media

31 of 34

31

32 of 34

Further resources (incomplete list):

White Paper: The Value of Hosted JupyterHubs in enabling Open NASA Earth Science in the Cloud - Nickels et al 2023
Openscapes Flywheel: A framework for managers to facilitate & scale inclusive Open science practices - Robinson & Lowndes 2022
Shifting institutional culture to develop climate solutions with Open Science Lowndes et al 2023
NASA-Openscapes.github.io - NASA Openscapes Mentors
Hello Quarto: share • collaborate • teach • reimagine - Lowndes &Çetinkaya-Rundel 2022 Keynote (video)
A Journey to Data Science: Tools for Equity and Diversity in STEM - Fenwick 2022 (video)
Open software means kinder science - Lowndes 2019
Supercharge your research: a ten-week plan for open data science - Lowndes et al. 2019
Our path to better science in less time using open data science tools - Lowndes et al. 2017
3 lessons from remote meetings we’re taking back to the office - Cabunoc Mayes et al. 2020
Toolkit for Incentivizing Open Science - National Academies (NASEM) Report; Openscapes blog post
All We Can Save - Johnson & Wilkinson 2020, eds
Braiding Sweetgrass - Kimmerer, 2013
From Open Data to Open Science - Ramachandran, Bugbee, & Murphy 2021
Unmet needs for analyzing biological big data - Barone et al. 2017
How to bring open source to a closed community - Cabunoc Mayes 2016

33 of 34

Collaborative software to build habits and community

Reflect and share ideas early

Meetings to learn together, invite others

Role-model open & build psychological safety

34 of 34

How we work

The Openscapes Flywheel:

A framework for managers to

facilitate and scale inclusive Open science practices

Robinson & Lowndes 2022 (preprint)

Leverage common workflows, skills, tools, approaches

Work Openly

Invest in learning and trust

Inspire

Flywheel

Create space and place

Welcome

Empower

Learning culture

Engage

A Future Us mindset

Amplify

Open leaders

Flywheel concept

transformations occur from consistently doing key activities that add up over time

(Collins)