1 of 30

Line of Effort 3: Fully Enable Open Science

Modernizing data workflows across an large and multi-faceted organization requires changes in how we work. LOE3 is about helping people and teams achieve these goals.

Earth Science Information Partners (ESIP)

Summer 2023, Burlington, VT

July 19, 2023

Eli Holmes

NOAA Fisheries

Northwest Fisheries Science Center

Seattle, WA

2 of 30

A little about me

Lead of NOAA Fisheries (NMFS) Open Science initiative starting Oct 1
Co-lead NMFS Openscapes the last 3 years
25 years with NOAA, working maining on Endangered Species Act issues in the PNW
Applied Mathematician

Develop algorithms for multivariate time-series analysis for noisy data. Matrix derivatives!!

Heavily involved in Open Source statistical package development
Part of a NOAA-India collaboration in Open Science training since 2015

Elizabeth Eli Holmes

NOAA Fisheries Northwest Fisheries Science Center

3 of 30

Line of Effort 3 Strategic area

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 3

4 of 30

Fisheries Enterprise Data Management Program

Data Governance

Communities & Partnerships

Data Culture

Data Management

& Data Lifecycle

Data Workforce Development

Enterprise Data Management

Applications & Tech Innovation

Data Science & Analytics Capabilities

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 4

5 of 30

Why “Fully Enable Open Science”?

“Open science practices have been shown to increase reproducibility and efficiency in scientific research AND federal agencies are required to have transparency and trust in the science that underpins regulatory decisions BUT our workflows are fundamentally opaque and non-reproducible and recent high-profile papers have shown that very few published scientific studies can be fully replicated THEREFORE federal agencies need to embrace open science practices in order to increase transparency, reproducibility, and open review of our scientific research. By doing so, we can increase the rigor and reliability of scientific research, reduce staff time and costs, and foster greater public trust in science-based regulatory decisions.”

– Eli’s elevator pitch

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 5

6 of 30

Ok, so you’re convinced that Open Science organizational change is a good goal.

But how do we do this?

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 6

7 of 30

Conditions for diffusion of innovation

Conditions for team continuous improvement

Solve problems that are causing “pain”

Lower barriers to adoption

Break down barriers to information spread

People and interactions over tools

Rapid feedback cycles

Connection to the end user

Respond to change rather than fixed plans

Adapted from the Agile Manifesto but these elements are found across all team CI ideas

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 7

8 of 30

Data, so much data

Analyses, plots, tables with no documentation (just the final product)
Manual undocumented manipulations
Many data file in different formats
Scripts of various analyses
Emails, emails, emails
Lots of Google docs
Files on individual folders
Data of unknown provenance

Unreproducible and inefficiently produced product:

Paper
Decision
Report

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 8

9 of 30

Artwork for Openscapes by Allison Horst!

Science Team

Dagobah Swamp of a big report

BEFORE

10 of 30

Cascading effects of climate change

New risks and unknown effects of all this change

Flat budgets amidst rising operation costs

Retirements leading to loss of knowledge and capacity

More data and new data streams

So much wasted staff time with inefficient workflows

Federal mandates for transparency and open data.

Artwork by Allison Horst for Openscapes

Why care about this? That is not a rhetorical question. Science has come a long way using approaches, that we know now are fundamentally unreproducible. Big government reports have

There's a lot of dark clouds coming especially with the cascading effects of climate change. that's going to affect all our science. Our models were developed and trained on data from an ocean and freshwater aquatic systems that look very different than the ones we will have in the coming years. Species are appearing where they've never appeared before and the basic environmental drivers are changing both in levels and variability. This means we are going to have so much extra work to change our models. Then on the good side wow we've got all this great new data and oh wow we got all this new data we need to incorporate that's hard. Then we have these you know mandates for transparency open data. So big motivations and reasons to help our science move towards these much more efficient open science workflows: from data to product. That takes us to the NOAA Fisheries Openscapes training program.

11 of 30

Good news!

We know how to create “reproducible science workflows”
NOAA Fisheries has teams & programs who do this well: e.g. FIMS and FIS communities
We have been training individuals and teams 2020-2022
Tools have gotten much better and easier to use

Fisheries Information Management Modernization Workshop 2020, Tech Memo September 17-19, 2019, NMFS Office of Science and Technology (OST)

Data

Analysis

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 11

12 of 30

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 12

13 of 30

Artwork by Allison Horst!

Better teamwork, and new skills and workflows

That big report

AFTER

14 of 30

Great! We can help individual teams but how do we make change go viral?

Me

15 of 30

3.1 Leverage Openscapes to Develop Best Practices and Direct Culture Change

NMFS Open Science local mentors
Team-based facilitation

3.2 Leverage Openscapes to Develop Open Science Human Capital Requirements

Skills and People

3.3: Support the tools and platforms for Open Science

Addressing structural barriers

Line of Effort 3

Separate talk

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 15

16 of 30

What is Openscapes?

Not trainings/workshops

Cohort-based remote sessions for teams: introduce concepts and workflows; facilitate teams to talk about problems then go and solve them, with accountability and support.
It’s about getting stuff done. It’s about identifying and making progress on barriers
“A process to help you build better lanes of communication” -Laura Waters, SE Regional Office
Strengthening a teaching & learning culture within teams & organizations

Openscapes works with many environmental orgs

*No coding or software skills required*

https://openscapes.org/

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 16

Openscapes has worked with many environmental organizations. This is crucial for us because it helps our early adopters connect with early adopters in other environmental organizations.

Eli presents

UNCESCO, NASA quotes. Eli + Julie just presented to UN. You can google this!

What is Openscapes? This is not your traditional training or workshop.

These are cohort-based remote sessions for teams, where Openscapes introduces concepts and workflows; facilitate teams to talk about problems then go and solve them, with accountability and support.

It’s about getting stuff done. It’s about identifying and making progress on barriers

Laura Waters at the SERO described it as “A process to help you build better lanes of communication”

What’s important too is that there is sustainability and scalability built-in. It’s about strengthening a teaching & learning culture within teams & orgs. Not just for scientists: supervisors, admin, IT staff, etc, welcomed. Equitable.

Openscapes has led 10 Champions Cohorts so far, half of which have been with NMFS! At NMFS it’s been a grassroots effort

During Openscapes sessions, scientists talk about issues that are hindering their work. Big issues: reproducibility and tracking, team awareness, on- and off-boarding, too much duplication of effort across the center, agency and year. Last need for team training to collaborate openly and effectively.

Next slides

17 of 30

2020-2022

Openscapes Champions program

10 cohorts, all science centers + HQ + regional offices 300+ staff

Grassroots science center led

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 17

18 of 30

NMFS Openscapes Mentors Program 2023-2026

TASK NO.	TASK TITLE	YEAR 1				YEAR 2				YEAR 3
		2023		2024				2025				2026
		Q3	Q4	Q1	Q2	Q3	Q4	Q1	Q2	Q3	Q4	Q1	Q2
1	Develop Mentor Community
1.1	Outreach to new NMFS mentors
1.2	Openscapes Mentors Cohort Activities
1.3	Identify and Address Cross-Center Roadblocks
2	Empower Research Teams
2.1	Engage NMFS teams
2.2	Openscapes Champions Cohorts
2.3	Community Skills Workshops (R/Python)
2.4	Pathways to Open Science (DEI)
3	Amplify Open Science Leaders
3.1	Community engagement & webinars
3.2	Coordinate, comms, tag-ups

Office of Science and Technology Initiative

19 of 30

NASA Openscapes Mentors

Slow down to speed up: deep investment in small numbers 1st.

Teams from 7/12 different data centers opted in, didn’t know each other beforehand. Thinking of themselves as a community helped them see the common parts of workflows for NASA Earthdata

NASA Award# 20-TWSC20-2-0003 Leads: Julia Stewart Lowndes & Erin Robinson

A mentor community across NASA Earth science data centers

Andy Barrett • Chris Battisto • Brandon Bottomley • Aaron Friesz • Alexis Hunzinger • Mahsa Jami • Alex Lewandowski • Bri Lind • Luis Lopez • Catalina Oaida Taglialatela • Celia Ou • Jack McNelis • Cassie Nickles • Brianna Pagán • Sargent Shriver • Amy Steiker • Michele Thornton • Makhan Virdi • Jess Welch

https://nasa-openscapes.github.io

Support researchers as they migrate analytical workflows to the Cloud:

Co-creating common tutorials; review & reuse process
Community of practice for teaching, mentoring, facilitation
Scaling open science leaders

Inspired by the NASA Openscapes Mentors Program

20 of 30

Identifying & responding to user needs

earthaccess Python library

Cookbook: current tutorials & onboarding

Cheatsheets & guides

2i2c JupyterHub: Python, R, Matlab,

corn base image: built on Pangeo stack

Value of Hosted JupyterHubs White paper / RFI

15+ workshops & talks led by Mentors since: reusing & extending tutorials with software & conceptual solutions

Mentors identify needs and develop collaborative projects to solve problems and lower barriers to adoption

–NASA Openscapes examples–

The mentors are learning how to work in the Cloud together using a 2i2c JupyterHub, where they are supporting researchers to work there in Python, R, and Matlab. They are leading learning events using tutorials they created and reuse and iterate for different audiences and events for researchers and other NASA staff, They are also identifying user needs and developing where they see needs including the earthaccess python library; cheatsheets and guides. They describe how they are now working openly with each other across their silos, feel braver speaking up in other places, and that they are comfortable sharing imperfect work with each other and getting and receiving feedback.

2i2c

Work of the Mentor Community

How to grow it - working on real researchers in the science teams

Luis showing this real example of work the Mentor community has done

We are really grateful for our NASA Mentors, and to NASA who is supporting their time so that open science can be a paid part of their jobs.

We’re supporting Support Mentors towards establishing a common set of NASA Earthdata tutorials that they can then build off their DAAC-specific and science examples

Again, these recurring themes. Wanted to reuse existing so needed to identify what is unique to NASA earthdata and what’s not, tease that apart to reuse existing. Only contribute to combine and fill gaps rather than reinventing

We center researchers, and we focus on this feedback loop between tooling and people

---

21 of 30

3.1 Direct Culture Change: Local mentor groups

We have developed a mentor community from the last 3 years of Openscapes cohorts

Collaborate with the local data governance teams

Identify problems that are causing “pain”

Break down barriers to information spread

People and interactions (face time)

22 of 30

3.1 Direct Culture Change: Team-based programs

10 weeks
5 facilitated sessions
5 team co-work sessions

It’s about making progress on specific work projects. It’s about identifying and making progress on barriers.

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 22

23 of 30

TASK

Set up single email address for pers comms re data requests

Example “data pathway” activity for an Openscapes cohort

This slide shows you an example of the type of work that a team does during an Openscapes Champions cohort idea. If you've ever done data journey mapping, that's what they're doing. The whole team is getting team awareness of all the different flows into some “product”. here it's a report chapter. The need to understand and talk about all the flows and then talk about “hey where are our problems?” “where's the manual stuff?” “where all the errors flowing in?” there's a lot of psychological safety work that has to be done with teams because the team has needs to have honest discussions about this and that’s hard. Next they decide where they want to go. Yeah, and then they get depressed because that seems so hard and so impossible. Then we help them stop being depressed about how much work there is and just focus on moving forward. Finding things that they can do to start moving forward and not just be stuck. Here's an example of something really small and here's an example of another team did something a little bigger. But making small steps to begin moving forward is key. Don't get so focused on the enormity of the goal.

24 of 30

24

Slide from NMFS R User Group Lightning Talks: Open Science Work in R Jan 31, 2023; (video)

People and interactions – Feedback cycles

Connection to the end user – Respond to change

25 of 30

NMFS Openscapes Mentors Program 2023-2026

TASK NO.	TASK TITLE	YEAR 1				YEAR 2				YEAR 3
		2023		2024				2025				2026
		Q3	Q4	Q1	Q2	Q3	Q4	Q1	Q2	Q3	Q4	Q1	Q2
1	Develop Mentor Community
1.1	Outreach to new NMFS mentors
1.2	Openscapes Mentors Cohort Activities
1.3	Identify and Address Cross-Center Roadblocks
2	Empower Research Teams
2.1	Engage NMFS teams
2.2	Openscapes Champions Cohorts
2.3	Community Skills Workshops (R/Python)
2.4	Pathways to Open Science (DEI)
3	Amplify Open Science Leaders
3.1	Community engagement & webinars
3.2	Coordinate, comms, tag-ups

Office of Science and Technology Initiative

26 of 30

Training in big data tools

Diversity, Equity, Inclusion, and Accessibility in data science

Openscapes: Pathways to Open Science

Promote role models and success stories
Mentorship opportunities
Creating safe spaces to have discussions about inclusion and diversity

Data science training

27 of 30

Address structural barriers to Open Science adoption

Support for NMFS developed research software, package development, templates, utilities

Soft infrastructure: development platforms, cloud virtual machines, and product delivery systems.

Governance groups and user groups

Wasting 10-20% of our time dealing with structural barriers. Data Science has soft infrastructure, governance, and community support needs. NMFS Open Science acts as a leader across our “open science adjacent” groups —intentionally working together to make things better for everyone rather than silo-ing, i.e. focusing only on solving our own problems. First off our soft infrastructure. NMFS has a Posit Connect server and GitHub Enterprise Cloud. Second Governance. Well managed systems and policies don’t just happen. We need governance groups and we think it is really important that those affected by policies are actively involved in developing those policies and governing the resources. We have two initiatives (FIT and FIMS) that help support our open source software development communities in different ways, including helping with soft infrastructure, best practices training, developing guidelines, speaker series.

28 of 30

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 28

29 of 30

Integrating Earth Systems/Climate models with fisheries population and ecosystem models (and socioeconomic models) is a major “big data” project which can be supported by an Open Science approach.

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service

Page 29

1 of 30

2 of 30

3 of 30

4 of 30

5 of 30

6 of 30

7 of 30

8 of 30

9 of 30

10 of 30

11 of 30

12 of 30

13 of 30

14 of 30

15 of 30

16 of 30

17 of 30

18 of 30

19 of 30

20 of 30

21 of 30

22 of 30

23 of 30

24 of 30

25 of 30

26 of 30

27 of 30

28 of 30

29 of 30

30 of 30